Method and system for profiling tcp traffic behavior based on network traffic metadata

Abstract

The application relates to the technical field of computer networks and discloses a TCP service behavior portrait method and system based on network flow metadata, which comprises the following steps: decoding captured TCP network flow and extracting distributed correlation identifiers to form application layer interaction records; converting the interaction records into unified standardized operation logs; organizing the logs into ordered behavior chains based on time windows; further analyzing the logs and the behavior chains, quantifying atomic behaviors by aggregation, and discovering frequent service process modes by sequence pattern mining; attributing the analysis results to specific service entities to form entity portraits containing service function portraits and service process atlases; and finally constructing a global distributed service atlas representing the calling relationship between services by using the distributed correlation identifiers. The dependence strength between services is accurately quantified, and data decision-making basis is provided for system operation and maintenance and architecture optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer networks, in particular to a TCP service behavior profiling method and system based on network traffic metadata. BACKGROUND

[0002] With the development of information technology, the core business systems of modern enterprises are increasingly evolving towards distributed and micro-service architectures. In such architectures, a single business request often needs to be completed by multiple independently deployed service entities in collaboration. Therefore, in order to ensure the stability of the system, carry out efficient fault diagnosis and continuous performance optimization, it is necessary to accurately observe and analyze the running state and internal behavior of the business system, which has become a key technical requirement in the field of system operation and management. Currently, the technical means for realizing system observability mainly include application log analysis, system indicator monitoring and distributed tracking, etc.

[0003] On the one hand, the scheme of relying on code instrumentation or deploying agent programs at the application layer to realize distributed tracking is invasive. This approach not only requires access to and modification of the source code of the business system, increasing the complexity and potential performance overhead of the system, but also is difficult to apply to third-party commercial software or old legacy systems that cannot obtain source code. On the other hand, existing analysis methods are mostly limited to the statistical level of atomic operations, such as counting the number of interface calls or response times, and lack the ability to automatically restore discrete operation logs to continuous business processes, making it difficult to automatically discover and abstract statistically significant behavior patterns that represent real business logic from massive data.

[0004] Most methods use the total number of request calls between services as an indicator to measure the strength of the dependency, which cannot effectively distinguish between "high-frequency calls triggered by a small number of business scenarios" and "extensive calls triggered by a large number of different business scenarios". This lack of precision in quantification leads to misjudgments in the tightness of business coupling between services and the scope of fault propagation, bringing uncertainty to the evolution and risk assessment of the system architecture. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a TCP service behavior profiling method and system based on network traffic metadata, which solves the problem that the prior art cannot perform deep behavior analysis on black-box TCP business systems, cannot automatically restore their internal business process patterns, and cannot objectively quantify the dependency strength between services.

[0006] To achieve the above purpose, the present application realizes the following technical scheme: a TCP service behavior profiling method and system based on network traffic metadata, the method comprising:

[0007] The captured TCP network traffic is multi-layer protocol decoded, and a distributed correlation identifier is extracted to form a structured application layer interaction record;

[0008] The structured application layer interaction record is converted into a standardized operation log with unified format;

[0009] Based on the operator, the service party and the preset time window, the discrete standardized operation log is organized into an ordered behavior chain.

[0010] The standardized operation log is aggregated to quantify atomic behaviors, and the behavior chain is sequentially pattern-mined to find frequent business process patterns.

[0011] The aggregated atomic behaviors and the mined business process patterns are attributed to specific service entities to complete the service function portrait and business process restoration of a single service.

[0012] The distributed correlation identifier is used to construct a global distributed business graph representing the calling relationship between services.

[0013] In a further technical solution, the step of extracting the distributed correlation identifier is: based on a preset correlation identifier lookup rule set that maps application layer protocol types to metadata field names to be looked up, in the metadata area of the decoded protocol data, the field value matching the field name in the rule set is found and extracted as the distributed correlation identifier.

[0014] In a further technical solution, the step of organizing the discrete standardized operation log into an ordered behavior chain specifically includes: grouping the standardized operation logs with the same operator address and service entity; and in the group, the time stamp difference between any two adjacent operation logs in the log sequence arranged in ascending order of operation time stamp is greater than or equal to the preset time window threshold, and the point is taken as a cutting point to divide the log sequence into one or more behavior chains.

[0015] In a further technical solution, the step of sequentially pattern-mining the behavior chain to find frequent business process patterns specifically includes: using a sequential pattern-mining algorithm to find potential business process patterns from the behavior chain set; calculating the support of each business process pattern, which is the proportion of the behavior chain containing the pattern as a subsequence in the entire behavior chain set; and determining the business process pattern with a support greater than or equal to the preset minimum support threshold as the frequent business process pattern.

[0016] In a further technical solution, the distributed business graph is a directed weighted graph, and vertices of the graph are composed of the service entities. A directed edge from a vertex representing a service entity A to a vertex representing a service entity B is established on the condition that there exist two standard operation logs of the service entities A and B, the two operation logs have the same and non-empty distributed correlation identifier, and the operation timestamp of the log of the service entity A is earlier than the operation timestamp of the log of the service entity B.

[0017] In a further technical solution, the weight of an edge in the directed weighted graph is the total number of non-repeated distributed correlation identifiers triggering the inter-service call represented by the edge. This definition can effectively reflect how many times an independent end-to-end business process is carried out, thereby providing accurate quantification of the degree of inter-service dependence.

[0018] The second aspect of the application provides a TCP business behavior profiling system based on network traffic metadata, which comprises:

[0019] a traffic acquisition and decoding module configured to perform multi-layer protocol decoding on captured TCP network traffic and extract distributed correlation identifiers to form structured application layer interaction records;

[0020] a log standardization module configured to convert the structured application layer interaction records into standard operation logs with unified format;

[0021] a behavior chain construction module configured to organize discrete standard operation logs into ordered behavior chains based on operation parties, service parties and a preset time window;

[0022] an analysis and mining module configured to aggregate the standard operation logs to quantify atomic behaviors and perform sequential pattern mining on the behavior chains to find frequent business process patterns;

[0023] a service profiling generation module configured to attribute the aggregated atomic behaviors and the mined business process patterns to specific service entities to complete service function profiling and business process restoration of a single service;

[0024] a distributed graph construction module configured to construct a global distributed business graph representing inter-service calling relationship by using the distributed correlation identifiers.

[0025] The application provides a TCP business behavior profiling method and system based on network traffic metadata, which has the following beneficial effects:

[0026] 1. The application realizes non-intrusive monitoring and profiling of the business system by analyzing real-time TCP network traffic captured by network switch equipment mirror ports or host network cards. This method does not require modifying any source code of the business system or deploying proxy programs inside the application for bytecode instrumentation, thereby avoiding code coupling, potential performance overhead and risks to business stability brought by intrusive solutions. It can objectively and truly reflect the actual behavior of the system running online.

[0027] 2. The application not only atomizes the statistics of discrete network interactions, but also constructs behavior chains and applies sequence pattern mining algorithms to improve the analysis dimension from a single operation to the business process level. Through this method, the system can automatically discover the operation sequence frequently called by service entities in real business scenarios, i.e., business process patterns, thereby forming a two-dimensional stereoscopic profile containing "service function profile" and "business process atlas", which can more deeply reveal the internal business logic of services than single function call statistics.

[0028] 3. The application uses distributed correlation identifiers to construct a global business atlas, and innovatively defines the edge weight of the graph as the total number of distributed correlation identifiers triggering inter-service calls without repetition. This weight definition discards the traditional request count statistics and can more accurately quantify the business breadth of the dependency relationship between two services rather than traffic load, providing more objective and accurate data decision basis for architecture optimization, key link identification, fault impact analysis and capacity planning of distributed systems. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a method flowchart of the application;

[0030] Figure 2 is a multi-layer protocol decoding and distributed correlation identifier extraction step flowchart of the application;

[0031] Figure 3 is a data mapping diagram of the standardized operation log extraction step of the application;

[0032] Figure 4 is a behavior chain construction process diagram of the application;

[0033] Figure 5 is a behavior aggregation and business process pattern mining step diagram of the application;

[0034] Figure 6 is a service profile construction diagram of the application;

[0035] Figure 7 is a structure diagram of the distributed business atlas of the application;

[0036] Figure 8System framework diagram of the present application. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the specification of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application.

[0038] Refer to the drawings Figure 1 , Figure 1 is a flowchart of a network traffic metadata-based TCP service behavior profiling method according to an embodiment of the present application. The present application provides a network traffic metadata-based TCP service behavior profiling method, which comprises the following steps:

[0039] S100, performing multi-layer protocol decoding on captured TCP network traffic, and extracting a correlation identifier for distributed tracking to form a structured application layer interaction record;

[0040] S200, converting the interaction records of different protocols in the previous step into standardized operation logs in a unified format;

[0041] S300, organizing discrete standardized operation logs into an ordered behavior chain based on an operation party, a service party and a preset time window;

[0042] S400, aggregating operation logs to quantify atomic behaviors, and performing sequential pattern mining on the behavior chain to find frequent service flow patterns;

[0043] S500, attributing the aggregated atomic behaviors and the mined service flow patterns to specific service entities to complete service function profiling and service flow restoration of a single service;

[0044] S600, constructing a global distributed service graph representing the calling relationship between services by using the distributed correlation identifier.

[0045] The above steps in the embodiments of the present application will be described in detail below.

[0046] Refer to the drawings Figure 2 , Figure 2 is an internal flowchart of the multi-layer protocol decoding and distributed correlation identifier extraction step according to an embodiment of the present application. The network traffic metadata-based TCP service behavior profiling method provided by the present application starts with step S100, which aims to process and convert the captured original network data stream into a structured application layer interaction record with distributed transaction tracking information.

[0047] S110, the system performs network traffic acquisition and TCP session reassembly. Network traffic acquisition can be achieved by configuring port mirroring (SPAN) on network switching devices or deploying network taps (TAPs) in physical links, etc., to obtain real-time copies of network packets in a non-intrusive manner. In another embodiment, an agent can also be deployed on a host running a business application to directly capture network traffic in and out of the host. After obtaining the packets, the system classifies packets belonging to the same TCP connection according to the five-tuple information contained in the IP header and TCP header, i.e., source IP address, destination IP address, source port number, destination port number, and transport layer protocol type. The system correctly sorts and de-duplicates packets that arrive out of order or are retransmitted by processing the sequence number (Sequence-Number) and acknowledgement number (Acknowledgement-Number) of the TCP protocol, and finally reconstructs an ordered and complete application layer data stream.

[0048] S120, the system performs deep packet inspection on the reassembled application layer data stream to identify the application layer protocol it carries. A variety of techniques are used in combination to ensure accuracy. One implementation is to make a preliminary judgment based on the standard port number, for example, data streams with destination port 80 or 443 are preliminarily determined as HTTP or HTTPS protocol. To improve the accuracy of identification, the system further adopts an identification method based on payload feature signatures. The identification method compares the first few bytes of the data stream with a pre-set protocol feature library that stores specific strings or binary patterns at the start of the protocol handshake or request message for a variety of protocols (such as HTTP, SQL, gRPC, Dubbo, Redis, etc.). Once the payload content of the data stream successfully matches a certain feature signature in the library, the application layer protocol type is determined. For application layer protocols encrypted by transport layer security protocol (TLS) or secure sockets layer protocol (SSL), such as HTTPS, at least two sub-implementation methods are provided to obtain the internal application layer data. The first method is to deploy an agent on the host as a service entity, which uses the interface provided by the operating system to intercept the application layer data before it is encrypted and sent to the protocol stack or after it leaves the protocol stack after decryption. The second method is to interact with the key management server in the network or by pre-configuring the private key and certificate of the business system on the traffic collection device to decrypt the encrypted session in real time during traffic collection. Through either of the above methods, the system can obtain the decrypted plaintext application layer data stream and hand it over to the subsequent protocol parser for processing.

[0049] S130, the system calls a special parser corresponding to the protocol type identified in the previous step to decode the application layer data stream and extract the distributed correlation identifier in the decoding process. The distributed correlation identifier is a credential used to uniquely mark and link multiple service nodes through which a complete business request flows in a distributed system or microservice architecture. The extraction of the distributed correlation identifier is the key to realizing cross-service behavior tracking.

[0050] The extraction mechanism of this step is based on a configurable and preset correlation identifier lookup rule set. The correlation identifier lookup rule set maps different application layer protocol types to one or more lists of metadata field names to be looked up. For example, for the HTTP protocol, its corresponding field name list can include "Trace-ID", "X-B3-TraceId", "traceparent", and other industry standard distributed tracking field names. For the gRPC protocol, its corresponding field name list is the key name to be looked up in its metadata payload.

[0051] When decoding the protocol data, the parser synchronously accesses the metadata area of the protocol, such as the request header part of the HTTP protocol. The parser compares all field names in this area with the list of field names defined in the rule set for the protocol. Once a field that exists in the protocol data and completely matches a field name in the rule set list is found, the system immediately extracts the corresponding value of the field and takes the corresponding value as the distributed correlation identifier of this network interaction. If no match is found after traversing all preset rules, the distributed correlation identifier of this interaction is assigned a specific null value or marker, indicating that it does not belong to a currently traceable distributed transaction.

[0052] S140, the system integrates the decoding and extraction results into a structured application layer interaction record. The application layer interaction record is a data structure that at least includes the following fields: the timestamp of the interaction, the source IP address, the destination IP address, the source port number, the destination port number, the identified application layer protocol type, the key business fields decoded from the protocol (such as the request method and URI path of HTTP or the query type and table name of SQL), and the distributed correlation identifier extracted in S130 (or its null marker). The sequence of interaction records output in this step provides a structured data input with business semantics and tracking capabilities for all subsequent analysis steps.

[0053] Refer to the accompanying Figure 3 , Figure 3is a data mapping diagram of the standardized operation log extraction step according to an embodiment of the present application. Step S200 in the method of the present application aims to map the structured interaction records of various protocol types generated in step S100 into standardized operation logs with fixed data structures through a deterministic conversion process. The execution of this step ensures that all subsequent analysis steps can process regular and consistent data units, and is the basis for realizing the unified analysis of heterogeneous network behaviors.

[0054] S210, the system defines a standardized operation log data structure. The operation log data structure aims to atomically describe the core elements of any network interaction in a protocol-independent manner. In a specific embodiment, the standardized operation log is defined as a multi-tuple:

[0055] ;

[0056] Wherein:

[0057] : Operation timestamp, recording the exact time when the operation occurs;

[0058] : Operation address, i.e. the IP address of the client initiating the network interaction;

[0059] : Service entity, a two-tuple composed of the server IP address and port number providing the service, used to uniquely identify a service instance in the network;

[0060] : Application layer protocol, the application layer protocol type carrying the operation, such as HTTP, SQL, etc.;

[0061] : Operation name, the standardized operation type or method name abstracted from the protocol interaction;

[0062] : Operation object, the identifier of the resource or object directly acted upon by the operation;

[0063] : Operation level, a preset level representing the importance or security sensitivity of the operation service;

[0064] : Distributed association identifier, inherited from the tracking ID of step S100, used for associating cross-service calls.

[0065] S220, the system applies a pre-configured field mapping rule set to convert the structured interaction record from step S100 into the standardized operation log described above. The field mapping rule set explicitly defines the mapping from the protocol-specific fields of each supported application layer protocol to the fields of the standardized operation log.

[0066] For example, for the Hypertext Transfer Protocol, the mapping rules are as follows: the timestamp of the interaction record is assigned to the operation timestamp; the source IP address of the TCP connection is assigned to the operator address; the destination IP address and port number are combined into a two-tuple and assigned to the service entity; the string "Hypertext Transfer Protocol" is assigned to the application layer protocol field; the request method (such as GET, POST, DELETE) is assigned to the operation name; and the path part of the requested uniform resource identifier is assigned to the operation object.

[0067] For example, for the Structured Query Language protocol, the mapping rules are similarly defined as follows: the timestamp of the interaction record is assigned to the operation timestamp; the IP address of the client that initiated the query is assigned to the operator address; the IP address and listening port number of the database server are combined and assigned to the service entity; the string "Structured Query Language" is assigned to the application layer protocol field; the first keyword of the statement (such as SELECT, INSERT, UPDATE, DELETE) is assigned to the operation name; and the main database table name in the statement is assigned to the operation object.

[0068] S230, the system determines the operation level for each generated standardized operation log according to an independent, configurable level determination rule base. The level determination rule base associates specific operation characteristics with a pre-set level (e.g., normal, sensitive, critical). The determination rules are flexible, and their lower-level implementations can include:

[0069] One implementation is based on the operation name. For example, the rules can define that in all Structured Query Language protocols, the operation level of operations with the operation name "query" is "normal", and the operation level of operations with the operation name "delete" or "discard" is "critical".

[0070] Another implementation is based on the content pattern of the operation object. For example, the rules can use regular expressions to define that in all Hypertext Transfer Protocol requests, the operation level of requests whose operation object matches the "api / admin / *" path pattern is "sensitive". In more complex implementations, the determination rules can combine multiple fields for compound condition judgment. For example, a rule can be defined as follows: when the application layer protocol is Hypertext Transfer Protocol, the operation name is "submit", and the operation object is "api / users / delete", the operation level is determined to be "critical".

[0071] S240, after the conversion and assignment process described above, each structured interaction record outputted by step S100 is deterministically converted into a piece of information-complete standardized operation log. The final output of this step is a uniform operation log sequence sorted in ascending order of operation timestamp, which provides a regular and standardized data input for the subsequent behavior chain construction and pattern mining steps.

[0072] Referring to the accompanying drawings Figure 4 , Figure 4 is a schematic diagram of the behavior chain construction process according to an embodiment of the present application. The core task of step S300 in the method of the present application is to organize the discrete standardized operation log sequence outputted by step S200 into an ordered operation sequence that can reflect a continuous business process, which is defined as a behavior chain in the present application.

[0073] S310, the system groups all the standardized operation logs with a binary tuple composed of the address of the operation party and the entity of the service party as the context aggregation key. Specifically, the system traverses all the operation logs and groups the logs with exactly the same address of the operation party and the entity of the service party into the same group. This step ensures that each group contains only all the interaction activities sent by the same client to the same service instance, thereby establishing a clear business context for the construction of the behavior chain.

[0074] S311, within each log group formed by context aggregation, the system applies a pre-set time window parameter to segment the log stream to divide different behavior chains. The time window represents the maximum duration that the system identifies as a continuous business interaction that can be sustained independently. The time window parameter is a configurable numerical value, which can be set to several minutes to several tens of minutes in different embodiments, for example, for a typical Web application session, it can be set to 30 minutes. The selection of the time window threshold has a direct impact on the granularity of the behavior chain segmentation. A shorter time window (e.g. 1-5 minutes) tends to segment the user's single, quick operation sequence into shorter and more accurate behavior chains, which is suitable for analyzing transactional businesses with compact interactions. A longer time window (e.g. 30-60 minutes) tends to aggregate all the operations of the user in a login session into a longer behavior chain, which is suitable for analyzing businesses with features such as browsing, thinking, and other long pauses. In a preferred implementation, statistical analysis can be performed on the adjacent time intervals of all operation logs first, the probability distribution is observed, and a turning point or trough point in the distribution is selected as an adaptive time window threshold.

[0075] S312, the specific process of the segmentation is: in a log group, the system first arranges the logs in ascending order according to their operation timestamps. The system takes the first log in the ordered sequence as the start of the first behavior chain. Then, the system sequentially checks each subsequent log. If the time interval between a log and its immediately preceding log is less than the preset time window value, the log is attributed to the current behavior chain being built. If the time interval is greater than or equal to the preset time window value, the system determines that the current continuous interaction has been interrupted, the current behavior chain is completed, and the log is taken as the start of the next new behavior chain. This process is repeated on all logs in the group until all logs are attributed to a behavior chain.

[0076] 313, after the above segmentation, each divided log set, after being strictly arranged in ascending order according to its operation timestamp, constitutes a complete behavior chain. Formally, a behavior chain is defined as an ordered sequence of standardized operation logs that meet certain constraints:

[0077] ;

[0078] The definition contains the following constraints:

[0079] 1. Context identity: all operation logs in the sequence (where from 1 to ) have exactly the same operation party address and service party entity .

[0080] 2. Time sequence continuity: for any two adjacent operation logs and in the sequence, the difference between their operation timestamps satisfies ;

[0081] 3. Time order: the operation logs in the sequence are strictly arranged in ascending order according to their operation timestamps, i.e. ;

[0082] Where:

[0083] represents a behavior chain, which is an ordered sequence of operation logs;

[0084] represents the th standardized operation log in the behavior chain;

[0085] represents the th standardized operation log in the behavior chain;​

[0086] represents the total number of operation logs contained in a behavior chain;

[0087] represents the operation timestamp of an operation log

[0088] represents a preset time window threshold.

[0089] Through this step, the basic unit of data analysis is promoted from isolated, static operation logs to behavior chains capable of describing dynamic processes. This provides necessary, time-structured data input for the subsequent step S400 of mining and identifying business process patterns, making it possible to automatically discover business-level logic from traffic.

[0090] Referring to the accompanying Figure 5 , Figure 5 is a schematic diagram of a behavior aggregation and business process pattern mining step according to an embodiment of the present application. The core of step S400 in the method of the present application is to perform in-depth analysis and knowledge refinement on the standardized data generated by the previous step. This step includes two parallel data processing processes: one is atomic behavior aggregation, and the other is business process pattern mining.

[0091] S410, the system performs atomic behavior aggregation. The input of this process is the set of all standardized operation logs generated in step S200. The goal is to statistically quantify the basic network interaction behavior. Specifically, this process groups and counts all standardized operation logs using the combination of the operation party address, the service party entity, and the operation name as the key. The output of the group count is an atomic behavior record, which explicitly quantifies the total frequency of a specific operation party performing a specific operation on a specific service entity. The calculation method of this operation frequency is defined in form as follows:

[0092] ;

[0093] wherein:

[0094] represents the operation frequency, and represents the total number of standardized operation logs that meet the specific aggregation condition;

[0095] represents the set of all standardized operation logs;

[0096] represents an independent operation log in the set

[0097] ​​represents the address of the operation party for aggregation;

[0098] represents the service party entity for aggregation;

[0099] represents the operation name for aggregation;

[0100] represents the cardinality operator of a set, used to calculate the number of elements in a set.

[0101] The output of this process is a collection of atomic behavior records, which constitutes a macroscopic statistical view of all atomic interactions in the network, providing the basic data for subsequent construction of service function portraits.

[0102] S420, the system performs business process pattern mining. The input of this process is the collection of all behavior chains generated in step S300. The goal is to automatically and unsupervisedly discover statistically significant and repeatedly occurring behavior sub-sequences from these behavior sequences representing real business processes, which are defined as business process patterns in this invention.

[0103] In a specific embodiment, this mining process is implemented using a sequence pattern mining algorithm. Sequence pattern mining algorithm is a class of data mining techniques that are specifically used to discover frequently occurring sub-sequences from sequence databases. This invention can use a variety of sub-implementations of this class of algorithms, such as the PrefixSpan algorithm, the GSP algorithm or the SPADE algorithm.

[0104] S421, to determine whether a business process pattern is statistically significant, the system introduces the concept of support. The support of a business process pattern is precisely defined as the proportion of behavior chains containing the business process pattern as a sub-sequence in the collection of all behavior chains. The formula for calculating this support is as follows:

[0105] ;

[0106] Wherein:

[0107] : the support of a business process pattern ;

[0108] : a business process pattern, which is itself an ordered sequence composed of one or more operations, such as <operation A, operation B, operation C>;

[0109] : the collection of all behavior chains

[0110] : set an independent behavior chain;

[0111] : subsequence relation symbol, indicating is a subsequence of. A sequence A is a subsequence of another sequence B if A can be derived from B by deleting zero or more elements from B, without changing the order of the remaining elements;

[0112] : cardinality operator of set.

[0113] S422, the system compares the support of each potential business process pattern calculated with a pre-set minimum support threshold. The minimum support threshold is a configurable parameter, whose value determines the sparsity of the mining result. The setting of the minimum support threshold directly affects the generality and quantity of the business process patterns discovered. A higher threshold will only retain those core business processes that are extremely common in the system, which helps to quickly locate the key business. A lower threshold can discover more long-tail, non-mainstream or occasional business processes, which helps to conduct more comprehensive business audit or abnormal behavior discovery. In practical applications, the minimum support threshold can be manually set by technical personnel according to the analysis target, or automatically optimized through information entropy and other indicators, so as to balance between the number and representativeness of the patterns. When the support of a business process pattern is greater than or equal to the minimum support threshold, the system determines that the pattern is a frequent business process pattern, and records it. The final output of this process is a set, which contains all the business process patterns determined to be frequent, and these business process patterns are the core basis for subsequent restoration of service business processes.

[0114] Referring to the accompanying Figure 6 , Figure 6 is a service portrait construction schematic diagram according to an embodiment of the present application. Step S500 in the method of the present application aims to accurately attribute the atomic behaviors and business process patterns analyzed and mined in step S400 to the specific service entity where they occur, and further construct a complete and three-dimensional behavior portrait of each independent service.

[0115] S510, the system performs the construction of the service function portrait. The input of this process is the set of all atomic behavior records output in step S410. The system takes the service side entity as the aggregation key, and performs secondary aggregation on these atomic behavior records. For any service entity in the network, a service function thereon is defined as a data record. The data record contains the following three core elements: function name, total usage amount of the function, and the set of clients calling the function.

[0116] The generation rule of the data record is clear: the function name is directly taken from the operation name field in the atomic behavior record; the total usage of the function is the sum of the operation frequencies of all atomic behavior records whose service entity and operation name match the current function; the client set is the set of all non-duplicate operation parties that have invoked the function. By performing this process on all atomic behavior records of a service entity, the system can generate a list of all service functions provided by the service and their quantitative usage. The complete set of all service function data records of a service constitutes the function portrait of the service.

[0117] S520, the system performs the restoration of the business process. The input of this process is the set of all frequent business process patterns mined in step S420. The system attributes these frequent business process patterns to the corresponding service entity where they occurred. Since each business process pattern originates from one or more behavior chains with a determined service party entity, and in step S300 all operations within a behavior chain are directed to the same service party entity, the attribution relationship is clear and unambiguous. A business process pattern is attributed to the service party entity to which its source behavior chain is directed.

[0118] The output of this process is to generate a business process atlas for each service entity. The business process atlas of a service entity is the set of all frequent business process patterns that occurred on the service.

[0119] S530, the system integrates the results of the above two parts to form the final portrait of a single service entity. The complete portrait of a service entity is composed of its function portrait and business process atlas. The complete portrait of a service entity not only reveals the atomic functions provided by the service and their frequencies and sources through the function portrait, but also describes the typical sequences in which these atomic functions are ordered and invoked in real business scenarios through the business process atlas. The output of this step, a series of portraited service entities, provides rich node information for the subsequent step S600 to build a global, cross-service distributed business atlas. In a specific embodiment, the final portrait of each service entity is stored as a structured data object, such as a JSON document. The top-level field of the JSON document contains the identification of the service entity, and under it there are two main sub-objects: a list named 'function portrait', where each element is a data record describing an atomic function and its statistics; and a list named 'business process atlas', where each element is an operation sequence representing a frequent business process pattern. This formatted output facilitates subsequent system retrieval, visualization display, or further automated analysis.

[0120] Refer to the attachedFigure 7 , Figure 7 is a structural diagram of a cross-service distributed business graph according to an embodiment of the present application. Step S600 in the method of the present application is a top-level analysis step of the entire method, and aims to correlate the behavior images scattered in various service entities using the distributed correlation identifiers extracted and standardized in the previous steps, and finally build a distributed business graph that can globally display the flow topology and dependency strength of the business process between different services.

[0121] S610, the system defines the constructed distributed business graph as a directed weighted graph. The structure of the directed weighted graph is composed of three elements: vertices, directed edges, and edge weights, and its construction process is completely generated by the network traffic metadata processed by the previous steps.

[0122] S620, the system determines the vertex set of the directed weighted graph. The vertex set is composed of all independent service entities recognized by the method of the present application in the network. Specifically, the system traverses all standardized operation logs generated in step S200, extracts all non-repeated values of the service entity field therein, and maps each non-repeated service entity to a unique vertex in the graph.

[0123] S630, the system establishes directed edges between the vertices of the directed weighted graph according to the time and logical relationship between the standardized operation logs. A directed edge from a vertex representing service entity A to a vertex representing service entity B is established, which must satisfy all the following conditions: there are at least two standardized operation logs, one of which is service entity A and the other is service entity B; A and B are different service entities; the two operation logs have exactly the same and non-empty distributed correlation identifier; the operation on service entity A holding the distributed correlation identifier has an earlier timestamp than any operation on service entity B. The direction of this directed edge, i.e. from the vertex representing the service entity where the previous operation occurs to the vertex representing the service entity where the subsequent operation occurs, accurately represents the flow direction of the business call chain.

[0124] S640, the system assigns a weight to each directed edge in the directed weighted graph, which is used to accurately quantify the strength of the call relationship between two service entities. In a specific embodiment, the weight of an edge is defined as the total number of unique distributed correlation identifiers that trigger the inter-service call represented by the directed edge. This definition can effectively reflect how many times the independent end-to-end business process is carried out, rather than simply counting the number of requests. The calculation formula of the weight is defined as:

[0125] satisfies the edge definition condition and ;

[0126] wherein:

[0127] : the weight of a directed edge in a graph from vertex to vertex ;

[0128] : a vertex in a graph, representing two different service entities respectively;

[0129] : a unique distributed association identifier string;

[0130] : two independent normalized operation logs, whose service entities are represented by and respectively;

[0131] : the distributed association identifier field in a normalized operation log;

[0132] : the cardinality operator of a set, which is used here to count the number of non-duplicate elements in a set.

[0133] By performing this step, the present application finally generates a data-driven global business view that objectively reflects the real topology relationship of microservice architecture in the runtime. This graph not only intuitively shows the calling relationship and direction between services, but also provides an accurate quantitative means for the tightness of service interdependence through its weight definition.

[0134] The present application also provides a TCP service behavior profiling system based on network traffic metadata. In a specific embodiment, the system can be deployed on one or more servers. Each server has a central processor, memory, a high-speed network interface card, and a non-volatile storage device for storing data. When the system is deployed in a cluster mode, the server nodes are interconnected through high-speed networks and can use distributed computing frameworks (such as Apache Spark or Flink) to cooperatively process large-scale network traffic data. The system is logically composed of the following function modules that cooperate with each other.

[0135] ​The traffic acquisition and decoding module is responsible for performing step S100 in the method of the present application. The traffic acquisition and decoding module receives packet copies from the network switch mirror port through its network interface. Inside the traffic acquisition and decoding module, there is a packet capture engine that grabs packets in real time and invokes a TCP session reassembly engine. The reassembly engine maintains a TCP connection state table in memory, classifies packets according to their five-tuple information, and reconstructs ordered application layer flows by processing TCP sequence and acknowledgement numbers. Next, a deep packet inspection engine uses its built-in, updatable protocol signature library to identify the protocol of the reassembled flows. Once the protocol is identified, the traffic acquisition and decoding module dispatches a dedicated parser corresponding to the protocol for deep decoding. During the decoding process, a correlation ID extraction unit looks for and extracts distributed tracing IDs in the protocol's metadata area (e.g. HTTP request headers) according to a configurable rule set. The final output of the traffic acquisition and decoding module is a stream of structured application layer interaction records, which are sent to the next module for processing.

[0136] The log standardization module is responsible for performing step S200 in the method of the present application. The log standardization module receives a stream of application layer interaction records from the previous module. Inside the log standardization module, there is a core mapping and conversion engine that loads a set of protocol field mapping rules. For each incoming interaction record, the engine extracts its unique fields (e.g. HTTP request method and URI) according to its protocol type and fills them into the corresponding fields of a standardized operation log data structure (e.g. operation name and operation object). Meanwhile, an operation level determination engine matches the operation name and operation object fields of the generated operation log against another independent rule base that supports regular expressions and compound condition judgments, and assigns a business importance level (e.g. normal, sensitive, or critical) to them. The log standardization module ensures that all network behaviors originating from different protocols are converted into standardized operation logs with uniform format and rich semantics, and outputs them in timestamp order.

[0137] The behavior chain construction module is responsible for performing step S300 in the method of the present application. The behavior chain construction module receives the standardized operation log stream as input. Inside the behavior chain construction module, the context aggregation unit first distributes the massive operation logs into different processing queues with the binary tuple of operation party address and service entity as the key, ensuring that the logs in each queue belong to the same client's interaction with the same service. In each queue, the sequence segmentation unit cuts the time-ordered log sequence according to the preset time window threshold. When the timestamp interval between two adjacent logs exceeds the threshold, it is determined that the continuous interaction is interrupted, and thus the log stream is cut into multiple independent behavior chains representing continuous business processes. The behavior chain construction module organizes the discrete log data into sequence data with temporal context, laying the foundation for subsequent pattern mining.

[0138] The analysis and mining module is responsible for performing step S400 in the method of the present application, and is the computing core of the system. The analysis and mining module includes two parallel sub-modules. The first is the atomic behavior aggregation sub-module, which processes all standardized operation logs to count the total occurrence frequency of each atomic operation with operation party, service entity, and operation name as the aggregation key, thereby obtaining the macroscopic statistical view. The second is the business process mining sub-module, which receives the behavior chain set output by the behavior chain construction module. The sub-module is embedded with an efficient sequence pattern mining algorithm (such as the implementation of PrefixSpan), and extracts all frequently occurring behavior subsequences from the behavior chain set according to the configurable minimum support threshold, which are business process patterns. The analysis and mining module realizes the key refinement from raw data to abstract knowledge.

[0139] The service portrait generation module is responsible for performing step S500 in the method of the present application. The service portrait generation module obtains the statistical results of atomic behaviors and frequent business process patterns from the analysis and mining module. The functional portrait generation unit in the service portrait generation module aggregates all atomic behavior statistics directed to the service in units of service entities, forming the function list of the service, the usage of each function, and the call source, etc., to constitute the functional portrait of the service. At the same time, the business process restoration unit collects the business process patterns mined from the service to constitute the business process atlas of the service. Finally, the integration unit merges the functional portrait and the business process atlas to generate a complete and structured two-dimensional portrait file for each service entity, and stores it in the portrait database of the system.

[0140] The distributed graph construction module is responsible for performing step S600 in the method of the present application. The distributed graph construction module is the last step to achieve the global business view. The distributed graph construction module first traverses all the normalized operation logs, extracts all non-repeated service entities, and instantiates them as vertices of the graph. Subsequently, the edge generation unit groups all the operation logs according to the distributed correlation identifiers carried by them. Within each group, the unit establishes directed edges between the corresponding vertices according to the time sequence of the operation occurrence and the difference of the service entities belonging to them, in order to depict the calling flow relationship between services. Finally, the weight calculation unit calculates the weight of each edge in the graph, and the weight value is defined as the total number of independent distributed correlation identifiers that triggered the inter-service call represented by the edge. The distributed graph construction module finally outputs a directed weighted graph data structure, which intuitively and quantitatively presents the global business topology and dependency strength of the entire distributed system at runtime.

Claims

1. A method for profiling TCP service behavior based on network traffic metadata, characterized in that, include: Step S100: Perform multi-layer protocol decoding on the captured TCP network traffic and extract distributed association identifiers to form a structured application layer interaction record. Step S200: Convert the structured application layer interaction records into standardized operation logs with a unified format; Step S300: Based on the operator, the service provider, and a preset time window, organize the discrete standardized operation logs into an ordered chain of behaviors. Step S400: Aggregate the standardized operation logs to quantify atomic behaviors, and perform sequence pattern mining on the behavior chain to discover frequent business process patterns. Step S500: Assign the aggregated atomic behaviors and the mined business process patterns to specific service entities to complete the service function profile and business process restoration of a single service. Step S600: Using the distributed association identifier, construct a global distributed business graph representing the inter-service call relationship; The specific steps for extracting the distributed association identifier in step S100 are as follows: Based on a preset set of association identifier lookup rules that maps application layer protocol types to the names of metadata fields to be searched, the field values ​​that match the field names in the rule set are searched and extracted in the metadata area of ​​the decoded protocol data, and used as the distributed association identifier.

2. The TCP service behavior profiling method based on network traffic metadata according to claim 1, characterized in that, The standardized operation log generated in step S200 is a data structure, which includes at least: operation timestamp, operator address, service entity, application layer protocol, operation name, operation object, operation level, and the distributed association identifier.

3. The TCP service behavior profiling method based on network traffic metadata according to claim 1, characterized in that, Step S300, which organizes the discrete standardized operation logs into an ordered chain of behaviors, specifically includes: Standardized operation logs with the same operator address and service provider entity are grouped together; Within a group, in a log sequence arranged in ascending order of operation timestamps, the point where the difference between the timestamps of any two adjacent operation logs is greater than or equal to the preset time window threshold is used as a splitting point, and the log sequence is divided into one or more of the aforementioned behavior chains.

4. The TCP service behavior profiling method based on network traffic metadata according to claim 1, characterized in that, The step S400, which involves performing sequence pattern mining on the behavior chain to discover frequent business process patterns, specifically includes: A sequence pattern mining algorithm is used to discover potential business process patterns from a set of behavior chains; Calculate the support of all business process patterns, where the support is the proportion of the behavior chain containing the business process pattern as a subsequence in the entire set of behavior chains; Business process patterns with a support level greater than or equal to a preset minimum support threshold are identified as frequent business process patterns.

5. The TCP service behavior profiling method based on network traffic metadata according to claim 1, characterized in that, The steps in step S500 that complete the service function profile and business process reconstruction for a single service specifically include: Construct a service function profile, which includes the service function name, total function usage, and set of clients calling the function; In addition, the business process graph is reconstructed, which is a collection of all frequent business process patterns belonging to the service entity.

6. The TCP service behavior profiling method based on network traffic metadata according to claim 1, characterized in that, The distributed service graph constructed in step S600 is a directed weighted graph, whose vertices are composed of the service entities.

7. The TCP service behavior profiling method based on network traffic metadata according to claim 6, characterized in that, In the directed weighted graph, a directed edge from a vertex representing service entity A to a vertex representing service entity B is established under the following conditions: there are two standardized operation logs with service providers A and B, the two operation logs have the same distributed association identifier that is not null, and the operation timestamp of the log with service provider A is earlier than the operation timestamp of the log with service provider B.

8. The TCP service behavior profiling method based on network traffic metadata according to claim 7, characterized in that, The weight of an edge in the directed weighted graph is the total number of unique distributed association identifiers that trigger inter-service calls represented by the directed edge.

9. A TCP service behavior profiling system based on network traffic metadata, wherein the TCP service behavior profiling method based on network traffic metadata according to any one of claims 1-8 is characterized in that, include: The traffic acquisition and decoding module is used to perform multi-layer protocol decoding on captured TCP network traffic and extract distributed association identifiers to form structured application layer interaction records. The log standardization module is used to convert the structured application layer interaction records into standardized operation logs with a unified format. The behavior chain construction module is used to organize the discrete standardized operation logs into an ordered behavior chain based on the operator, the service provider, and a preset time window. The analysis and mining module is used to aggregate the standardized operation logs to quantify atomic behaviors and to perform sequence pattern mining on the behavior chains to discover frequent business process patterns. The service profile generation module is used to assign the aggregated atomic behaviors and the mined business process patterns to specific service entities in order to complete the service function profile and business process restoration of a single service. The distributed graph construction module is used to construct a global distributed business graph representing the inter-service call relationships using the distributed association identifier; The specific steps for extracting the distributed association identifier are as follows: Based on a preset set of association identifier lookup rules that maps application layer protocol types to the names of metadata fields to be searched, the field values ​​that match the field names in the rule set are searched and extracted in the metadata area of ​​the decoded protocol data, and used as the distributed association identifier.

Images