Abnormality analysis method and device, equipment, storage medium and product

Abstract

The invention discloses an exception analysis method and device, equipment, a storage medium and a product, and relates to the technical field of exception analysis, and the method comprises the steps: carrying out the feature extraction of a pre-obtained HTTP network data flow according to a pre-configured communication feature extraction strategy, obtaining an HTTP communication feature data packet, and enabling the communication feature data packet to contain request association information; according to the request association information, target anomaly analysis parameters in the HTTP communication feature data packets are analyzed, system anomaly analysis is carried out according to the target anomaly analysis parameters, the feature data in the HTTP network data flow are extracted through the scheme, and the feature data can be extracted from the HTTP network data flow no matter the feature data appear at the request sending end or the request receiving end. According to the method, the fault occurrence position can be accurately positioned and abnormity analysis can be carried out for faults of important links in network transmission links or other network transmission links, so that the fault diagnosis precision and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of abnormality analysis technology, and in particular to an abnormality analysis method, device, equipment, storage medium and product. Background Art

[0002] In modern software architecture, distributed microservices have become the mainstream way to build large and complex systems. This architecture improves the scalability and reliability of the system by distributing different service nodes. However, with the increase of dependencies between services, tracking and monitoring the execution of requests on each node becomes complicated. Especially when failures occur, accurately locating the problem becomes a major challenge for system operation and maintenance. In order to meet this challenge, distributed link tracking systems are widely used. They can capture and record the performance data of requests on each service node, and display this data through a visual interface to help developers and operation and maintenance personnel understand the behavior of the system.

[0003] Although the distributed link tracking system provides the ability to track requests and monitor faults to a certain extent, the existing technology has obvious deficiencies in accuracy and ease of use. For example, traditional systems can often only monitor the call time between service nodes, but cannot go deep into the specific links of network transmission to identify the specific fault points. In addition, when there is a delay problem in the service call chain, the traditional APM system cannot accurately identify whether it is an abnormal write request on the server side, an abnormal receive request on the receiving end, or a problem in a certain link in the network transmission. This limitation causes operation and maintenance personnel to still need to perform tedious manual troubleshooting when faced with complex faults, especially when the transmission path is long or the fault frequency is low, the troubleshooting work becomes extremely time-consuming and inefficient.

[0004] In addition, existing distributed link tracking systems usually require specific monitoring codes to be embedded in the application code or use a specific framework, which not only increases the complexity of system development and maintenance, but also makes it difficult to access some purchased modules or services, resulting in interruptions in link tracking and reduced positioning accuracy. These problems show that although distributed link tracking systems provide effective monitoring methods in theory, they still face challenges of insufficient accuracy and high invasiveness in practical applications.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide an anomaly analysis method, apparatus, device, storage medium and computer program product, aiming to improve the accuracy and efficiency of distributed system fault diagnosis.

[0007] To achieve the above objectives, the present application proposes an abnormality analysis method, which comprises:

[0008] Extracting features from a pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy to obtain an HTTP communication feature data packet, wherein the communication feature data packet contains request association information;

[0009] According to the request association information, target anomaly analysis parameters in the plurality of HTTP communication characteristic data packets are parsed to perform system anomaly analysis according to the target anomaly analysis parameters.

[0010] In one embodiment, the HTTP network data stream is provided with an HTTP communication session record and a data packet buffer area, and the step of extracting features from the pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy to obtain an HTTP communication feature data packet includes:

[0011] Extracting a network data packet from the HTTP network data stream to obtain a TCP message based on the network data packet, wherein the network data packet includes HTTP communication characteristic data;

[0012] If the TCP message is a data packet message, then according to a preset data packet processing strategy, the network data packet is stored in the HTTP communication session record or the data packet buffer area;

[0013] If the TCP message is an end packet message or a reset packet message, based on the TCP quaternary group corresponding to the data packet message, the network data packet is stored in the HTTP communication session record;

[0014] When a preset record output condition is met, the HTTP communication characteristic data in the HTTP communication session record is output to generate the HTTP communication characteristic data packet based on the communication characteristic data.

[0015] In one embodiment, if the TCP message is a data packet message, the step of storing the network data packet in the HTTP communication session record or the data packet buffer according to a preset data packet processing strategy includes:

[0016] If the TCP message is a data packet message, obtain and sort the TCP four-tuple corresponding to the data packet message to obtain a sorted TCP four-tuple;

[0017] Querying whether there is a key-value pair of the HTTP network data stream in a preset double-column set according to the sorted TCP four-tuple;

[0018] If the key-value pair of the HTTP network data stream exists, the network data packet is stored in the HTTP communication session record or the data packet buffer according to the data packet type of the network data packet and the attribute information of the HTTP network data stream.

[0019] In one embodiment, the HTTP communication session record includes a first HTTP communication session record and a second HTTP communication session record, and when a preset record output condition is met, the step of obtaining HTTP communication feature data in the HTTP communication session record to generate the HTTP communication feature data packet based on the communication feature data includes:

[0020] Obtaining the sequence number and confirmation number of the network data packet, and the sequence number and confirmation number of the existing response data packet in the second HTTP communication session record;

[0021] If the sequence number and confirmation number of the network data packet are after the sequence number and confirmation number of the existing response data packet, generating a data packet concatenation payload based on the target data packet in the first HTTP communication session record;

[0022] Identify and obtain HTTP communication feature data in the data packet concatenation payload;

[0023] The HTTP communication characteristic data packet is generated based on the HTTP communication characteristic data and pre-acquired metadata information.

[0024] In one embodiment, the method further comprises:

[0025] Traverse the preset double column set;

[0026] Determine whether the current silence duration of the current HTTP network data stream in the dual-column set exceeds a preset silence duration threshold;

[0027] If the current silence duration exceeds a preset silence duration threshold, obtaining a target key-value pair of the current HTTP network data flow in the double-column set;

[0028] The buffer data in the current HTTP network data stream is acquired according to the target key-value pair, and stored in a preset HTTP communication session record.

[0029] In one embodiment, the request association information includes a session record ID, and the step of parsing target anomaly analysis parameters in a plurality of HTTP communication characteristic data packets according to the request association information to perform system anomaly analysis according to the target anomaly analysis parameters includes:

[0030] Determine the characteristic data packet with the same session record ID in the HTTP communication characteristic data packet as the target data packet, wherein the target data packet contains a request data packet and a response data packet;

[0031] Calculate the target anomaly analysis parameter based on the packet parameter information of the request packet and the response packet, wherein the target anomaly analysis parameter includes at least one of processing delay, packet sending speed, message interval time, transmission time and message capacity, and session end type;

[0032] The target abnormality analysis parameters are matched based on preset fault analysis rules to obtain system abnormality analysis results.

[0033] In addition, to achieve the above purpose, the present application also proposes an abnormality analysis device, which includes:

[0034] An extraction module, used to extract features from a pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy, to obtain an HTTP communication feature data packet, wherein the communication feature data packet contains request association information;

[0035] The parsing module is used to parse the target abnormality analysis parameters in the HTTP communication characteristic data packets according to the request association information, so as to perform system abnormality analysis according to the target abnormality analysis parameters.

[0036] In addition, to achieve the above objectives, the present application also proposes an anomaly analysis device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the anomaly analysis method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the anomaly analysis method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the abnormality analysis method described above are implemented.

[0039] One or more technical solutions proposed in the present application extract features from a pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy to obtain an HTTP communication feature data packet, wherein the communication feature data packet contains request association information; according to the request association information, target anomaly analysis parameters in several of the HTTP communication feature data packets are parsed to perform system anomaly analysis according to the target anomaly analysis parameters, and feature data in the HTTP network data stream is extracted by the above-mentioned solution. No matter the fault occurs at the request sending end or the request receiving end, or at other important links in the network transmission link, the location of the fault can be accurately located and anomaly analysis can be performed, thereby improving the accuracy and efficiency of fault diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the abnormality analysis method of the present application;

[0043] Figure 2 A schematic diagram of the architecture of an abnormality analysis system provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a flow chart provided for the second embodiment of the abnormality analysis method of the present application;

[0045] Figure 4 A schematic diagram of an overall flow of data packet processing provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of a data packet processing sub-flow provided in an embodiment of the present application;

[0047] Figure 6 A schematic diagram of a data packet processing sub-flow in a scenario where a data packet payload is not empty and a data stream does not exist provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of a processing sub-flow in a scenario where a data packet payload is not empty and a data stream exists provided in an embodiment of the present application;

[0049] Figure 8A schematic diagram of a sub-process for requesting cache area archiving provided in an embodiment of the present application;

[0050] Fig. 9 A flowchart of the third embodiment of the abnormality analysis method of the present application is provided;

[0051] Fig.10 A schematic diagram of a network data flow expiration check and processing flow provided in an embodiment of the present application;

[0052] Fig.11 This is a schematic diagram of the module structure of the abnormality analysis device according to an embodiment of the present application;

[0053] Fig.12 Schematic diagram of the device structure of the hardware operating environment involved in the abnormality analysis method in the embodiment of the present application.

[0054] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0056] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0057] The main solution of the embodiment of the present application is: to capture the HTTP network data stream at important nodes in the network transmission path of inter-service communication in the distributed system, such as four-layer load balancers, Kubernetes Ingress, SDN network entrances and exits, servers where services are located, etc., and use a set of HTTP communication feature extraction strategies to extract a series of feature data from the network data stream, and finally analyze these HTTP feature data to diagnose faults.

[0058] This solution can accurately locate the location of the fault, whether it occurs at the request sending end, an important link in the network transmission link, or at the request receiving end, improving the accuracy and efficiency of fault diagnosis. It effectively makes up for the lack of fault demarcation accuracy of traditional distributed link tracking solutions, which can only locate the communication process between which two services the problem occurs, but cannot locate the link in the message transmission path where the problem occurs.

[0059] At the same time, since this solution does not need to embed specific monitoring code in the application code or use a specific framework like traditional distributed link tracking, it can avoid the problem of increasing the cost of business system development and maintenance, and can also solve the problem of link tracking interruption caused by the inability of some services in the distributed system to access the existing distributed link tracking system due to various reasons such as technology and outsourcing.

[0060] Since modern software widely adopts distributed microservice architecture, a business often needs to cross multiple service nodes to complete. As the dependency between services becomes more complicated, how to effectively track and monitor the execution of requests on each node, and then identify and troubleshoot faults, has become a major challenge in system operation and maintenance. In order to solve this problem, the distributed link tracking system came into being. It is a tool for business processing link tracking in distributed systems.

[0061] Currently, the design concepts and implementation methods of most distributed link tracking systems on the market are based on the Google Dapper paper (Dapper, a Large-Scale Distributed Systems Tracing Infrastructure). This type of system can capture and record the performance data of requests on each service node, and analyze and display this data through a visual interface.

[0062] Traditional distributed link tracking systems still have obvious limitations in terms of fault location accuracy. Although these systems can monitor and locate abnormal call response times between service nodes, they cannot deeply analyze the specific causes of the abnormalities. For example, when service A calls service B, and service B calls service C, if there is a response delay in the process of service B calling service C, the traditional APM system can only indicate that there is a communication network failure between service B and service C. It is unable to identify abnormal write requests from service B, abnormal receive requests from service C, or failures in a certain link in network transmission (four-layer load balancer forwarding, Kubernetes Ingress forwarding, container network transmission, etc.). Traditional distributed link tracking systems lack such sophisticated diagnostic capabilities, so that operation and maintenance personnel still need to manually troubleshoot possible fault points layer by layer. When the transmission path is relatively long, the troubleshooting process is time-consuming and labor-intensive. In particular, for low-probability occasional fault scenarios, the difficulty and time consumption of troubleshooting increase dramatically.

[0063] In addition, traditional distributed link tracking systems usually have high code intrusion. To access these distributed link tracking systems, developers need to embed specific monitoring codes in the application code or use specific frameworks, which greatly increases the complexity of system development and maintenance. When there are some outsourced modules or services in the system, these modules and services are difficult to access, resulting in interruption of link tracking and reduced positioning accuracy.

[0064] The present application provides a solution, which extracts features from a pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy to obtain an HTTP communication feature data packet, wherein the communication feature data packet contains request association information; according to the request association information, a number of target anomaly analysis parameters in the HTTP communication feature data packets are parsed to perform system anomaly analysis according to the target anomaly analysis parameters, and feature data in the HTTP network data stream are extracted by the above-mentioned scheme. No matter the fault occurs at the request sending end or the request receiving end, or at other important links in the network transmission link, the location of the fault can be accurately located and anomaly analysis can be performed, thereby improving the accuracy and efficiency of fault diagnosis.

[0065] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an abnormality analysis system, etc. The following takes the abnormality analysis system as an example to illustrate this embodiment and the following embodiments.

[0066] Based on this, the present application embodiment provides an abnormality analysis method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the abnormality analysis method of the present application.

[0067] In this embodiment, the abnormality analysis method includes steps S1000 to S2000:

[0068] Step S1000: extracting features from a pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy to obtain an HTTP communication feature data packet, wherein the communication feature data packet contains request association information;

[0069] It should be noted that the abnormality analysis method proposed in the embodiment of the present application can work in conjunction with the traditional distributed link tracking system, or it can work independently from the traditional distributed link tracking system. It can be used in conjunction with the traditional distributed link tracking system to enhance and supplement it. When making a diagnosis, the traditional distributed link tracking system can only treat the communication link between the two services as a whole. It can only know that there is a problem with the communication, but it cannot know at which stage of the communication process the problem occurs. In the embodiment of the present application, the key links of the network communication path between services are analyzed by performing point-of-care packet capture, and the time when the HTTP communication characteristic message arrives at the key link is obtained. Information such as the time, and the characteristic data obtained at each key link is comprehensively analyzed to locate the location of the fault.

[0070] Specifically, please refer to Figure 2 , Figure 2 A schematic diagram of the architecture of an abnormality analysis system provided in an embodiment of the present application is provided. Figure 2 The blue-boxed part is the network data flow analysis program, online fault diagnosis service, data query and analysis service, which are the main microservices in the embodiment of this application. The yellow-boxed Kafka, Logstash, and Elastic Search are the middleware services used. Among them, Kafka is used for message peak shaving and valley filling, and data diversion, and can be replaced by other message middleware, such as RocketMQ. Logstash and Elastic Search are used for message transfer and query, and can also be replaced by other data storage middleware, such as the ClickHouse technology stack.

[0071] In addition, it should be noted that the above Figure 2In the system shown, the function of the network data flow analysis program is to filter and capture HTTP network data flows, and use a set of precisely designed HTTP communication feature extraction strategies to extract a series of HTTP communication feature data from the network data flow and send it to Kafka. The network data flow analysis program is deployed on both sides of the boundaries of the key links of the network communication path between services (such as four-layer load balancers, Kubernetes Ingress, container networks, application servers, etc.). The HTTP communication feature data collected by the network data flow analysis program at different locations will contain a request association information, so that when performing subsequent data analysis, the data collected at various locations can be associated for comprehensive analysis. The selection criteria for this request association information is that it is included in each HTTP request of concern, and its value is different in each request. In the embodiment of the present application, the two request headers TraceID (session ID) and SpanID (service ID) in the HTTP request header can be selected as the association information. Of course, in the embodiment of the present application, other features that meet the selection criteria can also be selected.

[0072] The function of the online fault diagnosis service is to subscribe to the HTTP communication feature data generated by the network data flow analysis program from Kafka, and to locate the fault by correlating and comprehensively analyzing the feature data generated by each key link, and to produce network transmission quality-related indicators for each key link of network communication, such as LoadBlancer forwarding time consumption indicator, K8S Ingress forwarding time consumption indicator, container network transmission delay indicator, etc.

[0073] The functions of the data query and analysis service are similar to those of the online fault diagnosis service. Both services diagnose the fault location by correlating and comprehensively analyzing the feature data extracted from each key link by the network data flow analysis program. The difference is that the online fault diagnosis service performs real-time analysis by subscribing to data from Kafka, while the data query and analysis service queries historical data from Elastic Search for offline analysis when the user calls its interface. The data query and analysis service uses the request association information (such as TraceID and SpanID) passed in by the user through the interface parameters to query ElasticSearch and retrieve the feature data extracted from each key link of the network communication path by the network data flow analysis program.

[0074] Specifically, in an embodiment of the present application, the network data flow analysis program will package the characteristic data of an HTTP communication into a message and send it to Kafka. In addition to the characteristic data of this HTTP communication, the message also contains some metadata information, such as the network data flow capture position code, etc.

[0075] More specifically, the network data flow analysis program uses the HTTP communication feature extraction strategy to extract the following HTTP communication feature data from the network data flow:

[0076] First of all, the most important ones are the request association information TraceID and SpanID. The function of these two features is to uniquely identify the request. When doing fault diagnosis, you need to use this information to retrieve the feature message, and use this information to associate and aggregate the feature messages captured at different packet capture locations. Among them, TraceID is used to uniquely identify a business in distributed link tracking, and SpanID is used to represent a service. In the entire request call chain, the request will always carry TraceID to the downstream service. Each service will also generate its own SpanID and pass it to the downstream service together with TraceID. Of course, other features can be selected as replacements according to actual conditions, as long as they can uniquely identify the request.

[0077] In addition, the HTTP communication characteristic data also includes but is not limited to: the time when the first message of the current HTTP communication request data arrives at the packet capture location, the time when the last message of the current HTTP communication request data arrives at the packet capture location, the time when the first message of the current HTTP communication response data arrives at the packet capture location, the time when the last message of the current HTTP communication response data arrives at the packet capture location, the time when the message with the smallest sequence number of the current HTTP communication request data arrives at the packet capture location, the time when the message with the largest sequence number of the current HTTP communication request data arrives at the packet capture location, the time when the message with the smallest sequence number of the current HTTP communication response data arrives at the packet capture location, the time when the message with the largest sequence number of the current HTTP communication response data arrives at the packet capture location, the source IP of the request message, the source port of the request message, the target IP of the request message, the target port of the request message, the sequence number (seq) of the TCP message carrying the HTTP request line of this HTTP communication, that is, the minimum sequence number of the request data packet.

[0078] In addition, the above-mentioned HTTP communication characteristic data also includes the sequence number (seq) of the TCP message carrying the last byte of the request data of this HTTP communication, the maximum sequence number of the request data packet, the payload length of the TCP message with the maximum sequence number of the request data of this HTTP communication, the cumulative value of the length of all request messages of this HTTP communication, the total number of all request messages of this HTTP communication, the maximum value of the time difference between two adjacent request messages in this HTTP communication arriving at the packet capture location, the cumulative value of the time difference between two adjacent request messages in this HTTP communication arriving at the packet capture location, the sequence number (seq) of the request message with the largest time difference between the previous request message and the packet capture location, and the acknowledgment number (seq) of the request message with the largest time difference between the previous request message and the packet capture location. Number), the arrival time of the request message with the largest time difference from the previous request message arriving at the packet capture location, the sequence number (Sequence Number, seq) of the TCP message carrying the HTTP response status line of this HTTP communication, that is, the minimum sequence number of the response data packet, the sequence number (Sequence Number, seq) of the TCP message carrying the last byte of the response data of this HTTP communication, that is, the maximum sequence number of the response data packet, the payload length of the TCP message with the maximum sequence number of the response data of this HTTP communication, the cumulative value of the length of all response messages of this HTTP communication, the total number of all response messages of this HTTP communication, the maximum value of the time difference between two adjacent response messages arriving at the packet capture location in this HTTP communication, the cumulative value of the time difference between two adjacent response messages arriving at the packet capture location in this HTTP communication, the sequence number (Sequence Number, seq) of the response message with the largest time difference from the previous response message arriving at the packet capture location, the acknowledgment number (Acknowledgment Number) of the response message with the largest time difference from the previous response message arriving at the packet capture location Number), the arrival time of the response message with the largest time difference from the previous response message arriving at the capture location, the information of all messages containing HTTP status codes in this HTTP communication (including HTTP status code, TCP message sequence number, TCP message confirmation number, message length, message arrival time, note, in one HTTP communication, there may be multiple response messages containing HTTP status codes at the same time, such as HTTP status code 100 and other status codes may appear at the same time), and the value of the response header content-length.

[0079] In addition, it also includes the method for the requesting direction to end the TCP connection, the sequence number of the TCP message used by the requesting direction to end the connection, the confirmation number of the TCP message used by the requesting direction to end the connection, the number of TCP FIN packets in the requesting direction, the method for the response direction to end the TCP connection, the sequence number of the TCP message used by the response direction to end the connection, the confirmation number of the TCP message used by the response direction to end the connection, the number of TCP FIN packets in the response direction, etc.

[0080] Step S2000: parsing target anomaly analysis parameters in a number of HTTP communication characteristic data packets according to the request association information, so as to perform system anomaly analysis according to the target anomaly analysis parameters.

[0081] It should be noted that the purpose of this step is to use the request association information to parse the target anomaly analysis parameters in the HTTP communication feature data packet. These parameters are crucial for identifying and diagnosing abnormal behavior in the network. By analyzing these parameters, the system can identify problems such as delays, packet loss, and retransmissions in network communications, thereby providing a basis for the system's abnormal analysis. In a specific implementation, the system first groups the data packets according to the session record ID, and then extracts the request and response data packets in each group. The system then calculates the target anomaly analysis parameters such as processing delay and packet sending speed, and matches these parameters with the preset fault analysis rules to determine whether there is an abnormal situation. For example, if the system detects that the processing delay exceeds a preset threshold, it may mark the communication session corresponding to the session record ID as a potential abnormal situation for further analysis and processing.

[0082] The present embodiment provides an exception analysis method, which extracts features from a pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy to obtain an HTTP communication feature data packet, wherein the communication feature data packet contains request association information; according to the request association information, a number of target exception analysis parameters in the HTTP communication feature data packets are parsed to perform system exception analysis according to the target exception analysis parameters. By extracting feature data in the HTTP network data stream through the above scheme, no matter the fault occurs at the request sending end or the request receiving end, or at other important links in the network transmission link, the location of the fault can be accurately located and exception analysis can be performed, thereby improving the accuracy and efficiency of fault diagnosis.

[0083] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 The HTTP network data flow is provided with an HTTP communication session record and a data packet buffer area, and step S1000 may include steps S1100 to S1300:

[0084] Step S1100: extracting a network data packet from the HTTP network data stream to obtain a TCP message based on the network data packet, wherein the network data packet includes HTTP communication characteristic data;

[0085] Step S1200: if the TCP message is a data packet message, then according to a preset data packet processing strategy, the network data packet is stored in the HTTP communication session record or the data packet buffer area;

[0086] Step S1300: if the TCP packet is an end packet or a reset packet, based on the TCP quaternary corresponding to the data packet, the network data packet is stored in the HTTP communication session record;

[0087] Step S1400: When a preset record output condition is met, the HTTP communication characteristic data in the HTTP communication session record is obtained to generate the HTTP communication characteristic data packet based on the communication characteristic data.

[0088] It should be noted that, in the embodiment of the present application, an HTTP communication can be abstracted as a session, and a TCP connection can be abstracted as a flow. Since a TCP connection may be used to transmit multiple HTTP communications, and there may be a time overlap between two adjacent HTTP communications, that is, when the previous HTTP communication response data packet has not been transmitted, a new HTTP request has been issued. Therefore, two HTTP communication session record traces need to be stored in an HTTP network data flow flow, and during the algorithm processing, it may be temporarily impossible to determine whether the current message belongs to the previous trace or the next trace. Therefore, the embodiment of the present application also defines two data packet buffers, which are used to store the request direction and the request direction. It is temporarily impossible to determine whether the current message belongs to the previous trace. ce or the next trace, and the storage response direction cannot temporarily determine whether it belongs to the previous trace or the next trace. In the process of extracting the characteristic data of HTTP communication by the network data flow analysis program, all flows being processed will be stored in a Map collection, and the Key will be set to the sorted TCP four-tuple (that is, source IP, source port, destination IP, destination port). The purpose of sorting is to allow both the request direction message and the response direction message to be stored in one flow. The TCP four-tuple of the request direction message and the TCP four-tuple of the response direction message will get the same result after sorting. Based on the sorted TCP four-tuple, the same Key can be generated, so that the response direction message and the request direction message can be stored in the same flow.

[0089] In addition, it should be noted that the generation of HTTP communication characteristic data packets depends on the processing of network data packets, specifically including obtaining and sorting the TCP four-tuple corresponding to the data packet message, querying whether there is a key-value pair of the HTTP network data stream in the preset double-column set, and storing the network data packet in an appropriate record according to the data packet type of the network data packet and the attribute information of the HTTP network data stream. In addition, when the sequence number and confirmation number of the network data packet are after the sequence number and confirmation number of the existing response data packet, a data packet splicing payload is generated, and the HTTP communication characteristic data therein is identified, which helps to extract key communication features. Specifically, in a feasible implementation method, the system first captures the network data stream, and then extracts the key information such as the sequence number, timestamp, source and destination IP address of the data packet through a specific algorithm. The system will check each TCP message, and if it is a data packet message, it will be stored in the HTTP communication session record or data packet buffer according to the preset strategy. For the end packet message or the reset packet message, the system will store them in the HTTP communication session record based on the TCP four-tuple. When the system detects that the preset output conditions are met, for example, when a session is completed or times out, the system outputs the HTTP communication feature data in the HTTP communication session record and generates HTTP communication feature data packets. These data packets contain rich information, such as TraceID, SpanID, timestamp, etc., which are the basis for fault diagnosis. The system further analyzes these feature data packets to identify and locate abnormal conditions in the network, thereby achieving accurate fault diagnosis.

[0090] More specifically, refer to Figure 4 , Figure 4 A schematic diagram of the overall flow of data packet processing provided for an embodiment of the present application, when a network data packet is captured, the IP header version number of the data packet is first checked to confirm whether the IP message is IPv4 or IPv6, and according to the different version numbers, the IPv4 or IPv6 message is further unpacked to obtain a TCP message, and then it is determined whether the payload length of the TCP message is not 0, that is, whether it is a data packet, if the payload length is not 0, that is, it is a data packet, then the system enters the "data packet processing sub-process" to process the data packet, if the payload length is 0, then it is further determined whether it is a fin packet or a rst packet, and enters the corresponding processing sub-process respectively, and messages that do not meet the above conditions will be regarded as useless data and discarded.

[0091] In a feasible implementation manner, step S1200 may include steps S1210 to S1240:

[0092] Step S1210: If the TCP message is a data packet message, obtain and sort the TCP 4-tuple corresponding to the data packet message to obtain sorted TCP 4-tuples;

[0093] Step S1220: querying whether there is a key-value pair of the HTTP network data flow in a preset double-column set according to the sorted TCP four-tuple;

[0094] Specifically, please refer to Figure 5 , Figure 5 A schematic diagram of a data packet processing sub-flow provided in an embodiment of the present application, when processing a data packet whose Payload is not empty (effective load is not empty), first obtain the quadruple of the data packet, sort the quadruple, and then generate the primary key value Key of the key-value pair from the sorted quadruple, and query the key through the double column set Map storing the flow to see whether the HTTP communication data flow corresponding to the quadruple exists. If not, enter the processing sub-flow for the scenario where the payload is not empty and the communication data flow does not exist for processing. In a feasible factual method, specific reference can be made to Figure 6 , Figure 6 A schematic diagram of a packet processing sub-process in a scenario where a packet payload is not empty and a data flow does not exist is provided for an embodiment of the present application. First, it is determined whether the packet is a packet in the request direction. If it is not a packet in the request direction, the message is discarded and the process is ended. If it is confirmed to be a request packet, a further check is made as to whether the size of the current cache exceeds a set threshold. If the cache size exceeds the threshold, the packet is directly discarded to prevent memory exhaustion. If the cache size is less than the set threshold, a new flow and its HTTP communication session record trace1 are created, and trace1 is initialized with the information of the packet, and the sequence number of the packet is assigned to reqMinSe of the communication session record trace1. q, reqMaxSeq, reqFirstSeq, reqLastSeq, assign the confirmation number of the data packet to reqMinAck and reqMaxAck of trace1, assign the payload of the data packet to reqMinPayload of trace1, assign the payload length of the data packet to reqMaxLen and reqPktLen of trace1, assign the receiving timestamp of the data packet to reqFirstTs and reqLastTs of trace1, assign reqPktCount of trace1 to 1, and store it in the HTTP communication data flow double-column set flow Map with Key as the key and the flow as the value.

[0095] Step S1240: If the key-value pair of the HTTP network data stream exists, the network data packet is stored in the HTTP communication session record or the data packet buffer according to the data packet type of the network data packet and the attribute information of the HTTP network data stream.

[0096] In another possible implementation, reference may be made to Figure 7 , Figure 7 A schematic diagram of a processing sub-process in a scenario where a data packet payload is not empty and a data stream exists is provided for an embodiment of the present application. If the data packet payload is not empty and an HTTP communication data stream exists, the network data packet is stored in the HTTP communication session record or the data packet cache area according to the type of the network data packet and the attribute information of the HTTP network data stream.

[0097] Specifically, first determine whether the data packet is a data packet in the request direction. If it is a data packet in the request direction, enter Figure 7 The request packet processing sub-process shown in the figure, if it is a response packet, it enters Figure 7 The response packet processing sub-process shown in the figure, wherein the core of the request packet processing sub-process is to determine whether the request packet (i.e., the network packet, which has been determined to be a request packet) is stored in trace1, trace2 or the request buffer area. In a feasible implementation, if the request packet meets one of the following conditions, the packet is stored in the HTTP communication session record trace1 through the request packet storage trace sub-process:

[0098] First, trace1 only contains packets in the request direction, but no corresponding packets in the response direction. It should be understood that this means that the communication session in trace1 has not ended, so new request packets should be added to trace1, just like in a conversation, if you only hear the question (request) but no answer (response), then new questions (request packets) should continue in the current conversation (trace1).

[0099] Secondly, trace1 has a response packet, and trace2 is empty, and the seq of the current packet is before the minimum seq of the request packet in trace2. It should be understood that if trace1 already has a response packet, and trace2 is empty, and the sequence number (seq) of the current packet is before the minimum sequence number of the request packet in trace2, this means that the new request packet actually belongs to the previous session (trace1) rather than the new session (trace2). This is like in a conversation, if the answer has been heard, but the new question (request packet) is earlier in time than the start of the next session, then it should be classified as the previous session.

[0100] Third, for the data packet in the response direction of trace1, the seq of the current data packet is less than or equal to the maximum seq in trace1 (Note: this condition does not need to be explicitly judged, because if it is judged that the message does not belong to trace2, it will be stored in the cache area and finally another attribution judgment will be made). It should be understood that if there is a data packet in the response direction in trace1, and the sequence number of the current data packet is less than or equal to the maximum sequence number in trace1, this means that the new request data packet is actually sent in the current session (trace1) rather than a new session. This is like in a conversation, if the new question (request data packet) does not exceed the range of the current session in terms of sequence number, then it should be included in the current session.

[0101] Fourth, trace1 has a data packet in the response direction, trace2 request is empty, and the first few bytes of the current data packet payload are HTTP request command strings (GET, POST, etc.), but the seq of the current packet is not before the ack of the trace1 minimum seq response packet or its ack is after the trace1 response packet minimum seq. It should be understood that if trace1 has a data packet in the response direction, trace2 is empty, and the first few bytes of the current data packet are HTTP request command strings (such as GET or POST), but the sequence number of the current data packet is not before the confirmation number (ack) of the trace1 minimum sequence number response packet, or its confirmation number is after the trace1 response packet minimum sequence number, this means that the new request data packet actually belongs to the current session (trace1), not the new session (trace2), just like in a conversation, if the new question (request data packet) conforms to the context of the current session in terms of content and matches the last answer of the current session in terms of sequence number, then it should be included in the current session.

[0102] In another feasible implementation manner, if the request data packet meets one of the following conditions, the request data packet is stored in the HTTP communication session record trace2 through the trace sub-process:

[0103] First, trace1 has a data packet in the response direction, trace2 request is not empty, and the seq of the current data packet is not before the minimum seq of the request packet in trace2, and the trace2 response is empty. It should be understood that if Trace1 already has a data packet in the response direction, it means that the session in trace1 is nearing the end or has ended. At the same time, trace2 already has a data packet in the request direction, which means that a new session (trace2) has begun. If the sequence number (seq) of the current request data packet is not before the minimum sequence number of the request packet in trace2, it indicates that it is not the first data packet in trace2. In addition, if there is no data packet in the response direction in trace2, then the current request data packet should be part of trace2 because it arrives after the request and before the response of trace2.

[0104] Secondly, trace1 has a response packet, trace2 request is empty, and the first few bytes of the current packet payload are HTTP request command strings (GET, POST, etc.), and the seq of the current packet is not before the ack of the trace1 minimum seq response packet, and its ack is after the trace1 response packet minimum seq. It should be understood that if trace1 already has a response packet, but trace2 does not have a request packet (i.e., trace2 request is empty), this may mean that the trace2 session has not officially started. If the payload (i.e., data content) of the current request packet starts with an HTTP request command string (such as GET or POST), this indicates that it is the beginning of a new request. At the same time, if the sequence number (seq) of this request packet is not before the acknowledgment number (ack) of the trace1 minimum sequence number response packet, and its acknowledgment number (ack) is after the trace1 response packet minimum sequence number, this indicates that this request arrived after the last response of trace1, so it should belong to a new session, i.e., trace2.

[0105] Finally, it should be noted that when the request data packet meets the second condition of storing in the HTTP communication session record trace2, it is necessary to immediately create a new trace2 and initialize it with this data packet. And use this data packet to mark the end of the trace1 request direction. Finally, enter the request buffer archiving sub-process to determine the ownership of the data packet in the buffer. If it is temporarily impossible to determine whether the data packet belongs to trace1 or trace2, enter the request buffer storage sub-process to put the data packet into the request buffer and wait for subsequent processing.

[0106] In addition, it should be noted that the above-mentioned request data packet is stored in the trace sub-process specifically including comparing the data packet's seq, ack, timestamp and other information with the existing data in the trace, and then updating the corresponding fields in the trace. If the current packet seq is greater than the maximum seq in the trace, the maximum packet information is updated. If the second small packet does not exist, the second small packet information is also updated to the current packet information. The current packet seq is equal to the maximum packet seq in the trace. If the current packet payload length is greater than the maximum packet payload length, the maximum packet information is updated; if not, the second small packet information is updated.

[0107] Furthermore, if the current packet seq is smaller than the minimum packet seq in the trace, determine whether the current trace already has a get / post packet. If it does, discard the current packet directly. If it does not exist, replace the minimum packet with the current packet, and replace the second small packet with the minimum packet. The current packet seq is equal to the minimum packet seq in the trace. If the current packet payload length is greater than the minimum packet payload length, replace the minimum packet.

[0108] Finally, if the current packet seq is less than the second packet seq in the trace, the second packet is replaced. If the current packet seq is not less than the second packet seq in the trace, it is further determined whether the current packet seq is equal to the second packet seq in the trace. 若等于且 If the current packet payload length is greater than the second packet payload length, the second packet is replaced.

[0109] In addition, it should be noted that the above request cache area archiving sub-process can refer to Figure 8 , Figure 8 A schematic diagram of a request buffer archiving sub-process provided in an embodiment of the present application, the sub-process will traverse the request buffer and determine whether the data packet belongs to trace1 or trace2 according to whether the data packet sequence number seq is before the minimum seq requested in trace2. If the data packet sequence number is before the minimum seq requested in trace2, the packet is attributed to trace1, if not, the packet is attributed to trace2.

[0110] In addition, it should be noted that the above-mentioned request buffer area is stored in the sub-process. When the request data packet cannot be temporarily determined whether it belongs to trace1 or trace2, the system will enter the sub-process and temporarily cache the network data packet for subsequent processing. This process is designed mainly to deal with the scenario where the captured messages are out of order. Assuming that the out-of-order messages will not deviate too much, the cache area only needs to be set to a certain size.

[0111] More specifically, in another case, if the network data packet is a response data packet, then enter Figure 7 The response data packet processing sub-process shown in the figure has the core of judging whether the response data packet can be placed in trace1 or trace2. If it cannot be judged temporarily, it is stored in the response buffer.

[0112] Specifically, if the response data packet meets one of the following conditions, the response data packet is included in trace1 through the response data packet inclusion trace sub-process:

[0113] First, there is no request packet in trace2. It should be understood that if no request packet has been recorded in trace2, it means that the session represented by trace2 has not yet started. In this case, the current response packet is obviously part of the current session (trace1).

[0114] Secondly, trace2 has a request data packet, and there is a response data packet in trace2, and the seq of this data packet is before the minimum seq of the response packet in trace2. It should be understood that if trace2 already has a request data packet and a response data packet, it indicates that the session represented by trace2 has started and is in progress. If the sequence number (seq) of the current response data packet is before the response packet with the minimum sequence number recorded in trace2, it means that the current response data packet is sent before the response of trace2, so the network data packet should belong to trace1.

[0115] Third, trace2 has a request packet, and there is no response packet in trace2, and the current packet payload does not start with http1. (that is, it is not a packet carrying the HTTP response status line), and the seq of the current packet is before the ack of the packet with the minimum seq requested by trace2. It should be understood that if there is a request packet in trace2 but no response packet yet, and the payload of the current response packet does not start with http1., this usually means that it is not a complete HTTP response. At the same time, if the sequence number (seq) of the current packet is before the acknowledgment number (ack) of the packet with the minimum sequence number requested by trace2, this indicates that it belongs to trace1, because it is sent after the request of trace2 and before the response of trace2. In this case, the packet can also be directly sent to the response buffer area to wait for subsequent processing and judgment.

[0116] Fourth, trace2 has a request data packet, and there is no response data packet in trace2, and the payload of the current data packet starts with http1., and the seq of the current data packet is before the seq of the trace2 request minimum seq packet or the ack of the current data packet is not after the trace2 request packet minimum seq. It should be understood that if there is a request data packet in trace2 but no response data packet, and the payload of the current response data packet starts with http1., it indicates that it is the beginning of an HTTP response. If the sequence number (seq) of the current data packet is before the sequence number (seq) of the trace2 request minimum sequence number packet, or its acknowledgment number (ack) is not after the trace2 request minimum sequence number, this means that the current response data packet belongs to trace1 because it is sent after the trace2 request. In addition, when the response data packet meets this condition, it is also necessary to parse the seq, ack, timestamp, HTTP response code, length and other information of the data packet and fill it into the response list of trace1.

[0117] Specifically, if the response data packet meets one of the following conditions, the response data packet is included in trace2 through the response data packet inclusion trace sub-process:

[0118] First, trace2 has a request data packet, and there is a response data packet in trace2, and the seq of this data packet is not before the minimum seq of the response packet in trace2. It should be understood that if the request data packet has been recorded in trace2, and there is also a response data packet, it indicates that the session represented by trace2 has started and data exchange has occurred. If the sequence number (seq) of the current response data packet is not before the minimum sequence number response packet recorded in trace2, it means that the current response data packet is sent after a response in trace2. Therefore, the current response data packet should be part of trace2.

[0119] Secondly, trace2 has a request data packet, and there is no response data packet in trace2, and the current data packet payload starts with http1., and the ack of the current data packet is after the minimum seq of the trace2 request packet, and the seq of the current data packet is not before the seq of the trace2 request minimum seq packet. It should be understood that if there is a request data packet in trace2 but no response data packet, it means that the trace2 session has started but has not been completed. If the payload of the current response data packet starts with http1., it means that it is the beginning of an HTTP response. If the confirmation number (ack) of the current data packet is after the confirmation number (ack) of the trace2 request minimum sequence number packet, and the sequence number (seq) of the current data packet is not before the sequence number (seq) of the trace2 request minimum sequence number packet, it indicates that the current response data packet is sent after the trace2 request and before the trace2 response. Therefore, the current response data packet should be a part of trace2. In addition, when the response data packet meets this condition, it is also necessary to parse the seq, ack, timestamp, HTTP response code, length and other information of the data packet and fill it into the response list of trace2. At the same time, respMinFindCode of trace2 is set to true. The information of the data packet is used to fill in the response end information of trace1, marking the response end method of trace1. Finally, the response buffer archiving sub-process is entered to determine the attribution of the response packet in the buffer.

[0120] If it is temporarily impossible to determine whether the response data packet should be assigned to trace1 or trace2, the response buffer storage sub-process is entered to temporarily store the data packet in the response buffer and wait for subsequent judgment.

[0121] Specifically, the response buffer storage sub-process includes when the response data packet cannot be determined whether it belongs to trace1 or trace2, it will enter the sub-process to be temporarily cached and wait for subsequent processing. We assume that the out-of-order message will not shift too much, that is, before receiving the minimum seq response packet of trace2, at most only a few response packets of trace2 will be received. Therefore, when the buffer is full, the data packets out of the queue can be directly stored in trace1.

[0122] Specifically, when the response buffer archiving sub-process includes a response data packet in the archiving response buffer, it is necessary to dequeue the data in the buffer in sequence, and then determine whether its seq is before the minimum seq of the response packet of trace2. If the condition is met, the response data packet is stored in the response buffer through the above-mentioned sub-process and the response data packet is stored in trace sub-process to store the relevant information of the data packet in trace1. If the condition is not met, the response data packet is stored in the response buffer through the above-mentioned sub-process and the response data packet is stored in trace sub-process to store the relevant information of the data packet in trace2.

[0123] In addition, in another feasible implementation manner, step S1300: if the TCP packet is an end packet or a reset packet, based on the TCP quaternary corresponding to the data packet, storing the network data packet in the HTTP communication session record may include steps S1310 to S1320:

[0124] Step S1310: if the TCP message is an end packet message, based on the TCP quaternary group corresponding to the data packet message, the network data packet is stored in the HTTP communication session record;

[0125] It should be noted that in the embodiment of the present application, when processing the fin packet, the TCP four-tuple (source IP: source port, destination IP: destination port) of the data packet is first sorted, a Key is generated, and it is recorded whether the four-tuple is the same before and after sorting, so as to facilitate the subsequent comparison with the dirFlag in the flow to determine whether the message is a request direction or a response direction message, and the generated key is used to query whether there is a corresponding flow in the flow map. If not, the data packet is directly discarded.

[0126] If it exists, take out the flow, and perform an XOR operation on the variable that records whether the quadruple is the same before and after sorting and the dirFlag in the flow. If the result is true, record the direction of the data packet as the request direction, otherwise, record it as the response direction.

[0127] If the direction of the data packet is the request direction, then add 1 to the reqFINCount of trace1 and trace2 respectively, and check whether the reqFinishType of trace1 and trace2 is none respectively. If it is none, then record the reqFinishType of the trace as the end of the fin packet, and assign the seq, ack, and arrival timestamp of the fin packet to the reqFinishSeq, reqFinishAck, and reqFinishTs attributes of the trace respectively.

[0128] If the direction of the data packet is not the request direction, then add 1 to the respFINCount of trace1 and trace2 respectively, and check whether the respFinishType of trace1 and trace2 is None. If it is None, the respFinishType of the trace is recorded as the end of the fin packet, and the seq, ack, and arrival timestamp of the fin packet are assigned to the respFinishSeq, respFinishAck, and respFinishTs attributes of the trace respectively.

[0129] Next, determine whether the reqFINCount and respFINCount of trace1 are both greater than 0. If not, end the sub-process. If yes, determine whether there is data in the request buffer and response buffer of the flow. If so, use the above request data packet to store in the trace sub-process and the above response data packet to store in the trace sub-process to store all the data packets in the corresponding buffer into trace1.

[0130] Step S1320: If the TCP packet is a reset packet, based on the TCP quaternary corresponding to the data packet, store the network data packet into the HTTP communication session record.

[0131] It should be noted that in the embodiment of the present application, when processing a Reset (abbreviated as rst) data packet, the TCP four-tuple (source IP: source port, destination IP: destination port) of the data packet is first sorted to generate a key. It is recorded whether the four-tuple is the same before and after sorting, so as to facilitate the subsequent comparison with the dirFlag in the flow to determine whether the message is a request direction or a response direction message. The generated key is used to query whether there is a corresponding flow in the flow map. If not, the data packet is directly discarded.

[0132] If it exists, take out the flow, and perform an XOR operation on the variable that records whether the quadruple is the same before and after sorting and the dirFlag in the flow. If the result is true, record the direction of the data packet as the request direction, otherwise, record it as the response direction.

[0133] If the direction of the data packet is the request direction, check whether the reqFinishType of trace1 and trace2 is None. If it is None, the reqFinishType of the trace is recorded as the end of the rst packet, and the seq, ack, and arrival timestamp of the rst packet are assigned to the reqFinishSeq, reqFinishAck, and reqFinishTs attributes of the trace respectively.

[0134] If the direction of the data packet is not the request direction, check whether the respFinishType attribute of trace1 and trace2 is None. If it is None, record the respFinishType attribute of the trace as the end of the rst packet, and assign the seq, ack, and arrival timestamp of the rst packet to the respFinishSeq, respFinishAck, and respFinishTs attributes of the trace respectively.

[0135] Next, determine whether there is data in the request buffer and response buffer of the flow. If so, use the above request data packet to store in the trace sub-process and the above response data packet to store in the trace sub-process to store all the data packets in the corresponding buffer into trace1.

[0136] The embodiment of the present application provides an abnormality analysis method, which extracts a network data packet from the HTTP network data stream to obtain a TCP message based on the network data packet; if the TCP message is a data packet message, the network data packet is stored in the HTTP communication session record or the data packet buffer according to a preset data packet processing strategy; if the TCP message is an end packet message or a reset packet message, the network data packet is stored in the HTTP communication session record based on the TCP four-tuple corresponding to the data packet message; when the preset record output condition is met, the HTTP communication feature data in the HTTP communication session record is obtained to generate the HTTP communication feature data packet based on the communication feature data, and the specific link of the network fault is non-invasively and efficiently located through the above scheme, thereby significantly improving the accuracy and response speed of fault diagnosis. At the same time, this method avoids the modification of the business system code, reduces the complexity of system development and maintenance, ensures that the services in the distributed system can seamlessly access the link tracking, and can quickly and accurately troubleshoot even in the face of complex network transmission paths and occasional low-probability faults.

[0137] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Fig. 9 The HTTP communication session record includes a first HTTP communication session record and a second HTTP communication session record, and step S1400 of the abnormality analysis method further includes steps S1410 to S1440:

[0138] Step S1410: obtaining the sequence number and confirmation number of the network data packet, and the sequence number and confirmation number of the existing response data packet in the second HTTP communication session record;

[0139] Step S1420: if the sequence number and confirmation number of the network data packet are subsequent to the sequence number and confirmation number of the existing response data packet, generating a data packet concatenation payload based on the target data packet in the first HTTP communication session record;

[0140] Step S1430: Identify and obtain HTTP communication characteristic data in the data packet concatenation payload;

[0141] Step S1440: Generate the HTTP communication characteristic data packet based on the HTTP communication characteristic data and pre-acquired metadata information.

[0142] It should be noted that, in an embodiment of the present application, when the HTTP network communication data flow satisfies that the first HTTP communication session record trace1 has a response data packet, the second HTTP communication session record trace2 has a request data packet, and the seq of the current data packet is not before the minimum seq of the request packet in trace2, and the trace2 response is not empty, and at the same time, the confirmation number ack of the current packet is after the minimum seq of the trace2 response packet, then trace1 can be output through the output sub-process, and trace2 can be assigned to trace1, and then trace2 can be set to empty, and finally, the data packet can be re-entered into the request data packet processing sub-process for processing.

[0143] Specifically, the inclusion of response direction data packets in the first HTTP communication session record race1 means that in the current flow, there is already at least one complete HTTP communication session (trace1), which includes not only request direction data packets but also corresponding response direction data packets.

[0144] The second HTTP communication session record Trace2 contains data packets in the request direction, indicating that the next HTTP communication session (trace2) after trace1 has started and at least data packets in the request direction have been received.

[0145] The sequence number (seq) of the current data packet is not before the minimum sequence number of the request packet in trace2, which means that the sequence number (seq) of the newly captured request data packet is not the one with the smallest sequence number among all the request data packets in trace2. In other words, this new data packet follows the existing request data packet in trace2 in terms of sequence number.

[0146] If the Trace2 response is not empty, it means that trace2 has received at least one response data packet. ack refers to the confirmation number of the data packet, which confirms the received data packet. This sentence means that the confirmation number of the current data packet is after the response packet with the minimum sequence number recorded in Trace2, indicating that the current data packet arrived after a response from Trace2.

[0147] Based on the above conditions, this paragraph describes a situation where the current flow has processed a complete HTTP communication session (trace1) and has started processing the next session (trace2). The newly captured request data packet follows the existing request data packet of trace2 in terms of sequence number. At this time, the output of the first HTTP communication session record trace1 actually outputs all the characteristic data included in a complete HTTP communication.

[0148] Specifically, the trace output sub-process will first determine whether the HTTP request line and response status line have been parsed. If not, they will be parsed first and the parsed information will be stored in the trace object. Next, the payloads of the two data packets with the smallest seq and the second smallest seq in the request will be spliced ​​and merged. Then, for the merged byte array, starting from the first byte, the line breaks are searched in sequence to obtain the request headers (HTTP Request Headers) one by one. It stops after identifying the traceid, spanid, content-length and other information from these request headers, or stops when two consecutive line breaks are encountered. Finally, after obtaining the complete HTTP communication characteristics, the characteristic data of an HTTP communication is packaged into a message and sent to Kafka. In addition to the above-mentioned characteristic data, the message also contains some metadata information, such as the network data flow capture position code, etc.

[0149] In a feasible implementation manner, the abnormality analysis method further includes steps A1000 to A4000:

[0150] Step A1000: traverse the preset double column set;

[0151] Step A2000: determining whether the current silence duration of the current HTTP network data stream in the dual-column set exceeds a preset silence duration threshold;

[0152] Step A3000: If the current silence duration exceeds a preset silence duration threshold, then obtaining a target key-value pair of the current HTTP network data flow in the double-column set;

[0153] Step A4000: Acquire the buffer data in the current HTTP network data stream according to the target key-value pair, and store it in a preset HTTP communication session record.

[0154] It should be noted that in the embodiment of the present application, in addition to the data packet processing flow being triggered when the data packet is captured, there will also be a timer that regularly triggers the flow expiration check and processing flow, so as to clean up those flows in the flow map that have been inactive for a long time (i.e., the current silence duration of the HTTP network data flow exceeds the preset silence duration threshold, where the silence duration threshold can be in minutes or hours). Fig.10 As shown, Fig.10 A schematic diagram of a network data flow expiration check and processing flow is provided for an embodiment of the present application. The system traverses the flow map, which is a storage structure that stores all currently active network sessions. During the traversal process, the system checks the last update time (lastUpdateTime) of each flow to determine whether this time is within 10 minutes before the current time. If so, it means that the flow is still active and the process will continue to traverse. If not, it means that the current silence duration of the HTTP network data flow exceeds the preset silence duration threshold. At this time, the flow has expired, and the process stores the key of the flow in the queue to be deleted, ready for cleanup.

[0155] Next, the system traverses the queue to be deleted to check whether it is completed. If not, the system will use the current key to retrieve the corresponding flow and delete the item corresponding to the key from the flow map. After that, the system will check whether there is data in the request buffer of the flow. If so, the system will use the request data packet to store trace sub-process to store all data packets in the buffer into trace1, and output trace1 and trace2 in the flow to complete the processing of expired flow.

[0156] This process ensures that the system can clean up inactive network sessions in a timely manner to avoid wasting resources, while also maintaining system performance and responsiveness. By placing the flow expiration check and packet processing code in the same thread or coroutine, the system avoids using locks or thread-safe maps, reducing performance overhead.

[0157] It should be noted that both the processing flow and the data packet processing flow will operate the flow map. Therefore, in the embodiment of the present application, the codes of the flow expiration check processing flow and the data packet processing flow are placed in the same thread (or coroutine). When a data packet is captured or the timer times out, the thread (or coroutine) is triggered to run through the signal pipeline, avoiding serious performance degradation problems caused by locking or using a thread-safe map.

[0158] In a feasible implementation manner, the request association information includes a session record ID, and step S2000 may include steps S2100 to S2300:

[0159] Step S2100: determining the characteristic data packet with the same session record ID in the HTTP communication characteristic data packet as the target data packet, wherein the target data packet contains a request data packet and a response data packet;

[0160] Step S2200: Calculate the target abnormality analysis parameter based on the data packet parameter information of the request data packet and the response data packet, wherein the target abnormality analysis parameter includes at least one of processing delay, packet sending speed, message interval time, transmission time and message capacity, and session end type;

[0161] Step S2300: Match the target abnormality analysis parameters based on preset fault analysis rules to obtain system abnormality analysis results.

[0162] It should be noted that in the embodiment of the present application, system fault diagnosis is performed through an online fault diagnosis service or a data query and analysis service, wherein the function of the online fault diagnosis service is to subscribe to the HTTP communication feature data generated by the network data flow analysis program from Kafka, and to associate and comprehensively analyze the feature data generated by each key link point, thereby locating the location of the fault, and producing network transmission quality related indicators for each key link of network communication. The function of the data query and analysis service is similar to that of the online fault diagnosis service, both of which are to associate and comprehensively analyze the feature data extracted from each key link point by the network data flow analysis program, thereby diagnosing the fault location. The difference is that the online fault diagnosis service performs real-time analysis by subscribing to data from Kafka, while the data query and analysis service only queries historical data from Elastic Search for offline analysis when the user calls its interface. The data query and analysis service uses the user to pass in the request association information (such as TraceID, SpanID) through the interface parameters, queries Elastic Search, and retrieves the feature data extracted by the network data flow analysis program from each key link of the network communication path.

[0163] Specifically, no matter which fault diagnosis service is used, its core lies in the in-depth analysis of HTTP communication feature data packets to identify and respond to abnormal situations in network communications. First, the HTTP network data stream is analyzed through the preset communication feature extraction strategy to extract the data packets containing request-related information. Then, using this related information, the system parses out the abnormal analysis parameters, which include key indicators such as processing delay, packet sending speed, message interval time, transmission time and message capacity, and session end type.

[0164] In addition, it should be noted that the system determines the data packets with the same ID as the target data packets based on the session record ID. These data packets contain request and response data packets. Based on the information of these data packets, the system calculates the target anomaly analysis parameters, which are used to identify anomalies in network communications. Finally, the system matches the target anomaly analysis parameters according to the preset fault analysis rules, obtains the system anomaly analysis results, and provides data support for the system anomaly analysis.

[0165] For example, in a feasible implementation, by measuring the time difference between the characteristic data packets of HTTP requests with the same traceID (the first packet of the request, the last packet of the request, the first packet of the response, and the last packet of the response) reaching each capture point, the processing delay of each intermediate link can be clearly known, and thus, whether there is a fault can be determined by the length of the delay.

[0166] Alternatively, the accumulated value of the time difference between adjacent request or response data packets and the total number of data packets can be used to calculate the average interval time between adjacent messages. This indicator can reflect the average packet sending speed of this packet capture point.

[0167] Alternatively, the maximum time difference between adjacent request or response data packets can reflect the fluctuation of the packet sending speed of the packet capturing point by comparing it with the average packet sending speed of the packet capturing point.

[0168] Alternatively, by analyzing the time difference between the last packet and the first packet of a request or response, the total transmission time in the request or response direction can be determined. By comparing the transmission time of each link, the faulty link can also be determined.

[0169] We also analyze the HTTP session end method of the request or response. If the end packet type is an RST packet, it may indicate that an unexpected connection interruption or failure occurred during the request or response process.

[0170] The total bytes of the request or response can be calculated through the maximum seq, minimum seq and maximum seq packet length of the request or response. By comparing it with the accumulated value of the request or response packet length, it can be determined whether there is a packet retransmission, thereby determining the network quality of the intermediate links.

[0171] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the abnormality analysis method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0172] Please refer to Fig.11 The present application also provides an abnormality analysis device, the abnormality analysis device comprising:

[0173] The extraction module 111 is used to extract features from the pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy to obtain an HTTP communication feature data packet, wherein the communication feature data packet contains request association information;

[0174] The parsing module 112 is used to parse the target abnormality analysis parameters in the HTTP communication characteristic data packets according to the request association information, so as to perform system abnormality analysis according to the target abnormality analysis parameters.

[0175] The abnormality analysis device provided by the present application adopts the abnormality analysis method in the above embodiment, which can solve the technical problem of abnormality analysis. Compared with the prior art, the beneficial effects of the abnormality analysis device provided by the present application are the same as the beneficial effects of the abnormality analysis method provided by the above embodiment, and the other technical features in the abnormality analysis device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0176] The present application provides an anomaly analysis device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the anomaly analysis method in the above-mentioned embodiment one.

[0177] Reference below Fig.12 , which shows a schematic diagram of the structure of an abnormality analysis device suitable for implementing the embodiment of the present application. The abnormality analysis device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The abnormality analysis device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0178] like Fig.12As shown, the abnormal analysis device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the abnormal analysis device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the abnormal analysis device to communicate with other devices wirelessly or wired to exchange data. Although the abnormal analysis device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0179] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0180] The abnormality analysis device provided in the present application adopts the abnormality analysis method in the above-mentioned embodiment. Compared with the prior art, the beneficial effects of the abnormality analysis device provided in the present application are the same as the beneficial effects of the abnormality analysis method provided in the above-mentioned embodiment, and other technical features in the abnormality analysis device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0181] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0182] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0183] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the anomaly analysis method in the above-mentioned embodiment.

[0184] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0185] The computer-readable storage medium may be included in the abnormality analysis device; or may exist independently without being assembled into the abnormality analysis device.

[0186] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0188] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0189] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned abnormality analysis method, and can solve the technical problems of abnormality analysis. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the abnormality analysis method provided in the above-mentioned embodiment, and will not be elaborated here.

[0190] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned abnormality analysis method when executed by a processor.

[0191] The computer program product provided by the present application can solve the technical problem of abnormal analysis. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the abnormal analysis method provided by the above embodiment, which will not be repeated here.

[0192] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An abnormality analysis method, characterized in that: The method comprises: Extracting features from a pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy to obtain an HTTP communication feature data packet, wherein the communication feature data packet contains request association information; According to the request association information, target anomaly analysis parameters in the plurality of HTTP communication characteristic data packets are parsed to perform system anomaly analysis according to the target anomaly analysis parameters.

2. The method according to claim 1, characterized in that The HTTP network data stream includes network data packets, HTTP communication session records and data packet buffers. The step of extracting features from the pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy to obtain HTTP communication feature data packets includes: Extracting a network data packet from the HTTP network data stream to obtain a TCP message based on the network data packet, wherein the network data packet includes HTTP communication characteristic data; If the TCP message is a data packet message, then according to a preset data packet processing strategy, the network data packet is stored in the HTTP communication session record or the data packet buffer area; If the TCP message is an end packet message or a reset packet message, based on the TCP quaternary group corresponding to the data packet message, the network data packet is stored in the HTTP communication session record; When a preset record output condition is met, the HTTP communication characteristic data of the network data packet in the HTTP communication session record is output to generate the HTTP communication characteristic data packet based on the communication characteristic data.

3. The method according to claim 2, characterized in that If the TCP message is a data packet message, the step of storing the network data packet in the HTTP communication session record or the data packet buffer according to a preset data packet processing strategy includes: If the TCP message is a data packet message, obtain and sort the TCP four-tuple corresponding to the data packet message to obtain a sorted TCP four-tuple; Querying whether there is a key-value pair of the HTTP network data stream in a preset double-column set according to the sorted TCP four-tuple; If the key-value pair of the HTTP network data stream exists, the network data packet is stored in the HTTP communication session record or the data packet buffer according to the data packet type of the network data packet and the attribute information of the HTTP network data stream.

4. The method according to claim 2, characterized in that The HTTP communication session record includes a first HTTP communication session record and a second HTTP communication session record. When a preset record output condition is met, the step of obtaining HTTP communication feature data in the HTTP communication session record to generate the HTTP communication feature data packet based on the communication feature data includes: Obtaining the sequence number and confirmation number of the network data packet, and the sequence number and confirmation number of the existing response data packet in the second HTTP communication session record; If the sequence number and confirmation number of the network data packet are after the sequence number and confirmation number of the existing response data packet, generating a data packet concatenation payload based on the target data packet in the first HTTP communication session record; Identify and obtain HTTP communication feature data in the data packet concatenation payload; The HTTP communication characteristic data packet is generated based on the HTTP communication characteristic data and pre-acquired metadata information.

5. The method according to claim 1, characterized in that The method further comprises: Traverse the preset double column set; Determine whether the current silence duration of the current HTTP network data stream in the dual-column set exceeds a preset silence duration threshold; If the current silence duration exceeds a preset silence duration threshold, obtaining a target key-value pair of the current HTTP network data flow in the double-column set; The buffer data in the current HTTP network data stream is acquired according to the target key-value pair, and stored in a preset HTTP communication session record.

6. The method according to claim 1, characterized in that The request association information includes a session record ID, and the step of parsing target anomaly analysis parameters in a plurality of HTTP communication characteristic data packets according to the request association information to perform system anomaly analysis according to the target anomaly analysis parameters includes: Determine the characteristic data packet with the same session record ID in the HTTP communication characteristic data packet as the target data packet, wherein the target data packet contains a request data packet and a response data packet; Calculate the target anomaly analysis parameter based on the packet parameter information of the request packet and the response packet, wherein the target anomaly analysis parameter includes at least one of processing delay, packet sending speed, message interval time, transmission time and message capacity, and session end type; The target abnormality analysis parameters are matched based on preset fault analysis rules to obtain system abnormality analysis results.

7. An abnormality analysis device, characterized in that: The device comprises: An extraction module, used to extract features from a pre-acquired HTTP network data stream according to a pre-configured communication feature extraction strategy, to obtain an HTTP communication feature data packet, wherein the communication feature data packet contains request association information; The parsing module is used to parse the target abnormality analysis parameters in the HTTP communication characteristic data packets according to the request association information, so as to perform system abnormality analysis according to the target abnormality analysis parameters.

8. An abnormality analysis device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the abnormality analysis method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the abnormality analysis method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the abnormality analysis method according to any one of claims 1 to 6 are implemented.

Images