Data monitoring method, device and system

Abstract

The invention provides a data monitoring method, device and system, and relates to the technical field of computers. The method comprises the following steps: intercepting a system call event in a kernel space of an operating system to capture a data packet transmitted between specified components through a network; checking the state of the plurality of protocol monitoring switches to determine a currently enabled protocol monitoring switch; identifying the application protocol type of the data packet by using an identification strategy matched with the currently started protocol monitoring switch; analyzing the data packet by using an analyzer matched with the application protocol type to obtain an analysis result; generating structured monitoring data according to an analysis result; and sending the structured monitoring data to the user space. By means of the method, self-adaptive monitoring can be efficiently and accurately conducted on network communication data of multiple application protocols, and the universality and expandability of data monitoring are improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data monitoring method, apparatus and system. Background Technology

[0002] Currently, traditional network traffic monitoring mainly employs network layer monitoring schemes and application layer monitoring schemes.

[0003] In network layer monitoring solutions, specialized network monitoring tools are typically required to capture network data. Common network monitoring tools include packet capture tools using the libpcap library such as Tcpdump and Wireshark, network probe devices, and network switch port mirroring tools such as Switched Port Analyzer (SPAN) and Test Access Point (TAP). This monitoring approach has the following drawbacks: high performance overhead, inability to meet real-time monitoring requirements, and complex deployment.

[0004] Application-layer monitoring solutions typically require modifying the user application's code or introducing proxy servers for each application to capture network data. This approach has the following drawbacks: it is intrusive to user applications, increasing development and maintenance costs; it cannot monitor all network interactions, creating monitoring blind spots; and code instrumentation impacts the performance of user applications.

[0005] To overcome the shortcomings of traditional network traffic monitoring solutions, a monitoring scheme based on the Extended Berkeley Packet Filter (EBPF) has been proposed. This scheme captures network data in kernel space, resulting in low performance overhead and non-intrusiveness to user application code, effectively overcoming the deficiencies of traditional network traffic monitoring solutions. However, EBPF-based monitoring schemes still have the following limitations: they typically only support network traffic monitoring for a single protocol, thus limiting their application to specific scenarios and lacking versatility; supporting network traffic monitoring for new protocols requires recompiling and deploying the entire EBPF application, increasing code development and maintenance costs. Summary of the Invention

[0006] To address the problems existing in network traffic monitoring schemes in related technologies, this disclosure provides a data monitoring method, apparatus, and system.

[0007] According to a first aspect of this disclosure, a data monitoring method is proposed, comprising: intercepting system call events in the kernel space of an operating system to capture data packets transmitted over a network between specified components; checking the status of multiple protocol monitoring switches to determine the currently enabled protocol monitoring switch; identifying the application protocol type of the data packets using an identification strategy matching the currently enabled protocol monitoring switch; parsing the data packets using a parser matching the application protocol type to obtain a parsing result; generating structured monitoring data based on the parsing result; and sending the structured monitoring data to user space.

[0008] In some embodiments, checking the status of multiple protocol monitoring switches to determine the currently enabled protocol monitoring switch includes: traversing the status flag bits of multiple protocol monitoring switches from a first mapping table in the kernel space; and selecting the protocol monitoring switch whose status flag bit has a specified value as the currently enabled protocol monitoring switch.

[0009] In some embodiments, identifying the application protocol type of the data packet using an identification strategy that matches the currently enabled protocol monitoring switch includes: checking whether the data packet contains a feature code of the application protocol corresponding to the currently enabled protocol monitoring switch to obtain a feature code check result, wherein the feature code includes at least one of a transmission port number, a feature string of the application protocol, and a load feature of the application protocol; and determining the application protocol type of the data packet based on the feature code check result.

[0010] In some embodiments, the signature includes a transmission port number, a feature string of the application protocol, and a load characteristic of the application protocol. Determining the application protocol type of the data packet based on the signature check result includes: determining the application protocol type of the data packet as a first score value of the target protocol based on the check result of the transmission port number, wherein the target protocol is each of all application layer protocols corresponding to the currently enabled protocol monitoring switch; determining the application protocol type of the data packet as a second score value of the target protocol based on the check result of the feature string of the application protocol; determining the application protocol type of the data packet as a third score value of the target protocol based on the check result of the load characteristic of the application protocol; performing a weighted summation of the first score value, the second score value, and the third score value to obtain a comprehensive score for the application protocol type of the data packet as the target protocol; and determining the application protocol type of the data packet based on the comprehensive score.

[0011] In some embodiments, the data monitoring method further includes: if the application protocol type of the data packet is not identified using an identification strategy that matches the currently enabled protocol monitoring switch, using a pre-trained machine learning model to identify the application protocol type of the data packet.

[0012] In some embodiments, the parsing result includes the message type of the data packet, and generating structured monitoring data based on the parsing result includes: if the message type of the data packet is a request message, generating a request record based on the data packet and storing the request record in a second mapping table in the kernel space; if the message type of the data packet is a response message, retrieving a request record matching the data packet from the second mapping table; and generating structured monitoring data based on the data packet and the request record matching the data packet.

[0013] In some embodiments, a second mapping table is queried based on the connection identifier of the response message to obtain a request record that matches the data packet. The connection identifier of the response message is a combination of the file descriptor, process identifier, Transport Layer Security (TLS) identifier, and stream identifier of the response message.

[0014] In some embodiments, the plurality of protocol monitoring switches include at least one of the following: a monitoring switch for the Hypertext Transfer Protocol (HTTP), a protocol monitoring switch for relational databases, a monitoring switch for the Remote Procedure Call (RPC) protocol, and a protocol monitoring switch for key-value data storage systems.

[0015] In some embodiments, sending the structured monitoring data to user space includes: determining the kernel version of the operating system; if the kernel version of the operating system belongs to a first version set, sending the structured monitoring data to user space based on a circular buffer mechanism; if the kernel version of the operating system belongs to a second version set, sending the structured monitoring data to user space based on a performance event array mechanism.

[0016] According to a second aspect of this disclosure, a data monitoring apparatus is provided, comprising: a module for performing the data monitoring method as described above.

[0017] According to a third aspect of this disclosure, a data monitoring apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the data monitoring method as described above based on instructions stored in the memory.

[0018] According to a fourth aspect of this disclosure, a data monitoring system is provided, comprising: a data monitoring device as described above; and a user-mode agent device for configuring and managing the data monitoring device and outputting monitoring data from the data monitoring device to a monitoring platform.

[0019] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the data monitoring method as described above.

[0020] According to a sixth aspect of this disclosure, a computer program product is provided having computer program instructions stored thereon, which, when executed by a processor, implement the data monitoring method as described above.

[0021] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0022] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0023] Figure 1 This is a flowchart illustrating a data monitoring method according to some embodiments of the present disclosure;

[0024] Figure 2 This is a flowchart illustrating the protocol identification step in a data monitoring method according to some embodiments of the present disclosure;

[0025] Figure 3 This is a flowchart illustrating the monitoring data generation step in a data monitoring method according to some embodiments of the present disclosure;

[0026] Figure 4 This is a schematic diagram of the structure of a data monitoring device according to some embodiments of the present disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of a data monitoring device according to other embodiments of the present disclosure;

[0028] Figure 6 This is a schematic diagram of the structure of a data monitoring system according to some embodiments of the present disclosure;

[0029] Figure 7 This is a schematic diagram of the structure of a user-mode agent device according to some embodiments of the present disclosure.

[0030] This disclosure can be more clearly understood with reference to the accompanying drawings and the following detailed description. Detailed Implementation

[0031] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0032] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0033] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0034] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0035] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0036] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0037] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0038] To address the shortcomings of existing network traffic monitoring solutions in related technologies, this disclosure provides a data monitoring method, apparatus, and system.

[0039] Figure 1 This is a flowchart illustrating a data monitoring method according to some embodiments of this disclosure. For example... Figure 1 As shown, the data monitoring method includes steps 11 to 16.

[0040] In step 11, system call events are intercepted in the kernel space of the operating system to capture data packets transmitted over the network between specified components.

[0041] Data monitoring methods can be executed by a data monitoring device, which is a device that runs in kernel space. For example, a data monitoring device can be a device built based on the Extended Berkeley Packet Filter (EBPF).

[0042] In some examples, system call events are intercepted using the EBPF kernel tracepoint mechanism. The kernel tracepoint mechanism is a static tracepoint mechanism provided by the Linux kernel, allowing EBPF programs to attach to specific kernel functions. System call events refer to events where user programs request kernel services through interface functions. System call events can include read, write, readv, writev, recv, and send function call events. By intercepting read and write function call events, data transmitted over the network via synchronous I / O operations can be captured; by intercepting readv and writev function call events, data transmitted over the network via vectorized I / O operations can be captured; and by intercepting recv and send function call events, data transmitted over the network via socket-specific operations can be captured. This achieves a comprehensive interception strategy, ensuring complete support for various network I / O modes between components and reducing the possibility of missing any network transmission data. Furthermore, in specific implementations, other tracing mechanisms can also be used to intercept system call events. For example, system call events can be intercepted based on dynamic kernel function hooking mechanisms such as kprobe.

[0043] In this embodiment of the disclosure, by intercepting system call events, interception at the entry point of the system call is achieved, which helps to comprehensively monitor all network data.

[0044] In some embodiments, the data monitoring method further includes filtering the intercepted data according to a set filtering strategy before step 12. This ensures that only data conforming to the filtering strategy enters subsequent processing stages, guaranteeing comprehensive data monitoring while avoiding unnecessary performance overhead. Furthermore, in specific implementations, in addition to filtering the data before step 12, other preprocessing can be performed on the intercepted data.

[0045] In step 12, the status of multiple protocol monitoring switches is checked to determine which protocol monitoring switches are currently enabled.

[0046] In some examples, the status flags of multiple protocol monitoring switches are pre-maintained in a first mapping table in kernel space. This first mapping table is a key-value pair storage structure maintained by the kernel. For example, the first mapping table could be a Least Recently Used (LRU) hash map.

[0047] In these examples, step 12 may include: traversing the status flag bits of multiple protocol monitoring switches from a first mapping table; and designating the protocol monitoring switches with status flag bits of a specified value as the currently enabled protocol monitoring switches. The multiple protocol monitoring switches include at least one of the following: a monitoring switch for the Hypertext Transfer Protocol, a protocol monitoring switch for a relational database, a monitoring switch for a remote procedure call (RPC) protocol, and a protocol monitoring switch for a key-value data storage system.

[0048] For example, the first mapping table sets status flags for monitoring switches of Hypertext Transfer Protocol (HTTP), relational database protocols, Remote Procedure Call (RPC), and key-value data storage systems. Each protocol monitoring switch's status flag can be distinguished as enabled or disabled by different values. For instance, when the status flag for a protocol monitoring switch is set to the first value (i.e., the "specified value" mentioned above), it indicates that the protocol monitoring switch is enabled; when the status flag for a protocol monitoring switch is set to the second value, it indicates that the protocol monitoring switch is disabled. In step 12, assuming that by querying the first mapping table, it is found that the status flag of the HTTP monitoring switch is the first value, the status flag of the relational database protocol monitoring switch is the second value, the status flag of the RPC protocol monitoring switch is the first value, and the status flag of the key-value data storage system protocol monitoring switch is the second value, then it is determined that the currently enabled protocol monitoring switches are the HTTP monitoring switch and the RPC protocol monitoring switch.

[0049] In this embodiment of the disclosure, by executing step 12, when a certain protocol monitoring switch is disabled, the system will skip the identification, parsing and other processing logic related to that protocol, thereby reducing unnecessary protocol identification and parsing processing overhead, which helps to improve the real-time performance of data monitoring and reduce the hardware and software resource overhead required for data monitoring.

[0050] In some embodiments, the data monitoring method further includes: modifying the value of the status flag bit of the protocol monitoring switch in the first mapping table through a user-mode agent device. This allows for real-time control of enabling or disabling specific protocol monitoring switches, achieving hot configuration updates of the protocol monitoring switches. This eliminates the need to restart the system after configuration changes, ensuring the continuity of the data monitoring service and significantly reducing system overhead.

[0051] In step 13, the application protocol type of the data packet is identified using an identification strategy that matches the currently enabled protocol monitoring switch. This identification strategy is based on the signature of known application protocols.

[0052] The protocol identification process can be implemented in several ways. The following examples illustrate how to identify the application protocol type of data packets using two different implementation methods.

[0053] In the first implementation, the specific processing flow for protocol identification is as follows: Figure 2 As shown, steps 131 to 132 are included.

[0054] In step 131, the data packet is checked to see if it contains the feature code of the application protocol corresponding to the currently enabled protocol monitoring switch, so as to obtain the feature code check result. The feature code includes at least one of the following: transmission port number, application protocol feature string, and application protocol payload characteristics. The application protocol payload characteristics can generally be understood as the characteristics possessed by the application layer's business data.

[0055] In some examples, the application protocol's signature includes a characteristic string for the application protocol. In these examples, step 131 includes checking whether the data packet contains a characteristic string for the application protocol corresponding to the currently enabled protocol monitoring switch.

[0056] In practical implementation, multiple protocol feature string matching rules based on regular expressions can be pre-set and used as the identification strategy for matching with multiple protocol monitoring switches. For example, pre-set feature string matching rules for HTTP, application protocol feature strings for relational databases, application protocol feature strings for key-value data storage systems, and RPC protocol feature strings can be used as the identification strategy for matching with multiple protocol monitoring switches. During execution... Figure 1 In the method described, step 12 is used to determine the currently enabled protocol monitoring switch, and the protocol feature string matching rule corresponding to the currently enabled protocol monitoring switch is selected from the preset matching rules. Then, in step 131, based on the protocol feature string matching rule corresponding to the currently enabled protocol monitoring switch, the data packet is checked to see if it matches the above matching rule, thereby obtaining the feature code check result.

[0057] For example, for HTTP, its characteristic strings include the method keyword (such as GET, POST, PUT, etc.) and version identifier (such as HTTP 1.1, HTTP 2.0) in the request line. If the currently enabled protocol monitoring switch includes the HTTP monitoring switch, then the pre-set HTTP characteristic string matching rules can be used to check whether the data packet contains the above characteristic strings in order to determine whether the application protocol type of the data packet is HTTP.

[0058] For example, for a company's self-developed RPC protocol, its characteristic string is the magic code at the beginning of the data packet. The magic code is a specific character sequence present in the data packets of this protocol. If the currently enabled protocol monitoring switch includes the monitoring switch for this RPC protocol, then the pre-set characteristic string matching rules of the RPC protocol can be used to check whether the packet header contains the aforementioned magic code, in order to determine whether the application protocol type of the data packet is the company's self-developed RPC protocol.

[0059] For example, the characteristic strings for relational database application protocols include header-specific length fields, sequence numbers, and command types. If the currently enabled protocol monitoring switch includes the monitoring switch for this relational database application protocol, then the pre-set characteristic string matching rules for this relational database application protocol can be used to check whether the packet header contains length fields, sequence numbers, and command types, in order to determine whether the application protocol type of the packet is a relational database application protocol.

[0060] For example, for application protocols of key-value data storage systems (such as Redis's RESP application protocol), their characteristic string is a unique prefix string. If the currently enabled protocol monitoring switch includes the monitoring switch for the application protocol of this key-value data storage system, then the pre-set characteristic string matching rules for this application protocol can be used to check whether the data packet contains the unique prefix string of this application protocol, thereby determining whether the application protocol type of the data packet is the application protocol of the key-value data storage system.

[0061] In this embodiment of the disclosure, by checking whether the data packet contains the characteristic string of the application protocol, the application protocol type of the data packet can be quickly and accurately determined, which helps to quickly select a suitable parser for protocol parsing and other subsequent processing, thereby improving the efficiency of the data monitoring process.

[0062] In other examples, the application protocol's signature includes a transport port number, a signature string for the application protocol, and a payload signature for the application protocol. In these examples, step 131 includes: checking whether the data packet contains the transport port number of the application protocol corresponding to the currently enabled protocol monitoring switch; checking whether the data packet contains the signature string of the application protocol corresponding to the currently enabled protocol monitoring switch; and checking whether the data packet contains the payload signature of the application protocol corresponding to the currently enabled protocol monitoring switch.

[0063] For example, assuming that the currently enabled protocol monitoring switches include the HTTP monitoring switch and the RPC protocol monitoring switch, then in step 131, it can be checked whether the data packet contains the HTTP port number, the HTTP characteristic string, and the HTTP load characteristics, and whether the data packet contains the RPC protocol port number, the RPC protocol characteristic string, and the RPC protocol load characteristics.

[0064] In step 132, the application protocol type of the data packet is determined based on the signature check result.

[0065] In some examples, the signature check result includes the signature string check result. In these examples, step 132 includes: if the data packet contains the signature string of an application protocol corresponding to a currently enabled protocol monitoring switch, then using that application protocol as the application protocol type of the data packet. If the data packet does not contain the signature string of an application protocol corresponding to any currently enabled protocol monitoring switch, then filtering out the data packet.

[0066] In other examples, the feature check results include the check results for the transport port number, the check results for the feature string of the application protocol, and the check results for the payload characteristics of the application protocol. In these examples, step 132 includes steps A1 through A5.

[0067] Step A1: Based on the check result of the transmission port number, determine the application protocol type of the data packet as the first score value of the target protocol. Here, the target protocol is each of all application layer protocols corresponding to the currently enabled protocol monitoring switch.

[0068] In specific implementation, in step A1, the first score value of the data packet for each application protocol can be determined based on preset scoring rules and the check result of the transmission port number. For example, the scoring rules can be set as follows: when the transmission port number of the data packet belongs to the transmission port number of a certain application protocol, the score is 1 point; when the transmission port number of the data packet does not belong to the transmission port number of a certain application protocol, the score is 0 points. Assuming that the currently enabled protocol monitoring switches are the HTTP monitoring switch and the RPC protocol monitoring switch, and the check result of the transmission port number shows that the transmission port number of the data packet belongs to the HTTP transmission port number, then the first score value of the data packet with the protocol type of HTTP can be set to 1 point, and the first score value of the data packet with the protocol type of RPC can be set to 0 points.

[0069] Step A2: Based on the inspection results of the characteristic string of the application protocol, determine the application protocol type of the data packet as the second score value of the target protocol.

[0070] In specific implementation, in step A2, the second score value of the data packet for each application protocol can be determined based on preset scoring rules and the inspection results of feature strings. For example, the scoring rules can be set as follows: when the data packet has the feature string of a certain application protocol, the score is 1 point; when the data packet does not have the feature string of a certain application protocol, the score is 0 points. Assuming that the currently enabled protocol monitoring switches are the HTTP monitoring switch and the RPC protocol monitoring switch, and the inspection results of the feature strings show that the data packet has the feature string of the RPC protocol, then the second score value of the data packet with the protocol type of HTTP can be set to 0 points, and the second score value of the data packet with the protocol type of RPC can be set to 1 point.

[0071] Step A3: Based on the inspection results of the application protocol's load characteristics, determine the application protocol type of the data packet as the third score value of the target protocol.

[0072] In specific implementation, in step A3, the third score value of each application protocol can be determined based on preset scoring rules and the inspection results of the application protocol's load characteristics. For example, the scoring rules can be set as follows: when a data packet possesses the load characteristics of a certain application protocol, the score is 1 point; when a data packet does not possess the load characteristics of a certain application protocol, the score is 0 points. Assuming that the currently enabled protocol monitoring switches are the HTTP monitoring switch and the RPC protocol monitoring switch, and the inspection results of the feature string indicate that the data packet possesses the load characteristics of the RPC protocol, then the third score value of the data packet with the protocol type of HTTP can be set to 0 points, and the third score value of the data packet with the protocol type of RPC can be set to 1 point.

[0073] Step A4: The first score, the second score, and the third score are weighted and summed to obtain a comprehensive score for the application protocol type of the data packet as the target protocol.

[0074] After obtaining the first to third ratings, the three ratings are weighted and summed according to their respective weights to obtain a comprehensive score. The weights for each rating from the first to the third can be set based on experience or through machine learning.

[0075] For example, assuming the currently enabled protocol monitoring switches are the HTTP monitoring switch and the RPC protocol monitoring switch, the first to third score values ​​for HTTP data packets are 1, 0, and 0 respectively, and the first to third score values ​​for RPC protocol data packets are 0, 1, and 1 respectively, and the weights corresponding to the first to third score values ​​are 0.3, 0.5, and 0.2 respectively, then the overall score for HTTP data packets is 0.3, and the overall score for RPC protocol data packets is 0.7.

[0076] Step A5: Determine the application protocol type of the data packet based on the comprehensive score.

[0077] In some embodiments, in step A5, the application protocol with the highest overall score among the target protocols is taken as the application protocol type of the data packet.

[0078] In this embodiment of the disclosure, on the one hand, by comprehensively considering multiple application protocol features such as the transmission port number and protocol feature string of the data packet, the application protocol type to which the data packet belongs can be determined more accurately than considering only a single application protocol feature; on the other hand, by scoring the inspection results of the data packet in different protocol feature dimensions and by weighted summing of the scores, the importance of different protocol feature dimensions in determining the application protocol type of the data packet can be better measured and distinguished, thereby further improving the accuracy of the determined application protocol type.

[0079] In the second implementation, the protocol identification includes, in addition to, Figure 2 In addition to the illustrated process, the procedure also includes: if the application protocol type of the data packet cannot be identified using the identification strategy matched with the currently enabled protocol monitoring switch, then using a pre-trained machine learning model to identify the application protocol type of the data packet. The machine learning model can be of various types. For example, it can be a decision tree classifier, random forest, support vector machine, deep neural network model, or ensemble voting system, etc.

[0080] For example, assuming the currently enabled protocol monitoring switches are the HTTP monitoring switch and the RPC protocol monitoring switch, and the application protocol type of the data packet is not identified through step 13, the application protocol type of the data packet is predicted using a pre-trained machine learning model.

[0081] In this embodiment of the disclosure, on the one hand, by combining the identification strategy based on known application protocol signatures to determine the protocol type of data packets with the prediction of the protocol type of data packets based on machine learning models, the system's ability to identify the protocol type of data packets can be improved. On the other hand, by first determining the protocol type of data packets based on the identification strategy, and then determining the application protocol type of data packets based on machine learning models when the identification strategy cannot identify the application protocol type of data packets, the efficiency of data packet protocol type identification can be further improved compared to directly using machine learning models to determine the application protocol type of data packets, thereby helping to improve the overall efficiency of the data monitoring process.

[0082] Back Figure 1 In step 14, the data packets are parsed using a parser that matches the application protocol type to obtain the parsing result.

[0083] For example, if the application protocol type of the data packet is determined to be HTTP, an HTTP parser is used to parse the data packet to obtain the parsing result. If the application protocol type of the data packet is determined to be RPC, an RPC protocol parser is used to parse the data packet to obtain the parsing result.

[0084] In some examples, parsing packets using a parser may include determining the message type to which the packet belongs and extracting key information from the packet (such as request timestamps, request content summaries, connection information, etc.).

[0085] In step 15, structured monitoring data is generated based on the parsing results.

[0086] In some examples, the processing flow for generating monitoring data is as follows: Figure 3 As shown, steps 151 to 153 are included.

[0087] In step 151, if the message type of the data packet is a request message, a request record is generated based on the data packet and stored in the second mapping table in the kernel space.

[0088] The second mapping table is a key-value pair storage structure maintained by the kernel. For example, the second mapping table can be a Least Recently Used (LRU) hash map. The second mapping table can be the same as the first mapping table mentioned above, or it can be a different mapping table.

[0089] In some examples, step 151 includes: when the data monitoring device detects an outbound data packet and the message type of the data packet is a request message, extracting the request timestamp, request content summary, connection information, etc., from the data packet, and storing this information in a second mapping table as a request record to provide a basis for subsequent response matching. The connection information may include one or more of the following: TCP connection four-tuple information, file descriptor, process identifier and stream identifier, and Transport Layer Security (TLS) identifier. The Transmission Control Protocol (TCP) connection four-tuple information includes the source IP address, destination IP address, source port, and destination port. The TCP connection four-tuple information uniquely identifies a TCP connection. A file descriptor is an integer identifier used by the operating system to identify resources such as open files or network connections. A process identifier is a unique non-negative integer (PID) assigned to each process by the operating system to distinguish different processes. A stream identifier is a number or marker used to uniquely identify a data stream in network communication. The TLS identifier includes TLS session parameters.

[0090] In step 152, if the message type of the data packet is a response message, a request record matching the data packet is retrieved from the second mapping table.

[0091] In some examples, step 152 includes: if the message type of the data packet is a response message, querying a second mapping table based on the file descriptor of the data packet to obtain a request record that matches the data packet.

[0092] File descriptors are the key link in matching requests and responses. To facilitate request-response matching based on file descriptors, the data monitoring device maintains a mapping relationship between file descriptors and TCP connections. When a response packet is detected, the corresponding TCP connection information can be quickly located, and the file descriptor can be quickly determined based on the TCP connection information. Then, the request matching the response can be found based on the file descriptor. In this way, through the associated management of file descriptors, scattered network I / O operations can be organized into complete request-response pairs, providing a foundation for subsequent performance analysis.

[0093] In other examples, step 152 includes: when the data monitoring device detects an inbound data packet and the message type of the data packet is a response message, querying a second mapping table based on the connection identifier of the response message to obtain a request record matching the data packet. The connection identifier of the response message is a combination of the file descriptor, process identifier, TLS identifier, and stream identifier of the response message.

[0094] In this embodiment of the disclosure, by establishing a second mapping table and using a combination of file descriptor, process ID, TLS identifier and stream ID as the unique key value of the request record, this design can support a variety of complex scenarios such as concurrent processing of multiple requests on a single connection, differentiated processing of TLS encrypted connections, and stream-level matching of multiplexing protocols such as HTTP / 2, which facilitates more accurate matching between request messages and response messages in the future.

[0095] In some embodiments, the data monitoring method further includes: monitoring the state of TCP connections through a kernel tracepoint mechanism, and maintaining a connection state table based on the TCP connection state monitoring results. The connection state table records all state transitions of a TCP connection from establishment to closure, including: 1. State transitions during normal connection establishment (e.g., from entering the SYN_SENT state to entering the ESTABLISHED state); 2. Connection state maintenance during data transmission; 3. Various state transitions during connection closure. By comprehensively tracking and maintaining the TCP connection state, accurate connection context information is provided for subsequent request-response matching and monitoring data generation, thereby improving the accuracy of data monitoring.

[0096] In step 153, structured monitoring data is generated based on the data packets and the request records that match the data packets.

[0097] In some examples, after matching response messages with request messages, the data monitoring device calculates performance metrics such as response time and data transfer volume based on the matched request-response pairs, and detects anomalies such as network timeouts and connection resets. Then, the performance metrics, anomaly detection results, and response status codes extracted from the response messages are processed to obtain structured monitoring data. For example, the performance metrics, anomaly detection results, and response status codes extracted from the response messages are converted into structured data in JavaScript Object Notation (JSON) format.

[0098] In some embodiments, the data monitoring method further includes: cleaning up completed or timed-out request records in the second mapping table according to a set cleanup strategy to prevent memory leaks. The cleanup strategy may include a time-based automatic cleanup strategy and a connection-state-based proactive cleanup strategy.

[0099] Back Figure 1 In step 16, the structured monitoring data is sent to the user space.

[0100] In some examples, considering the differences in EBPF feature support across different operating system kernel versions, an adaptive data transfer mechanism is used to send monitoring data to user space. Specifically, in these examples, sending structured monitoring data to user space includes: determining the operating system kernel version; if the operating system kernel version belongs to the first set, sending the structured monitoring data to user space based on a circular buffer mechanism; if the operating system kernel version belongs to the second set, sending the structured monitoring data to user space based on a performance event array mechanism.

[0101] For example, operating systems with kernel version 5.8 and above use a ring buffer (e.g., RingBuffer) mechanism for data transfer. Ring Buffer is a newer eBPF feature that offers higher performance and better memory management, supporting bulk data transfer and event notification mechanisms. For operating systems with kernel versions below 5.8, a performance event array (e.g., Perf Event Array) mechanism is used for data transfer. While the performance of the Perf Event Array mechanism is slightly lower than that of the Ring Buffer mechanism, it has better compatibility and can run stably on most operating system distributions.

[0102] In this embodiment, by directly reading data from an efficient cached data structure (such as RingBuffer or Perf Event Array) in the kernel buffer and transferring the read data to user space, unnecessary memory copy operations can be avoided, reducing the resource overhead required for data transfer. Furthermore, by employing an adaptive data transfer mechanism based on the operating system kernel version, the versatility of the data monitoring method is improved. In addition, in specific implementations, the security of memory access can be ensured during data extraction using an EBPF verifier. Moreover, during data extraction, for excessively large data packets, only critical protocol header information can be extracted based on an intelligent data truncation mechanism. This ensures both the effectiveness of data monitoring and control over memory usage.

[0103] In this embodiment, the above process enables efficient and accurate adaptive monitoring of network communication data across multiple application protocols, improving the versatility and scalability of data monitoring. This not only solves the problems of high performance overhead and code intrusion into user applications inherent in related network traffic monitoring solutions, but also addresses the issues of these solutions supporting only a single protocol, limiting monitoring to specific scenarios, and lacking versatility and scalability.

[0104] Figure 4 This is a schematic diagram of the structure of a data monitoring device according to some embodiments of this disclosure. Figure 4 As shown, the data monitoring device 40 includes a capture module 41, a determination module 42, an identification module 43, a parsing module 44, a generation module 45, and a sending module 46.

[0105] The capture module 41 is configured to intercept system call events in the kernel space of the operating system to capture data packets transmitted over the network between specified components.

[0106] Module 42 is configured to check the status of multiple protocol monitoring switches to determine the currently enabled protocol monitoring switch.

[0107] The identification module 43 is configured to identify the application protocol type of data packets using an identification strategy that matches the currently enabled protocol monitoring switch.

[0108] Parsing module 44 is configured to parse data packets using a parser that matches the application protocol type to obtain the parsing result.

[0109] The generation module 45 is configured to generate structured monitoring data based on the parsing results.

[0110] Sending module 46 is configured to send structured monitoring data to user space.

[0111] For details on how the capture module 41, determination module 42, identification module 43, parsing module 44, generation module 45, and sending module 46 perform their respective functions, please refer to the relevant content in the data monitoring method described above.

[0112] In some embodiments, the data monitoring device 40 further includes modules for performing other steps of the data monitoring method as described above.

[0113] In the embodiments of this disclosure, the above-mentioned device can efficiently and accurately perform adaptive monitoring of network communication data of various application protocols, thereby improving the versatility and scalability of data monitoring.

[0114] Figure 5 This is a structural schematic diagram of a data monitoring device 50 according to other embodiments of this disclosure. For example... Figure 5 As shown, the data monitoring device 50 includes a memory 51 and a processor 52 coupled to the memory 51. The memory 51 is used to store instructions for executing embodiments of the data monitoring method. The processor 52 is configured to execute the data monitoring method in any of the embodiments of this disclosure based on the instructions stored in the memory 51.

[0115] Figure 6This is a schematic diagram of the structure of a data monitoring system according to some embodiments of this disclosure. For example... Figure 6 As shown, the data monitoring system 60 includes a data monitoring device 40 and a user-mode agent device 61.

[0116] The data monitoring device 40 is used to perform the data monitoring method as described above.

[0117] User-mode agent device 61 is used to configure and manage the data monitoring device and output monitoring data from the data monitoring device to the monitoring platform.

[0118] In some examples, the user-space agent device 61 includes a configuration management module and a data reporting module. The configuration management module can be used to implement dynamic configuration management functions, including at least one of the following: real-time control of protocol monitoring switches, dynamic adjustment of load balancing strategies, and maintenance of process-application mapping relationships. This module communicates in real-time with the kernel-space data monitoring device through an EBPF mapping table to achieve hot configuration updates without restarting the monitoring system.

[0119] The data reporting module can aggregate and calculate monitoring indicators based on the monitoring data obtained from the data monitoring device, and push the obtained monitoring data or the data obtained through aggregation and calculation to the monitoring platform.

[0120] In this embodiment of the disclosure, the above system can not only efficiently and accurately perform adaptive monitoring of network communication data of various application protocols, improving the universality and scalability of data monitoring, but also realize dynamic control of protocol monitoring switches during runtime, significantly reducing system overhead.

[0121] Figure 7 This is a schematic diagram of the structure of a user-mode agent device according to some embodiments of this disclosure. For example... Figure 7 As shown, the user-mode agent device 61 includes a protocol parsing module 611, a data reporting module 612, and a configuration management module 613.

[0122] The protocol parsing module 611 receives monitoring data from the data monitoring device and parses it according to the protocol type of the monitoring data. This module supports parallel processing of multiple protocols, including HTTP, a self-developed RPC protocol, MySQL database protocols, and Redis protocols. Each protocol has an independent parser to ensure parsing efficiency and accuracy.

[0123] The data reporting module 612 can perform aggregate calculations of monitoring indicators based on the monitoring data obtained from the data monitoring device, and push the obtained monitoring data or the data obtained through aggregate calculation to the monitoring platform.

[0124] The configuration management module 613 can be used to implement dynamic configuration management functions, including at least one of the following: real-time control of protocol monitoring switches, dynamic adjustment of load balancing strategies, and maintenance of process-application mapping relationships. This module communicates in real-time with the kernel-mode data monitoring device through an EBPF mapping table to achieve hot configuration updates without requiring a restart of the monitoring system.

[0125] In this embodiment of the disclosure, by combining the above-mentioned user-mode proxy device with the data monitoring device, it is possible not only to efficiently and accurately perform adaptive monitoring of network communication data of various application protocols, thereby improving the versatility and scalability of data monitoring, but also to realize dynamic control of the protocol monitoring switch during runtime, thereby significantly reducing system overhead.

[0126] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0128] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0129] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0130] The data monitoring methods, apparatus, and systems according to this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

Claims

1. A data monitoring method, comprising: Intercept system call events in the operating system's kernel space to capture data packets transmitted over the network between specified components; Check the status of multiple protocol monitoring switches to determine which protocol monitoring switches are currently enabled; The application protocol type of the data packet is identified using an identification strategy that matches the currently enabled protocol monitoring switch. The data packet is parsed using a parser that matches the application protocol type to obtain the parsing result; Based on the analysis results, structured monitoring data is generated; The structured monitoring data is sent to the user space.

2. The data monitoring method according to claim 1, wherein, The step of checking the status of multiple protocol monitoring switches to determine the currently enabled protocol monitoring switches includes: From the first mapping table in the kernel space, traverse the status flag bits of multiple protocol monitoring switches; The protocol monitoring switch with the status flag bit set to a specified value is used as the currently enabled protocol monitoring switch.

3. The data monitoring method according to claim 1, wherein, Using an identification strategy that matches the currently enabled protocol monitoring switch, the application protocol type of the data packet is identified, including: Check whether the data packet contains the feature code of the application protocol corresponding to the currently enabled protocol monitoring switch to obtain the feature code check result, wherein the feature code includes at least one of the following: transmission port number, feature string of application protocol, and load feature of application protocol; Based on the signature check results, the application protocol type of the data packet is determined.

4. The data monitoring method according to claim 3, wherein, The signature includes the transmission port number, the characteristic string of the application protocol, and the payload characteristics of the application protocol. Determining the application protocol type of the data packet based on the signature check result includes: Based on the inspection result of the transmission port number, the application protocol type of the data packet is determined as the first score value of the target protocol, wherein the target protocol is each of the application protocols among all application layer protocols corresponding to the currently enabled protocol monitoring switch. Based on the inspection result of the characteristic string of the application protocol, the application protocol type of the data packet is determined to be the second score value of the target protocol; Based on the inspection results of the load characteristics of the application protocol, the application protocol type of the data packet is determined as the third score value of the target protocol; The first score, the second score, and the third score are weighted and summed to obtain a comprehensive score for the application protocol type of the data packet as the target protocol. The application protocol type of the data packet is determined based on the comprehensive score.

5. The data monitoring method according to claim 1 further includes: If the application protocol type of the data packet is not identified using an identification strategy that matches the currently enabled protocol monitoring switch, the application protocol type of the data packet is identified using a pre-trained machine learning model.

6. The data monitoring method according to claim 1, wherein, The parsing result includes the message type of the data packet, and the generation of structured monitoring data based on the parsing result includes: If the message type of the data packet is a request message, a request record is generated based on the data packet, and the request record is stored in a second mapping table in the kernel space; If the message type of the data packet is a response message, a request record matching the data packet is retrieved from the second mapping table; based on the data packet and the request record matching the data packet, structured monitoring data is generated.

7. The data monitoring method according to claim 6, wherein, The second mapping table is queried based on the connection identifier of the response message to obtain the request record that matches the data packet. The connection identifier of the response message is a combination of the file descriptor, process identifier, Transport Layer Security (TLS) identifier, and stream identifier of the response message.

8. The data monitoring method according to any one of claims 1 to 7, wherein, The plurality of protocol monitoring switches include at least one of the following: a monitoring switch for the Hypertext Transfer Protocol (HTTP), a protocol monitoring switch for relational databases, a monitoring switch for the Remote Procedure Call (RPC) protocol, and a protocol monitoring switch for key-value data storage systems.

9. The data monitoring method according to any one of claims 1 to 7, wherein, Sending the structured monitoring data to the user space includes: Determine the kernel version of the operating system; If the kernel version of the operating system belongs to the first version set, the structured monitoring data is sent to user space based on the ring buffer mechanism. If the kernel version of the operating system belongs to the second version set, the structured monitoring data is sent to user space based on the performance event array mechanism.

10. A data monitoring device, comprising: A module for performing the data monitoring method as described in any one of claims 1 to 9.

11. A data monitoring device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the data monitoring method as described in any one of claims 1 to 9 based on instructions stored in the memory.

12. A data monitoring system, comprising: The data monitoring device as described in claim 10 or 11; The user-mode agent device is used to configure and manage the data monitoring device and output monitoring data from the data monitoring device to the monitoring platform.

13. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the data monitoring method as described in any one of claims 1 to 9.

14. A computer program product having stored computer program instructions thereon, which, when executed by a processor, implement the data monitoring method as described in any one of claims 1 to 9.

Images