Application observable data processing method and device, computer equipment and storage medium

Abstract

The application relates to an application observable data processing method and device, computer equipment and a storage medium, wherein the application observable data processing method comprises the following steps: determining state information corresponding to a plurality of associated application programs, the state information being used for indicating the request state of each application program; based on the state information, obtaining network data information generated by the application program in a network communication process, the network data information comprising network data packets; preprocessing the network data packets to obtain corresponding link information; and further converting the link information protocol into corresponding first target protocol packets. Through the application, the problem that link concatenation needs to rely on additional process information, resulting in a too complex data processing process, is solved, link concatenation is realized without relying on additional process information, and the complexity of data processing is reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to methods, apparatuses, computer equipment, and storage media for processing observable data. Background Technology

[0002] With the application of cloud computing, microservices, and distributed technologies in the security field, the call layers and relationships between security business software systems are becoming increasingly complex. When a security business software system malfunctions, given the complexity of the system, observable data from the application, such as link data, logs, and metrics, is typically used to analyze the fault location and root cause to achieve timely and effective early warning.

[0003] Existing data processing methods associate collected network packets with process metadata, and then process the associated network packets to query data. However, the link data obtained by the above methods is single-link data, which requires additional process information to connect the links, making the data processing process overly complex.

[0004] There is currently no effective solution to the problem that related technologies require additional process information to connect links, which makes the data processing process too complicated. Summary of the Invention

[0005] This embodiment provides an application observable data processing method, apparatus, computer device, and storage medium to solve the problem in related technologies that require additional process information to achieve link concatenation, resulting in an overly complex data processing process.

[0006] Firstly, this embodiment provides an application observable data processing method, the method comprising:

[0007] Determine the status information corresponding to multiple associated applications; the status information is used to indicate the request status of each application;

[0008] Based on the status information, network data information generated by the application during network communication is obtained; the network data information includes network data packets.

[0009] The network data packets are preprocessed to obtain the corresponding link information;

[0010] The link information protocol is converted into the corresponding first target protocol message.

[0011] In some embodiments, the process of obtaining network data information generated by the application during network communication is based on the state information; the network data information includes network data packets, including:

[0012] When the status information indicates that a request was initiated by the source application in each of the applications, the network data packets generated by the application during network communication are stored in the kernel-mode cache module;

[0013] The network data packets in the cache module are mapped and collected to the user space.

[0014] In some embodiments, the process of obtaining network data information generated by the application during network communication is based on the state information; the network data information includes network data packets, including:

[0015] When the status information indicates that a lower-level application in each of the applications is waiting to receive a request, network data packets generated by the application during network communication are collected.

[0016] The network data packets are associated with the corresponding application process information and stored in the kernel-mode cache unit;

[0017] The network data packets associated with the cache unit are mapped and collected to the user space.

[0018] In some embodiments, after determining the status information corresponding to the multiple associated applications, the method further includes:

[0019] When the status information indicates a request to be processed by a lower-level application within each of the applications, network data information generated by the application during network communication is obtained; the network data information includes log data.

[0020] The log data is associated with the corresponding link information and stored in the kernel-mode cache unit;

[0021] The log data associated with the cache unit is mapped and collected to the user space;

[0022] The parsed log data is then subjected to protocol conversion to obtain the corresponding second target protocol message.

[0023] In some embodiments, the process of obtaining network data information generated by the application during network communication is based on the state information; the network data information includes network data packets, including:

[0024] When the status information indicates that a request is initiated by a lower-level application within each of the applications, the corresponding link information is determined based on the network data packets generated by the application during network communication.

[0025] When a target network data packet with the same link information is detected, the header field information of the target network data packet is modified according to the currently generated random number;

[0026] The modified target network data packet is stored in a kernel-mode cache unit;

[0027] The target network data packets in the cache unit are mapped and collected to user space.

[0028] In some embodiments, the preprocessing of the network data packets to obtain corresponding link information includes:

[0029] Filter the network data packets;

[0030] Session aggregation is performed on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session;

[0031] The aggregated network data packets are parsed to obtain link information.

[0032] In some embodiments, the filtered network packets are subjected to session aggregation; the network packets include request packets and response packets within the same communication session, including:

[0033] Determine whether the network data packet contains a flow protocol tag;

[0034] When the flow protocol marker is detected in the network data packet, the application layer protocol information corresponding to the network data packet is obtained;

[0035] Session aggregation is performed on network data packets that have the same application layer protocol information; the network data packets include request data packets and response data packets.

[0036] In some embodiments, after determining the status information corresponding to multiple associated applications, the method further includes:

[0037] Acquire network data information generated by the application during network communication; the network data information includes indicator data.

[0038] The indicator data is analyzed;

[0039] The parsed and processed indicator data is then converted into a protocol to obtain the corresponding third target protocol message.

[0040] Secondly, this embodiment provides an application observable data processing device, which includes: a detection module, an acquisition module, a preprocessing module, and a conversion module;

[0041] The detection module is used to determine the status information corresponding to multiple associated applications; the status information is used to indicate the request status of each application.

[0042] The acquisition module is used to acquire network data information generated by the application during network communication based on the status information; the network data information includes network data packets;

[0043] The preprocessing module is used to preprocess the network data packets to obtain the corresponding link information;

[0044] The conversion module is used to convert the link information protocol into the corresponding first target protocol message.

[0045] Thirdly, this embodiment provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the application observable data processing method described in the first aspect above.

[0046] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the application observable data processing method described in the first aspect above.

[0047] Compared with related technologies, the application observable data processing method, apparatus, computer equipment, and storage medium provided in this embodiment determine the status information corresponding to multiple associated applications, which is used to indicate the request status of each application; based on the status information, obtain network data information generated by the applications during network communication, including network data packets; preprocess the network data packets to obtain the corresponding link information; further, convert the link information protocol into the corresponding first target protocol message, which solves the problem that the data processing process is too complicated due to the need to rely on additional process information to realize link concatenation, and realizes that link concatenation can be performed without relying on additional process information, thereby reducing the complexity of data processing.

[0048] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1This is a hardware structure block diagram of a terminal device for applying an observable data processing method according to an embodiment of this application;

[0051] Figure 2 This is a flowchart of an application observable data processing method provided in an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of the system design architecture provided in one embodiment of this application;

[0053] Figure 4 This is a schematic flowchart of an application observable data processing method provided in an embodiment of this application;

[0054] Figure 5 This is a flowchart illustrating an application observable data processing method provided in another embodiment of this application;

[0055] Figure 6 This is a schematic flowchart of a log data collection method provided in an embodiment of this application;

[0056] Figure 7 This is a schematic diagram of the data processing flow when a lower-level application initiates a request, provided in an embodiment of this application.

[0057] Figure 8 This is a schematic diagram of an aggregation process provided in an embodiment of this application;

[0058] Figure 9 This is a flowchart of an application observable data processing method provided in a preferred embodiment of this application;

[0059] Figure 10 This is a structural block diagram of an application observable data processing device provided in an embodiment of this application.

[0060] In the diagram: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 10, detection module; 20, acquisition module; 30, preprocessing module; 40, conversion module. Detailed Implementation

[0061] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0062] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0063] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal applying the observable data processing method in this embodiment. For example... Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0064] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the observable data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0065] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0066] This embodiment provides a method for processing observable data. Figure 2 This is a flowchart of the application observable data processing method in this embodiment, such as... Figure 2 As shown, the process includes the following steps:

[0067] Step S210: Determine the status information corresponding to multiple associated applications; the status information is used to indicate the request status of each application.

[0068] Step S220: Based on the status information, obtain network data information generated by the application during network communication; the network data information includes network data packets;

[0069] Step S230: Preprocess the network data packets to obtain the corresponding link information;

[0070] Step S240: Convert the link information protocol into the corresponding first target protocol message.

[0071] In this embodiment, please refer to Figure 3The system architecture is divided into kernel mode and user mode. Kernel mode includes a data acquisition module, a caching module, a correlation management module, a log acquisition module, and a first control module. User mode includes a first preprocessing module, a second preprocessing module, a protocol conversion module, a second control module, and a sending module. The acquisition module uses an Extended Berkeley Packet Filter (eBPF), and the corresponding caching module is an eBPF map buffer to allow data sharing between kernel mode and user mode; that is, data is stored in kernel mode while user mode can access it. The protocol conversion module can use an OpenTelemetry protocol conversion module to convert data into standard OpenTelemetry protocol format packets. Efficient packet processing operations are performed through the Express Data Path (XDP), and network packet processing is implemented through packet address families (AF_PACKET) or the Berkeley Packet Filter (BPF).

[0072] Specifically, multiple applications within the server are interconnected, including the source application and its corresponding subordinate applications, such as... Figure 3 The source application, sub-application 1, and sub-application 2 are shown. Please refer to [link / reference]. Figure 4 When the source application initiates a request, the corresponding link identifier is added to the header field information of the request according to the actual needs. This is used to uniquely identify the complete call chain of a request. The link identifier includes trace ID, span ID, and opcode, etc., so that important request chains can be customized, reducing the pressure of data collection, processing and storage.

[0073] For example, the link identifier is X-B3-SpanId, used to identify the application corresponding to the current request, and can be used to link multiple applications. For example, X-B3-SpanId: 1000000000000000 represents the source application, and 16 random characters represent other subordinate applications; the link identifier is X-B3-TraceId, used to identify the link information of a request, for example, X-B3-TraceId: ffffffe2118b5bad52588ada81f1199d, with the identifier format of "6 f's + 26 random numbers"; the link identifier is X-OP-Code, used to identify a request instance of a certain action, for example, X-OP-Code: Encryption {"User": "system"; "Platform": "Business Name"; "Action": "Upgrade"}.

[0074] In kernel mode, the acquisition module collects network data packets generated during the establishment of socket connections, thus completing the raw data acquisition of all application processes under the current operating system. This involves collecting network data packets from kernel functions at each stage of the socket connection, including kernel functions such as sendto() and recv(). The collected network data packets are stored in a cache module and then mapped to the first preprocessing module in user mode. This module filters the acquired network data packets to remove invalid and non-configured protocols. Session aggregation is then performed on the filtered network data packets to form a complete link. Finally, the aggregated network data packets are parsed to obtain the corresponding link information.

[0075] Furthermore, the link information protocol is converted into the corresponding first target protocol message through the protocol conversion module, and the first target protocol message is placed into the link (trace) queue. The first target protocol message is then transmitted to the remote server through the sending module.

[0076] Please see Figure 5 Before a lower-level application receives a request, a collection module pre-mounted on a kernel function collects network packets. An association management module associates these packets with application process information, such as process metadata and thread metadata, and stores them in a kernel-mode cache. Process metadata includes the process ID (PID) and the file descriptor (fd) for the connection. Based on the application process information, the link information is stored in the corresponding application's kernel threading model, ensuring that all thread calls can access the link information within the kernel threading model. The network packets stored in the cache are then mapped and collected in user space. A first preprocessing module filters and aggregates the acquired network packets to form a complete link through session aggregation. The aggregated network packets are then parsed to obtain the corresponding link information. A protocol conversion module converts the link information protocol into the corresponding first target protocol message and places it in a link queue. Finally, a sending module transmits the first target protocol message to a remote server.

[0077] Please see Figure 6When a lower-level application processes a request, it invokes the context thread to determine if the corresponding link information exists in the kernel threading model. If it does, the log data collected by the log collection module is associated with the corresponding link information and stored in a kernel-mode cache unit. The associated log data stored in the cache unit is then mapped and collected in user mode. The second preprocessing module in user mode parses the acquired log data, converts the parsed log data protocol into the corresponding second target protocol message, and then places the second target protocol message into a log queue. Finally, the sending module transmits the second target protocol message to the remote server.

[0078] Please see Figure 7 When a lower-level application initiates a request, as network data packets pass through the acquisition module, the second control module in user space, after determining the link information corresponding to the current network data packet, generates a random number of preset lengths. When a target network data packet with the same link information is detected, the first control module in kernel space modifies the header information of the target network data packet based on the currently generated random number. The modified target network data packet is stored in the kernel space cache unit, and the target network data packets in the cache unit are mapped and acquired in user space. The first preprocessing module filters and aggregates the acquired target network data packets to form a complete link through session aggregation. The aggregated target network data packets are parsed to obtain the corresponding link information. The protocol conversion module converts the link information protocol into the corresponding first target protocol message, puts the first target protocol message into the link queue, and transmits the first target protocol message to the remote server through the sending module.

[0079] It should be noted that in the above parsing process, the aggregated network data packets are parsed according to the preset protocol format template to obtain link information, i.e., the link structure field. After the parsing is completed, the link structure field is converted into the corresponding first target protocol message.

[0080] Furthermore, under different request states of various applications, the kernel periodically collects metric data and stores it in the cache module. The second preprocessing module in user space maps the collected metric data to user space, parses the data, and performs protocol conversion on the parsed data to obtain the corresponding third-target protocol message. This third-target protocol message is then placed in the metric queue and transmitted to the remote server via the sending module.

[0081] Existing data processing methods associate collected network packets with process metadata, and then process the associated network packets to query data. However, the link data obtained by the above methods is single-link data, which requires additional process information to connect the links, making the data processing process overly complex.

[0082] Compared to existing technologies, this application determines the status information corresponding to multiple associated applications, which indicates the request status of each application. Based on the status information, it acquires network data information generated by the applications during network communication, including network data packets. The network data packets are preprocessed to obtain corresponding link information. Furthermore, the link information is protocol-converted into the corresponding first target protocol message. Based on this, by non-intrusively collecting application link data in kernel mode, and then concatenating the collected link data after protocol conversion to obtain the link data corresponding to a single request, this application solves the problem of relying on additional process information for link concatenation, which leads to overly complex data processing. It achieves link concatenation without relying on additional process information, reducing data processing complexity, and is suitable for parallel request processes, making it applicable to a wide range of scenarios.

[0083] In some embodiments, step S220 involves obtaining network data information generated by the application during network communication based on state information; the network data information includes network data packets, and includes the following steps:

[0084] When the status information indicates that the request was initiated by the source application in each application, the network data packets generated by the application during network communication are stored in the kernel-mode cache module;

[0085] The network data packets in the cache module are mapped and collected to user space.

[0086] Specifically, when the source application initiates a request, it adds corresponding link identifiers to the request header field information according to actual needs. These identifiers uniquely identify the complete call chain of a request. Link identifiers include traceID, spanID, and opCode, facilitating the customization of important request chains and reducing the pressure on data collection, processing, and storage. For example, a link identifier of X-B3-SpanId identifies the application corresponding to the current request and can be used to link multiple applications; a link identifier of X-B3-TraceId identifies the request chain information; and a link identifier of X-OP-Code identifies a specific action request instance.

[0087] In kernel mode, the acquisition module collects network packets generated during the socket connection establishment process, thus completing the raw data acquisition of all application processes under the current operating system. It collects network packets from kernel functions at each stage of the socket connection, including kernel functions such as sendto() and recv(). The collected network packets are stored in the cache module and then mapped to the first preprocessing module in user mode.

[0088] Furthermore, in the first preprocessing module, the acquired network data packets are filtered to remove invalid and non-configured protocols. The filtered network data packets are then subjected to session aggregation to form a complete link. Next, the aggregated network data packets are parsed to obtain the corresponding link information. The protocol conversion module converts the link information protocol into the corresponding first target protocol message, which is then placed in the link queue and transmitted to the remote server via the sending module.

[0089] It should be noted that the kernel periodically collects metric data and stores it in the cache module. The second preprocessing module in user space maps the collected metric data to user space, parses the data, and performs protocol conversion on the parsed data to obtain the corresponding third-target protocol message. This third-target protocol message is then placed in the metric queue and transmitted to the remote server via the sending module.

[0090] In this embodiment, when the status information indicates that the source application in each application initiated the request, the network data packets generated by the application during network communication are stored in the kernel-mode cache module, and the network data packets in the cache module are mapped and collected in the user mode, thereby achieving non-intrusive collection of network data packets.

[0091] In some embodiments, step S220 involves obtaining network data information generated by the application during network communication based on state information; the network data information includes network data packets, and includes the following steps:

[0092] When the status information indicates that the lower-level application in each application is waiting to receive a request, collect the network data packets generated by the application during network communication.

[0093] Network data packets are associated with their corresponding application process information and stored in a kernel-mode cache unit;

[0094] The network data packets associated with the cache unit are mapped and captured in user space.

[0095] Specifically, before the lower-level application receives a request, a collection module pre-mounted on a kernel function collects network packets generated during the establishment of the sokect connection. Through an association management module, these network packets are associated with application process information such as process metadata and thread metadata and stored in a kernel-mode cache. Based on the application process information, the link information is stored in the corresponding application's kernel threading model, ensuring that all thread calls can access the link information within the kernel threading model. Specifically, the corresponding process information is obtained through the network connection's remote address, and the thread identifier currently handling the request is obtained by examining the process's thread table. Process metadata includes the process ID (PID) and the file descriptor (fd) of the connection.

[0096] Furthermore, the network data packets associated with the cache unit are mapped and collected to the first preprocessing module in user space. The acquired network data packets are filtered and aggregated to form a complete link through session aggregation. Then, the aggregated network data packets are parsed to obtain the corresponding link information. The link information protocol is converted into the corresponding first target protocol message by the protocol conversion module, and the first target protocol message is placed in the link queue. Finally, the first target protocol message is transmitted to the remote server through the sending module.

[0097] It should be noted that the kernel periodically collects metric data and stores it in the cache module. The second preprocessing module in user space maps the collected metric data to user space, parses the data, and performs protocol conversion on the parsed data to obtain the corresponding third-target protocol message. This third-target protocol message is then placed in the metric queue and transmitted to the remote server via the sending module.

[0098] In this embodiment, when the status information is that the lower-level application in each application is waiting to receive a request, the network data packets generated by the application during network communication are collected, the network data packets are associated with the corresponding application process information and stored in the kernel-mode cache unit, and the network data packets associated and stored in the cache unit are mapped and collected in the user mode, thereby achieving non-intrusive collection of network data packets.

[0099] In some embodiments, after determining the status information corresponding to multiple associated applications, the processing method further includes the following steps:

[0100] When the status information is that a lower-level application in each application is processing a request, obtain the network data information generated by the application during network communication; the network data information includes log data.

[0101] Log data is associated with the corresponding link information and stored in a kernel-mode cache unit;

[0102] Map and collect the log data associated with the cache unit to user space;

[0103] The parsed log data is then converted to a different protocol to obtain the corresponding second target protocol message.

[0104] Specifically, when the lower-level application processes a request, it calls standard library functions such as open, read, write, and close to write logs to a file in user space, and then calls corresponding file operation functions in kernel space to perform the actual file operations, such as filp_open(), vfs_read(), vfs_write(), and filp_close(). This allows the log collection module to be pre-mounted on the kernel printing function vfs_write() to obtain the log content printed by the application.

[0105] Furthermore, the context thread is invoked to determine if the corresponding link information exists in the kernel threading model. If it does, the log data collected by the log acquisition module is associated with the corresponding link information to form a data format message, which is then stored in the kernel-mode cache unit. The associated log data stored in the cache unit is then mapped and acquired in user mode. The second preprocessing module in user mode parses the acquired log data, converts the parsed log data protocol into the corresponding second target protocol message, and then places the second target protocol message into the log queue. Finally, the sending module transmits the second target protocol message to the remote server.

[0106] In this embodiment, when the status information is a request processed by a lower-level application in each application, log data generated by the application during network communication is obtained. The log data is associated with the corresponding link information and stored in the kernel-mode cache unit. The log data associated with the cache unit is then mapped and collected in the user mode. The parsed log data is then converted to a protocol to obtain the corresponding second target protocol message. This achieves non-intrusive collection of log data from internal applications, thereby ensuring sufficient data for subsequent fault analysis and improving the accuracy of fault analysis.

[0107] In some embodiments, step S220 involves obtaining network data information generated by the application during network communication based on state information; the network data information includes network data packets, and includes the following steps:

[0108] When the status information indicates that a request is initiated by a lower-level application within each application, the corresponding link information is determined based on the network data packets generated by the application during network communication.

[0109] When a target network data packet with the same link information is detected, the header field information of the target network data packet is modified according to the currently generated random number;

[0110] The modified target network data packet is stored in the kernel-mode cache unit;

[0111] The target network data packets in the cache unit are mapped and collected to user space.

[0112] Specifically, when a request is initiated by a lower-level application, the currently collected network data packets are filtered and aggregated in the first preprocessing module to obtain the corresponding link information. The second control module in user mode generates a random number of a preset number of bits. For example, if the link identifier is X-B3-SpanId: 10000000000000000, a new 16-bit random number is generated.

[0113] Furthermore, when the second control module detects a target network data packet with the same link information, the first control module in the kernel space modifies the header information of the target network data packet based on a currently generated random number. The modified target network data packet is stored in the kernel-space cache unit, and the target network data packets in the cache unit are mapped and acquired in user space. The first preprocessing module in user space filters and aggregates the acquired target network data packets to form a complete link through session aggregation. Then, the aggregated target network data packets are parsed to obtain the corresponding link information. The protocol conversion module converts the link information protocol into the corresponding first target protocol message, places the first target protocol message in the link queue, and transmits the first target protocol message to the remote server through the sending module.

[0114] It should be noted that the kernel periodically collects metric data and stores it in the cache module. The second preprocessing module in user space maps the collected metric data to user space, parses the data, and performs protocol conversion on the parsed data to obtain the corresponding third-target protocol message. This third-target protocol message is then placed in the metric queue and transmitted to the remote server via the sending module.

[0115] In this embodiment, when the status information indicates that a request is initiated by a lower-level application within each application, the link information corresponding to the current network data packet is determined. When a target network data packet with the same link information is detected, the header field information of the target network data packet is modified according to the currently generated random number. The modified target network data packet is stored in the kernel-mode cache unit, and the target network data packet in the cache unit is mapped and collected in the user mode. This allows the link identification information of the network data packet to be modified when a lower-level application initiates a request, so as to accurately obtain the link data, log data, and indicator data corresponding to the current request.

[0116] In some embodiments, the preprocessing of network data packets in step S230 to obtain the corresponding link information includes the following steps:

[0117] Step S231: Filter network data packets;

[0118] Step S232: Perform session aggregation on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session;

[0119] Step S233: Parse the aggregated network data packets to obtain link information.

[0120] Specifically, after the first preprocessing module obtains network data packets, it filters the network data packets to filter out invalid protocols and non-configured protocols. The filtered network data packets are then aggregated for session to form a complete link. The aggregated network data packets are then parsed according to a preset protocol format template to obtain the link information, i.e., the link structure field.

[0121] It should be noted that the network packets processed by aggregation include both request packets and response packets in the same communication session.

[0122] In this embodiment, network data packets are filtered, and session aggregation is performed on the filtered network data packets. The aggregated network data packets are then parsed to obtain link information. In this way, session aggregation is performed on the network data packets collected by mapping to form a complete link.

[0123] In some embodiments, step S232 involves session aggregation of the filtered network packets; the network packets include request packets and response packets within the same communication session, and includes the following steps:

[0124] Determine if a network data packet contains a flow protocol tag;

[0125] When a flow protocol marker is detected in a network data packet, the application layer protocol information corresponding to the network data packet is obtained;

[0126] Session aggregation is performed on network packets with the same application layer protocol information; network packets include request packets and response packets.

[0127] For details, please refer to Figure 8 The process involves: acquiring network data packets (S801), determining whether the network data packets contain a flow protocol tag (S802), and iterating through each configured protocol header to parse the payload header of the data packet for verification (S803). If header parsing fails, the current data is discarded (S804), and if multiple failures occur, an "unknown" tag is added to the current flow (S805). Otherwise, the corresponding flow protocol tagging operation is performed based on the verification result (S806).

[0128] Furthermore, upon detecting the presence of a flow protocol marker in a network data packet, the packet's payload is parsed to obtain the corresponding application layer protocol information (S807). The network data packet is then added to the session aggregation, and the system waits for the corresponding request or response data packet (S808). Upon receiving the corresponding request or response, a link is formed (S809). If no response is received for an extended period, a timeout error is reported (S810).

[0129] It should be noted that if the network data packet has a flow protocol tag, and the preset statistical period is reached, the indicator data will be collected in S811.

[0130] In this embodiment, it is determined whether a network data packet has a flow protocol mark. When a flow protocol mark is detected in a network data packet, the application layer protocol information corresponding to the network data packet is obtained. Request data packets and response data packets with the same application layer protocol information are aggregated to achieve data packet aggregation in the same communication session.

[0131] In some embodiments, after determining the status information corresponding to multiple associated applications, the method further includes the following steps:

[0132] Acquire network data information generated by the application during network communication; network data information includes indicator data.

[0133] Analyze the indicator data;

[0134] The parsed and processed indicator data is then converted into a protocol to obtain the corresponding third target protocol message.

[0135] Specifically, under different request states of various applications, the kernel space periodically collects metric data according to a preset statistical cycle and sends the collected metric data to the cache module for storage. The second preprocessing module in user space maps the collected metric data to user space for parsing.

[0136] Furthermore, the parsed and processed indicator data undergoes protocol conversion to obtain the corresponding third target protocol message. This third target protocol message is then placed in the indicator queue and transmitted to the remote server via the sending module.

[0137] This embodiment acquires indicator data generated by the application during network communication, parses the indicator data, and performs protocol conversion on the parsed indicator data to obtain a third target protocol message. This achieves non-intrusive acquisition of indicator data, which helps in subsequent fault analysis and improves the accuracy of fault analysis.

[0138] The present embodiment will now be described and illustrated through preferred embodiments.

[0139] Figure 9 This is a flowchart of the application observable data processing method of this preferred embodiment, such as... Figure 9 As shown, the observable data processing method of this application includes the following steps:

[0140] Step S910: Determine the status information corresponding to multiple associated applications; the status information is used to indicate the request status of each application.

[0141] Step S920: Based on the status information, obtain the network data packets generated during the socket connection establishment process;

[0142] Step S930: Filter the network data packets and determine whether the filtered network data packets contain flow protocol tags;

[0143] Step S940: When a flow protocol marker is detected in a network data packet, the application layer protocol information corresponding to the network data packet is obtained;

[0144] Step S950: Perform session aggregation on request packets and response packets that have the same application layer protocol information;

[0145] Step S960: Parse the aggregation results to obtain the corresponding link information;

[0146] Step S970: Convert the link information protocol into the corresponding first target protocol message.

[0147] This embodiment determines the status information corresponding to multiple associated applications. This status information indicates the request status of each application. Based on the status information, network data packets generated during the socket connection establishment process are obtained. The network data packets are filtered to determine if they contain flow protocol markers. When a flow protocol marker is detected, the application layer protocol information corresponding to the network data packet is obtained. Session aggregation is performed on request and response data packets with the same application layer protocol information. Furthermore, the aggregation result is parsed to obtain link information, and the link information protocol is converted into the corresponding first target protocol message. This solves the problem of relying on additional process information for link concatenation, which leads to overly complex data processing. It achieves link concatenation without relying on additional process information, reducing the complexity of data processing.

[0148] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0149] This embodiment also provides an application observable data processing device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0150] Figure 10 This is a structural block diagram of the observable data processing device used in this embodiment, as shown below. Figure 10 As shown, the device includes: a detection module 10, an acquisition module 20, a preprocessing module 30, and a conversion module 40;

[0151] The detection module 10 is used to determine the status information corresponding to multiple associated applications; the status information is used to indicate the request status of each application.

[0152] The acquisition module 20 is used to acquire network data information generated by the application during network communication based on status information; the network data information includes network data packets;

[0153] Preprocessing module 30 is used to preprocess network data packets to obtain corresponding link information;

[0154] The conversion module 40 is used to convert the link information protocol into the corresponding first target protocol message.

[0155] The apparatus provided in this embodiment determines the status information corresponding to multiple associated applications, which is used to indicate the request status of each application; based on the status information, it obtains network data information generated by the applications during network communication, including network data packets; it preprocesses the network data packets to obtain the corresponding link information; further, it converts the link information protocol into the corresponding first target protocol message, which solves the problem that the data processing process is too complicated due to the need to rely on additional process information to realize link concatenation, and realizes that link concatenation can be performed without relying on additional process information, thereby reducing the complexity of data processing.

[0156] In some embodiments, the acquisition module 20 is further configured to, when the status information indicates that the source application in each application initiated the request, store the network data packets generated by the application during network communication in the kernel-mode cache module; and map the network data packets in the cache module to the user-mode.

[0157] In some embodiments, the acquisition module 20 is further configured to: collect network data packets generated by the application during network communication when the status information is that the lower-level application in each application is waiting to receive a request; associate the network data packets with the corresponding application process information and store them in the kernel-mode cache unit; and map the network data packets associated and stored in the cache unit to the user mode.

[0158] In some of these embodiments, in Figure 10 Based on this, the device also includes a log acquisition module, which is used to acquire network data information generated by the application during network communication when the status information is a request to a lower-level application in each application. The network data information includes log data. The log data is associated with the corresponding link information and stored in a kernel-mode cache unit. The log data associated with the cache unit is mapped and acquired in user mode. The parsed log data is converted to a protocol to obtain the corresponding second target protocol message.

[0159] In some embodiments, the acquisition module 20 is further configured to: determine the corresponding link information based on the network data packets generated by the application during network communication when the status information indicates that a request was initiated by a lower-level application in each application; modify the header field information of the target network data packet based on the currently generated random number when a target network data packet with the same link information is detected; store the modified target network data packet in the kernel-mode cache unit; and map the target network data packet in the cache unit to the user-mode.

[0160] In some embodiments, the preprocessing module 30 is also used to filter network data packets; perform session aggregation on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session; and parse the aggregated network data packets to obtain link information.

[0161] In some embodiments, the preprocessing module 30 is further configured to determine whether a network data packet contains a flow protocol tag; when a flow protocol tag is detected in a network data packet, to obtain the application layer protocol information corresponding to the network data packet; and to perform session aggregation on network data packets with the same application layer protocol information; the network data packets include request data packets and response data packets.

[0162] In some of these embodiments, in Figure 10 Based on this, the device also includes an indicator acquisition module, which is used to acquire network data information generated by the application during network communication; the network data information includes indicator data; the indicator data is parsed; and the parsed indicator data is converted into a protocol to obtain the corresponding third target protocol message.

[0163] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0164] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0165] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0166] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0167] S1, determine the status information corresponding to multiple associated applications; the status information is used to indicate the request status of each application;

[0168] S2, based on state information, obtain network data information generated by the application during network communication; network data information includes network data packets;

[0169] S3, preprocess network data packets to obtain the corresponding link information;

[0170] S4 converts the link information protocol into the corresponding first target protocol message.

[0171] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0172] Furthermore, in conjunction with the application observable data processing methods provided in the above embodiments, this embodiment can also provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the application observable data processing methods in the above embodiments.

[0173] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0174] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0175] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0176] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for processing observable data, characterized in that, The method includes: Determine the status information corresponding to multiple associated applications; the status information is used to indicate the request status of each application; the multiple associated applications are multiple applications associated within the server; the multiple associated applications include a source application and a subordinate application corresponding to the source application; wherein, when the source application initiates a request, a corresponding link identifier is added to the header field information of the request; the link identifier is used to uniquely identify the complete call chain of the request; Based on the status information, network data information generated by the application during network communication is obtained; the network data information includes network data packets. The network data packets are preprocessed to obtain the corresponding link information; The step of preprocessing the network data packets to obtain corresponding link information includes: filtering the network data packets; performing session aggregation on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session; and parsing the aggregated network data packets to obtain link information. The step of performing session aggregation on the filtered network data packets, wherein the network data packets include request data packets and response data packets in the same communication session, includes: determining whether the network data packets have a flow protocol tag; when the flow protocol tag is detected in the network data packets, obtaining the application layer protocol information corresponding to the network data packets; and performing session aggregation on network data packets with the same application layer protocol information; wherein the network data packets include request data packets and response data packets. The link information protocol is converted into the corresponding first target protocol message.

2. The application observable data processing method according to claim 1, characterized in that, Based on the status information, network data information generated by the application during network communication is obtained; The network data information includes network data packets, including: When the status information indicates that a request was initiated by the source application in each of the applications, the network data packets generated by the application during network communication are stored in the kernel-mode cache module; The network data packets in the cache module are mapped and collected to user space.

3. The method for processing observable data according to claim 1, characterized in that, Based on the status information, network data information generated by the application during network communication is obtained; The network data information includes network data packets, including: When the status information indicates that a lower-level application in each of the applications is waiting to receive a request, network data packets generated by the application during network communication are collected. The network data packets are associated with the corresponding application process information and stored in the kernel-mode cache unit; The network data packets associated with the cache unit are mapped and collected to the user space.

4. The method for processing observable data according to claim 1, characterized in that, After determining the status information corresponding to multiple associated applications, the method further includes: When the status information indicates a request to be processed by a lower-level application within each of the applications, network data information generated by the application during network communication is obtained; the network data information includes log data. The log data is associated with the corresponding link information and stored in the kernel-mode cache unit; The log data associated with the cache unit is mapped and collected to the user space; The parsed log data is then subjected to protocol conversion to obtain the corresponding second target protocol message.

5. The method for processing observable data according to claim 1, characterized in that, Based on the status information, network data information generated by the application during network communication is obtained; The network data information includes network data packets, including: When the status information indicates that a request is initiated by a lower-level application within each of the applications, the corresponding link information is determined based on the network data packets generated by the application during network communication. When a target network data packet with the same link information is detected, the header field information of the target network data packet is modified according to the currently generated random number; The modified target network data packet is stored in a kernel-mode cache unit; The target network data packets in the cache unit are mapped and collected to user space.

6. The method for processing observable data according to claim 1, characterized in that, After determining the status information corresponding to multiple associated applications, the method further includes: Acquire network data information generated by the application during network communication; the network data information includes indicator data. The indicator data is analyzed; The parsed and processed indicator data is then converted into a protocol to obtain the corresponding third target protocol message.

7. An application of observable data processing apparatus, characterized in that, The device includes: a detection module, an acquisition module, a preprocessing module, and a conversion module; The detection module is used to determine the status information corresponding to multiple associated applications; the status information is used to indicate the request status of each application; the multiple associated applications are multiple applications associated within the server; the multiple associated applications include a source application and a subordinate application corresponding to the source application; wherein, when the source application initiates a request, a corresponding link identifier is added to the header field information of the request; the link identifier is used to uniquely identify the complete call chain of the request; The acquisition module is used to acquire network data information generated by the application during network communication based on the status information; the network data information includes network data packets; The preprocessing module is used to preprocess the network data packets to obtain the corresponding link information; The preprocessing module is further configured to filter the network data packets; perform session aggregation on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session; and parse the aggregated network data packets to obtain link information. The preprocessing module is further configured to determine whether the network data packet contains a flow protocol tag; when the flow protocol tag is detected in the network data packet, obtain the application layer protocol information corresponding to the network data packet; perform session aggregation on network data packets with the same application layer protocol information; the network data packet includes request data packets and response data packets; The conversion module is used to convert the link information protocol into the corresponding first target protocol message.

8. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the application observable data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the application observable data processing method as described in any one of claims 1 to 6.

Images