Application observable data processing method and device, computer equipment and storage medium

Abstract

The invention relates to an application observable data processing method and device, computer equipment and a storage medium, and the application observable data processing method comprises the steps that state information corresponding to a plurality of associated application programs is determined, and the state information is used for indicating the request state of each application program; based on the state information, network data information generated by the application program in the network communication process is obtained, and the network data information comprises a network data packet; preprocessing the network data packet to obtain corresponding link information; further, the link information protocol is converted into a corresponding first target protocol message. By means of the method and device, the problem that the data processing process is too complex due to the fact that link series connection needs to be achieved by depending on additional process information is solved, link series connection can be achieved without depending on the additional process information, and the data processing complexity is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an observable data processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the application of cloud computing, microservices and distributed technologies in the security field, the call levels and call relationships between security business software systems are becoming increasingly complex. When a security business software system fails, in the face of a complex software system, the observable data of the application such as links, logs and indicator data is usually used to analyze the failure point and root cause of the failure in order to achieve timely and effective early warning.

[0003] In the existing data processing method, the collected network data packets are associated with the process metadata, and the associated network data packets are processed to query the data. However, the link data obtained by the above method is single link data, which needs to rely on additional process information to realize link series connection, making the data processing process too complicated.

[0004] There is currently no effective solution to the problem that related technologies require reliance on additional process information to achieve link concatenation, which makes the data processing process too complicated. Summary of the invention

[0005] In this embodiment, an application observable data processing method, apparatus, computer device and storage medium are provided to solve the problem in the related art that the data processing process is too complicated due to the need to rely on additional process information to realize link series connection.

[0006] In a first aspect, a method for processing application observable data is provided in this embodiment, and the method includes:

[0007] Determining status information corresponding to a plurality of associated applications; the status information is used to indicate a request status of each of the applications;

[0008] Based on the state information, obtaining network data information generated by the application during network communication; the network data information includes network data packets;

[0009] Preprocessing the network data packet to obtain corresponding link information;

[0010] The link information protocol is converted into a corresponding first target protocol message.

[0011] In some embodiments, the network data information generated by the application during the network communication process is obtained based on the state information; the network data information includes a network data packet, including:

[0012] When the state information indicates that a source application in each of the applications initiates a request, the network data packets generated by the application during the network communication are stored in a cache module in the kernel state;

[0013] The network data packets in the cache module are mapped and collected into the user state.

[0014] In some embodiments, the network data information generated by the application during the network communication process is obtained based on the state information; the network data information includes a network data packet, including:

[0015] When the status information indicates that a lower-level application in each of the applications is waiting to receive a request, collecting network data packets generated by the application during network communication;

[0016] Associating the network data packet with the corresponding application process information and storing it in a cache unit in kernel state;

[0017] The network data packet mapping associated with and stored in the cache unit is collected into the user state.

[0018] In some of the embodiments, after determining the state information corresponding to the plurality of associated applications, the method further includes:

[0019] In the case where the status information is a processing request of a lower-level application in each of the applications, obtaining network data information generated by the application during network communication; the network data information includes log data;

[0020] The log data is associated with the corresponding link information and stored in a cache unit in kernel state;

[0021] Mapping and collecting the log data associated and stored in the cache unit to the user state;

[0022] The log data after parsing is converted into a protocol to obtain a corresponding second target protocol message.

[0023] In some embodiments, the network data information generated by the application during the network communication process is obtained based on the state information; the network data information includes a network data packet, including:

[0024] In the case where the state information is a request initiated by a lower-level application in each of the applications, determining corresponding link information according to network data packets generated by the application during network communication;

[0025] When a target network data packet with the same link information is detected, the header field information of the target network data packet is modified according to the currently generated random number;

[0026] Storing the modified target network data packet in a cache unit in kernel mode;

[0027] The target network data packet in the cache unit is mapped and collected into the user state.

[0028] In some embodiments, the preprocessing of the network data packet to obtain corresponding link information includes:

[0029] Filtering the network data packets;

[0030] Performing session aggregation on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session;

[0031] The network data packets after aggregation processing are parsed to obtain link information.

[0032] In some embodiments, the filtering of the network data packets is subject to session aggregation; the network data packets include request data packets and response data packets in the same communication session, including:

[0033] Determine whether the network data packet has a stream protocol tag;

[0034] When detecting that the network data packet has the stream protocol tag, obtaining application layer protocol information corresponding to the network data packet;

[0035] Session aggregation is performed on network data packets with the same application layer protocol information; the network data packets include request data packets and response data packets.

[0036] In some of the embodiments, after determining the status information corresponding to the plurality of associated applications, the method further includes:

[0037] Acquire network data information generated by the application during network communication; the network data information includes indicator data;

[0038] Analyzing the indicator data;

[0039] The parsed indicator data is subjected to protocol conversion to obtain a corresponding third target protocol message.

[0040] In a second aspect, in this embodiment, an application observable data processing device is provided, the device comprising: a detection module, an acquisition module, a preprocessing module and a conversion module;

[0041] The detection module is used to determine the status information corresponding to the multiple associated applications; the status information is used to indicate the request status of each of the applications;

[0042] The acquisition module is used to acquire network data information generated by the application during network communication based on the status information; the network data information includes network data packets;

[0043] The preprocessing module is used to preprocess the network data packet to obtain corresponding link information;

[0044] The conversion module is used to convert the link information protocol into a corresponding first target protocol message.

[0045] In a third aspect, a computer device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the application observable data processing method described in the first aspect when executing the computer program.

[0046] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the application observable data processing method described in the first aspect is implemented.

[0047] Compared with the related art, the application observable data processing method, apparatus, computer equipment and storage medium provided in this embodiment determine the status information corresponding to multiple associated applications, where the status information is used to indicate the request status of each application; based on the status information, network data information generated by the application during network communication is obtained, and the network data information includes network data packets; the network data packets are pre-processed to obtain the corresponding link information; further, the link information protocol is converted into the corresponding first target protocol message, which solves the problem of relying on additional process information to achieve link series connection, which makes the data processing process too complicated, and achieves link series connection without relying on additional process information, thereby reducing the complexity of data processing.

[0048] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0050] Figure 1It is a hardware structure block diagram of a terminal device to which an observable data processing method is applied provided by an embodiment of the present application;

[0051] Figure 2 is a flow chart of an application observable data processing method provided by an embodiment of the present application;

[0052] Figure 3 It is a schematic diagram of a system design architecture provided by an embodiment of the present application;

[0053] Figure 4 It is a flowchart of an application observable data processing method provided by an embodiment of the present application;

[0054] Figure 5 is a flowchart of an application observable data processing method provided by another embodiment of the present application;

[0055] Figure 6 It is a flow chart of a log data collection method provided by an embodiment of the present application;

[0056] Figure 7 It is a schematic diagram of a data processing flow when a lower-level application initiates a request provided by an embodiment of the present application;

[0057] Figure 8 is a schematic diagram of an aggregation processing flow provided by an embodiment of the present application;

[0058] Fig. 9 is a flow chart of an application observable data processing method provided by a preferred embodiment of the present application;

[0059] Fig.10 It is a structural block diagram of an application observable data processing device provided in one embodiment of the present application.

[0060] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, detection module; 20, acquisition module; 30, pre-processing module; 40, conversion module. DETAILED DESCRIPTION

[0061] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0062] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0063] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal to which the observable data processing method of this embodiment is applied. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0064] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the application observable data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0065] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0066] In this embodiment, a method for processing application observable data is provided. Figure 2 is a flowchart of the application observable data processing method of this embodiment, such as Figure 2 As shown, the process includes the following steps:

[0067] Step S210, determining status information corresponding to a plurality of associated applications; the status information is used to indicate the request status of each application;

[0068] Step S220, based on the state information, obtaining network data information generated by the application during the network communication process; the network data information includes network data packets;

[0069] Step S230, pre-processing the network data packet to obtain corresponding link information;

[0070] Step S240: convert the link information protocol into a corresponding first target protocol message.

[0071] In this example, see Figure 3The system architecture is divided into kernel state and user state. The kernel state includes a collection module, a cache module, an association management module, a log collection module and a first control module. The user state includes a first preprocessing module, a second preprocessing module, a protocol conversion module, a second control module and a sending module. The collection module adopts an extended Berkeley Packet Filter (eBPF), and the corresponding cache module is an eBPF map cache area to allow data to be shared between the kernel state and the user state, that is, to store data in the kernel state and allow the user state to access the data. The protocol conversion module can adopt the OpenTelemetry protocol conversion module to convert data into a standard OpenTelemetry protocol format message. Among them, the Express Data Path (XDP) is used to perform efficient packet processing operations, and the network packet processing is realized through the packet address family (AF_PACKET) or the Berkeley Packet Filter (BPF).

[0072] Specifically, multiple applications within the server are associated, including source applications and corresponding subordinate applications, such as Figure 3 Source application, subordinate application 1, and subordinate application 2 are shown. See Figure 4 When the source application initiates a request, the corresponding link identifier is added to the header field information of the request according to actual needs to uniquely identify the complete call chain of a request. The link identifier includes the trace identifier (traceID), span identifier (spanID) and operation code (opCode), etc., so that important request links can be customized to reduce the pressure of data collection, processing and storage.

[0073] Exemplarily, the link identifier is X-B3-SpanId, which is used to identify the application corresponding to the current request and is used in series with multiple applications. For example, X-B3-SpanId: 1000000000000000 represents the source application, and random 16-bit characters represent other subordinate applications; the link identifier is X-B3-TraceId, which is used to identify a request link information, such as X-B3-TraceId: ffffffe2118b5bad52588ada81f1199d, and the identification format is "6 f+26 random numbers"; the link identifier is X-OP-Code, which is used to identify a behavior request instance, such as X-OP-Code: encrypted {"User":"system"; "Platform":"Business Name"; "Action":"Upgrade"}.

[0074] In kernel state, the network data packets generated during the establishment of the socket connection are collected through the collection module, that is, the original data collection of all application processes under the current operating system is completed, and the network data packets on the kernel functions at each stage of the socket connection are collected, including the kernel functions sendto() and recv(), etc. The collected network data packets are stored in the cache module, and the network data packets are mapped and collected to the first preprocessing module in user state, and the obtained network data packets are filtered to filter invalid protocols and non-configured protocols, and the filtered network data packets are aggregated by session to form a complete link through session aggregation, and the aggregated network data packets are parsed to obtain the corresponding link information.

[0075] Furthermore, the link information protocol is converted into a corresponding first target protocol message through a protocol conversion module, and the first target protocol message is put into a link (trace) queue, and the first target protocol message is transmitted to a remote server through a sending module.

[0076] See also Figure 5 , when the lower-level application is waiting to receive a request, that is, before the lower-level application receives a request, the network data packet is collected by the collection module pre-mounted on the kernel function, and the network data packet is associated with the process metadata, thread metadata and other application process information and stored in the kernel state cache unit through the association management module. The process metadata includes the process number (pid) and the connected file descriptor (fd), and the link information is stored in the kernel threading model (kernel Threading Model) of the corresponding application according to the application process information, so that the thread calling process can obtain the link information in the kernel Threading Model. The network data packets associated and stored in the cache unit are mapped and collected to the user state, and the obtained network data packets are filtered and aggregated by the first preprocessing module to form a complete link through session aggregation, and the aggregated network data packets are parsed to obtain the corresponding link information. Through the protocol conversion module, the link information protocol is converted into the corresponding first target protocol message, and the first target protocol message is placed in the link queue, and the first target protocol message is transmitted to the remote server through the sending module.

[0077] See also Figure 6, when the lower-level application processes the request, the context thread is called to determine whether the kernelThreading Model has the corresponding link information. If it does, the log data collected by the log collection module is associated with the corresponding link information and stored in the cache unit of the kernel state, and the log data mapping associated with the storage in the cache unit is collected in the user state. Through the second pre-processing module in the user state, the obtained log data is parsed, and the log data protocol after the parsing process is converted into the corresponding second target protocol message, and then the second target protocol message is placed in the log queue, and the second target protocol message is transmitted to the remote server through the sending module.

[0078] See also Figure 7 When the lower-level application initiates a request, when the network data packet passes through the acquisition module, the second control module in the user state generates a random number of preset bits after determining the link information corresponding to the current network data packet. When a target network data packet with the same link information is detected, the first control module in the kernel state is controlled to modify the header information of the target network data packet according to the currently generated random number. The modified target network data packet is stored in the cache unit of the kernel state, and the target network data packet in the cache unit is mapped and collected to the user state. The acquired target network data packet is filtered and aggregated by the first preprocessing module to form a complete link through session aggregation, and the aggregated target network data packet is parsed to obtain the corresponding link information. The link information protocol is converted into the corresponding first target protocol message by the protocol conversion module, and the first target protocol message is placed in the link queue, and the first target protocol message is transmitted to the remote server by the sending module.

[0079] It should be noted that in the above-mentioned parsing process, the aggregated network data packet is parsed according to the preset protocol format template to obtain link information, namely the link structure field, and after the parsing is completed, the link structure field is converted into the corresponding first target protocol message.

[0080] In addition, under different request states of each application, the kernel state regularly collects indicator data and sends it to the cache module for storage. Through the second preprocessing module in the user state, the indicator data is mapped and collected to the user state, the indicator data is parsed, and the parsed indicator data is converted to a protocol to obtain a corresponding third target protocol message. The third target protocol message is placed in the indicator (metric) queue, and the third target protocol message is transmitted to the remote server through the sending module.

[0081] In the existing data processing method, the collected network data packets are associated with the process metadata, and the associated network data packets are processed to query the data. However, the link data obtained by the above method is single link data, which needs to rely on additional process information to realize link series connection, making the data processing process too complicated.

[0082] Compared with the prior art, the present application determines the status information corresponding to multiple associated applications, and the status information is used to indicate the request status of each application; based on the status information, the network data information generated by the application during the network communication process is obtained, and the network data information includes network data packets; the network data packets are preprocessed to obtain the corresponding link information; further, the link information protocol is converted into the corresponding first target protocol message. Based on this, by non-invasively collecting application link data in the kernel state, the collected link data is serialized through protocol conversion to obtain the link data corresponding to a request, which solves the problem of relying on additional process information to realize link serialization, resulting in an overly complicated data processing process, and realizes link serialization without relying on additional process information, reducing the complexity of data processing, and is suitable for parallel processes of requests, with a wide range of applicable scenarios.

[0083] In some embodiments, the step S220 of acquiring network data information generated by the application during the network communication process based on the state information; the network data information includes network data packets, includes the following steps:

[0084] When the state information indicates that a source application in each application initiates a request, a network data packet generated by the application during the network communication process is stored in a cache module in the kernel state;

[0085] Map the network data packets in the cache module to the user state.

[0086] Specifically, when the source application initiates a request, the corresponding link identifier is added to the header field information of the request according to actual needs to uniquely identify the complete call chain of a request. The link identifier includes traceID spanID and opCode, etc., which facilitates the customization of important request links and reduces the pressure of data collection, processing and storage. For example, the link identifier is X-B3-SpanId, which is used to identify the application corresponding to the current request and is used in series with multiple applications; the link identifier is X-B3-TraceId, which is used to identify the link information of a request; the link identifier is X-OP-Code, which is used to identify a behavior request instance.

[0087] In kernel state, the network data packets generated during the socket connection establishment process are collected through the collection module, that is, the original data collection of all application processes under the current operating system is completed, and the network data packets on the kernel functions at each stage of the socket connection are collected, including the kernel functions sendto(), recv(), etc. The collected network data packets are stored in the cache module, and the network data packets are mapped and collected to the first preprocessing module in user state.

[0088] Furthermore, in the first preprocessing module, the acquired network data packets are filtered to filter invalid protocols and non-configured protocols, and session aggregation is performed on the filtered network data packets to form a complete link through session aggregation. After that, the aggregated network data packets are parsed to obtain corresponding link information, and the link information protocol is converted into the corresponding first target protocol message through the protocol conversion module, and the first target protocol message is placed in the link queue, and the first target protocol message is transmitted to the remote server through the sending module.

[0089] It should be noted that the kernel state periodically collects indicator data and sends it to the cache module for storage. Through the second preprocessing module in the user state, the indicator data is mapped and collected to the user state, the indicator data is parsed, and the parsed indicator data is converted to a protocol to obtain a corresponding third target protocol message. The third target protocol message is placed in the indicator queue, and the third target protocol message is transmitted to the remote server through the sending module.

[0090] Through this embodiment, when the status information is a request initiated by a source application in each application, the network data packets generated by the application during the network communication process are stored in the kernel state cache module, and the network data packets in the cache module are mapped and collected to the user state, thereby realizing non-intrusive collection of network data packets.

[0091] In some embodiments, the step S220 of acquiring network data information generated by the application during the network communication process based on the state information; the network data information includes network data packets, includes the following steps:

[0092] When the status information indicates that the lower-level application in each application is waiting to receive a request, the network data packets generated by the application during the network communication are collected;

[0093] Associating the network data packet with the corresponding application process information and storing it in a cache unit in the kernel state;

[0094] Mapping the network data packets associated and stored in the cache unit to the user state.

[0095] Specifically, when the lower-level application is waiting to receive a request, that is, before the lower-level application receives the request, the collection module pre-mounted on the kernel function collects the network data packets generated during the establishment of the sokect connection. Through the association management module, the network data packets are associated with the application process information such as process metadata and thread metadata and stored in the cache unit of the kernel state, and the link information is stored in the kernel threading model (kernelThreading Model) of the corresponding application according to the application process information, so that the thread calling process can obtain the link information in the kernel Threading Model. Among them, the corresponding process information is obtained through the network connection remote address (Remote Address), and the thread identifier of the current processing request is obtained by checking the thread table of the process. The process metadata includes the process number (pid) and the file descriptor (fd) of the connection.

[0096] Furthermore, the network data packets associated with the storage in the cache unit are mapped and collected to the first preprocessing module in the user state, and the obtained network data packets are filtered and aggregated to form a complete link through session aggregation. After that, the aggregated network data packets are parsed to obtain the corresponding link information, and the link information protocol is converted into the corresponding first target protocol message through the protocol conversion module, and the first target protocol message is placed in the link queue, and the first target protocol message is transmitted to the remote server through the sending module.

[0097] It should be noted that the kernel state periodically collects indicator data and sends it to the cache module for storage. Through the second preprocessing module in the user state, the indicator data is mapped and collected to the user state, the indicator data is parsed, and the parsed indicator data is converted to a protocol to obtain a corresponding third target protocol message. The third target protocol message is placed in the indicator queue, and the third target protocol message is transmitted to the remote server through the sending module.

[0098] Through this embodiment, when the status information is that the lower-level applications in each application are waiting to receive requests, the network data packets generated by the application during the network communication process are collected, the network data packets are associated with the corresponding application process information and stored in the kernel state cache unit, and the network data packets associated and stored in the cache unit are mapped and collected to the user state, thereby realizing non-intrusive collection of network data packets.

[0099] In some of the embodiments, after determining the status information corresponding to the plurality of associated applications, the processing method further comprises the following steps:

[0100] When the status information is a request for processing by a lower-level application in each application, obtaining network data information generated by the application during network communication; the network data information includes log data;

[0101] The log data is associated with the corresponding link information and stored in a cache unit in the kernel state;

[0102] Mapping and collecting the log data associated with the storage in the cache unit to the user state;

[0103] The parsed log data is converted into a protocol to obtain a corresponding second target protocol message.

[0104] Specifically, when the lower-level application processes the request, the log is written to the file through the standard library function in the user state, such as open, read, write and close functions, and the corresponding file operation function is called in the kernel state to perform the actual file operation, such as filp_open(), vfs_read(), vfs_write() and filp_close() functions, etc., so as to obtain the application print log content by pre-mounting the log collection module on the kernel print function vfs_write().

[0105] Further, the context thread is called to determine whether the kernel Threading Model has corresponding link information. If so, the log data collected by the log collection module is associated with the corresponding link information to form a data format message, which is stored in the cache unit of the kernel state, and the log data associated and stored in the cache unit is mapped and collected to the user state. The acquired log data is parsed by the second preprocessing module in the user state, and the protocol of the parsed log data is converted into the corresponding second target protocol message, and then the second target protocol message is placed in the log queue, and the second target protocol message is transmitted to the remote server by the sending module.

[0106] Through this embodiment, when the status information is a request processed by a lower-level application in each application, the log data generated by the application during the network communication process is obtained, the log data and the corresponding link information are associated and stored in the kernel state cache unit, and the log data associated and stored in the cache unit are mapped and collected to the user state, and the protocol conversion is performed on the parsed log data to obtain the corresponding second target protocol message, thereby realizing non-intrusive collection of log data called within the application, thereby ensuring that the amount of data used for subsequent fault analysis is sufficient and improving the accuracy of fault analysis.

[0107] In some embodiments, the step S220 of acquiring network data information generated by the application during the network communication process based on the state information; the network data information includes network data packets, includes the following steps:

[0108] In the case where the status information is a request initiated by a lower-level application in each application, the corresponding link information is determined according to the network data packets generated by the application during the network communication process;

[0109] When a target network data packet with the same link information is detected, the header field information of the target network data packet is modified according to the currently generated random number;

[0110] The modified target network data packet is stored in a cache unit in the kernel state;

[0111] Map the target network data packets in the cache unit to the user state.

[0112] Specifically, when the lower-level application initiates a request, the currently collected network data packets are filtered and aggregated in the first preprocessing module to obtain the corresponding link information, and the second control module in the user state generates a random number of preset bits. For example, when the link identifier is X-B3-SpanId: 1000000000000000, a new 16-bit random number is generated.

[0113] Furthermore, when the second control module detects a target network data packet with the same link information, the first control module in the kernel state controls the header information of the target network data packet according to the currently generated random number. The modified target network data packet is stored in the cache unit of the kernel state, and the target network data packet in the cache unit is mapped and collected to the user state. The acquired target network data packet is filtered and aggregated by the first preprocessing module in the user state to form a complete link through session aggregation. Afterwards, the aggregated target network data packet is parsed to obtain the corresponding link information, and the link information protocol is converted into the corresponding first target protocol message through the protocol conversion module, and the first target protocol message is placed in the link queue, and the first target protocol message is transmitted to the remote server through the sending module.

[0114] It should be noted that the kernel state periodically collects indicator data and sends it to the cache module for storage. Through the second preprocessing module in the user state, the indicator data is mapped and collected to the user state, the indicator data is parsed, and the parsed indicator data is converted to a protocol to obtain a corresponding third target protocol message. The third target protocol message is placed in the indicator queue, and the third target protocol message is transmitted to the remote server through the sending module.

[0115] Through this embodiment, when the status information is a request initiated by a lower-level application in each application, the link information corresponding to the current network data packet is determined, and when a target network data packet with the same link information is detected, the header field information of the target network data packet is modified according to the currently generated random number, and the modified target network data packet is stored in the kernel state cache unit, and the target network data packet in the cache unit is mapped and collected to the user state, so that when the lower-level application initiates a request, the link identification information of the network data packet can be modified to facilitate accurate acquisition of the link data corresponding to the current request as well as the log data and indicator data.

[0116] In some embodiments, the preprocessing of the network data packet in step S230 to obtain the corresponding link information includes the following steps:

[0117] Step S231, filtering network data packets;

[0118] Step S232, performing session aggregation on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session;

[0119] Step S233, parsing the aggregated network data packet to obtain link information.

[0120] Specifically, after the first preprocessing module obtains the network data packet, the network data packet is filtered; invalid protocols and non-configuration protocols are filtered out, session aggregation is performed on the filtered network data packet to form a complete link, and the aggregated network data packet is parsed according to a preset protocol format template to obtain link information, i.e., the link structure field.

[0121] It should be noted that the aggregated network data packets include request data packets and response data packets in the same communication session.

[0122] Through this embodiment, the network data packets are filtered, and the filtered network data packets are session aggregated, and the aggregated network data packets are parsed to obtain link information, so as to perform session aggregation on the mapped and collected network data packets to form a complete link.

[0123] In some embodiments, the step S232 performs session aggregation on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session, and includes the following steps:

[0124] Determine whether the network data packet has a stream protocol tag;

[0125] When a stream protocol tag is detected in a network data packet, application layer protocol information corresponding to the network data packet is obtained;

[0126] Session aggregation is performed on network data packets with the same application layer protocol information; the network data packets include request data packets and response data packets.

[0127] For details, please refer to Figure 8 , obtain the network data packet S801, and determine whether the network data packet has a stream protocol tag S802. When it is detected that the network data packet does not have a stream protocol tag, traverse each configured protocol header to parse the payload protocol header of the data packet for verification S803. When the protocol header parsing fails, discard the current data S804, and add the unknown tag to the current stream when multiple failures occur S805; otherwise, perform the corresponding stream protocol tag operation S806 according to the verification result.

[0128] Further, when a stream protocol tag is detected in a network data packet, the payload of the data packet is parsed to obtain the corresponding application layer protocol information S807, the network data packet is added to the session aggregation, and the corresponding request data packet or response data packet is waited for S808. When the corresponding request or response is received, it is merged into a link S809, and if no response is received for a long time, a timeout error is reported S810.

[0129] It should be noted that, when a network data packet has a stream protocol tag, if a preset statistical period is reached, indicator data collection S811 is performed.

[0130] Through this embodiment, it is determined whether a network data packet has a stream protocol tag. When it is detected that a network data packet has a stream protocol tag, the application layer protocol information corresponding to the network data packet is obtained, and the request data packet and the response data packet with the same application layer protocol information are aggregated, thereby realizing data packet aggregation in the same communication session.

[0131] In some of the embodiments, after determining the status information corresponding to the plurality of associated applications, the method further comprises the following steps:

[0132] Obtain network data information generated by the application during network communication; network data information includes indicator data;

[0133] Analyze the indicator data;

[0134] The parsed indicator data is converted into a protocol to obtain a corresponding third target protocol message.

[0135] Specifically, under different request states of each application, the kernel state regularly collects indicator data according to a preset statistical cycle, and sends the collected indicator data to the cache module for storage. The second preprocessing module in the user state maps the collected indicator data to the user state and analyzes the indicator data.

[0136] Furthermore, the parsed indicator data is converted into a protocol to obtain a corresponding third target protocol message, the third target protocol message is put into an indicator queue, and the third target protocol message is transmitted to a remote server through a sending module.

[0137] Through this embodiment, the indicator data generated by the application during the network communication process is obtained, the indicator data is parsed, and the parsed indicator data is converted into a protocol to obtain a third target protocol message, thereby realizing non-invasive collection of indicator data, facilitating subsequent fault analysis, and improving the accuracy of fault analysis.

[0138] The present embodiment is described and illustrated below through preferred embodiments.

[0139] Fig. 9 is a flow chart of the method for processing observable data applied in the preferred embodiment. Fig. 9 As shown, the application observable data processing method includes the following steps:

[0140] Step S910, determining status information corresponding to a plurality of associated applications; the status information is used to indicate the request status of each application;

[0141] Step S920, based on the status information, obtaining the network data packet generated during the socket connection establishment process;

[0142] Step S930, filtering the network data packets to determine whether the filtered network data packets have a flow protocol tag;

[0143] Step S940, when it is detected that the network data packet has a stream protocol tag, obtaining application layer protocol information corresponding to the network data packet;

[0144] Step S950, performing session aggregation on the request data packet and the response data packet having the same application layer protocol information;

[0145] Step S960, parsing the aggregation processing result to obtain corresponding link information;

[0146] Step S970: convert the link information protocol into a corresponding first target protocol message.

[0147] Through this embodiment, the status information corresponding to multiple associated applications is determined, and the status information is used to indicate the request status of each application. Based on the status information, the network data packet generated during the socket connection establishment process is obtained. The network data packet is filtered to determine whether the filtered network data packet has a stream protocol tag. When the network data packet is detected to have a stream protocol tag, the application layer protocol information corresponding to the network data packet is obtained, and the request data packet and the response data packet with the same application layer protocol information are session aggregated. Furthermore, the aggregation processing result is parsed to obtain the link information, and the link information protocol is converted into the corresponding first target protocol message, which solves the problem of relying on additional process information to realize link series connection, resulting in an overly complicated data processing process, and realizes link series connection without relying on additional process information, reducing the complexity of data processing.

[0148] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0149] In this embodiment, an application observable data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0150] Fig.10 is a structural block diagram of the application observable data processing device of this embodiment, such as Fig.10 As shown, the device includes: a detection module 10, an acquisition module 20, a preprocessing module 30 and a conversion module 40;

[0151] The detection module 10 is used to determine the status information corresponding to the multiple associated applications; the status information is used to indicate the request status of each application;

[0152] An acquisition module 20 is used to acquire network data information generated by an application during network communication based on the state information; the network data information includes network data packets;

[0153] The preprocessing module 30 is used to preprocess the network data packets to obtain corresponding link information;

[0154] The conversion module 40 is used to convert the link information protocol into a corresponding first target protocol message.

[0155] Through the device provided in this embodiment, status information corresponding to multiple associated applications is determined, and the status information is used to indicate the request status of each application; based on the status information, network data information generated by the application during the network communication process is obtained, and the network data information includes network data packets; the network data packets are preprocessed to obtain corresponding link information; further, the link information protocol is converted into the corresponding first target protocol message, which solves the problem of relying on additional process information to realize link series connection, resulting in an overly complicated data processing process, and realizes link series connection without relying on additional process information, thereby reducing the complexity of data processing.

[0156] In some of the embodiments, the acquisition module 20 is also used to store the network data packets generated by the application during the network communication process into the kernel state cache module when the status information initiates a request for the source application in each application; and map the network data packets in the cache module to the user state.

[0157] In some of the embodiments, the acquisition module 20 is also used to collect network data packets generated by applications during network communications when the status information is that lower-level applications in each application are waiting to receive requests; store the network data packets in association with the corresponding application process information in a kernel-state cache unit; and map the network data packets associated and stored in the cache unit to the user state.

[0158] In some of these embodiments, Fig.10 On the basis of, the device also includes a log collection module, which is used to obtain network data information generated by the application during the network communication process when the status information is a lower-level application processing request in each application; the network data information includes log data; the log data is associated with the corresponding link information and stored in the kernel state cache unit; the log data associated and stored in the cache unit is mapped and collected to the user state; the protocol conversion is performed on the parsed log data to obtain the corresponding second target protocol message.

[0159] In some of the embodiments, the acquisition module 20 is also used to determine the corresponding link information based on the network data packets generated by the application during the network communication process when the status information is a request initiated by a lower-level application in each application; when a target network data packet with the same link information is detected, the header field information of the target network data packet is modified according to the currently generated random number; the modified target network data packet is stored in a cache unit in the kernel state; and the target network data packet in the cache unit is mapped and collected to the user state.

[0160] In some of the embodiments, the preprocessing module 30 is also used to filter network data packets; perform session aggregation on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session; and parse the aggregated network data packets to obtain link information.

[0161] In some of the embodiments, the preprocessing module 30 is also used to determine whether a network data packet has a flow protocol tag; when a flow protocol tag is detected in the network data packet, obtain application layer protocol information corresponding to the network data packet; perform session aggregation on network data packets with the same application layer protocol information; the network data packet includes a request data packet and a response data packet.

[0162] In some of these embodiments, Fig.10 On the basis of, the device also includes an indicator collection module, which is used to obtain network data information generated by the application during network communication; the network data information includes indicator data; the indicator data is parsed; the parsed indicator data is converted into a protocol to obtain a corresponding third target protocol message.

[0163] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0164] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0165] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0166] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0167] S1, determining status information corresponding to a plurality of associated applications; the status information is used to indicate the request status of each application;

[0168] S2, based on the state information, obtaining network data information generated by the application during the network communication process; the network data information includes network data packets;

[0169] S3, pre-processing the network data packet to obtain the corresponding link information;

[0170] S4, converting the link information protocol into a corresponding first target protocol message.

[0171] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0172] In addition, in combination with the application observable data processing method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any application observable data processing method in the above embodiments is implemented.

[0173] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0174] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0175] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0176] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for processing application observable data, characterized in that: The method comprises: Determining status information corresponding to a plurality of associated applications; the status information is used to indicate a request status of each of the applications; Based on the state information, obtaining network data information generated by the application during network communication; the network data information includes network data packets; Preprocessing the network data packet to obtain corresponding link information; The link information protocol is converted into a corresponding first target protocol message.

2. The method for processing application observable data according to claim 1, characterized in that: Based on the status information, acquiring network data information generated by the application during network communication; The network data information includes network data packets, including: When the state information indicates that a source application in each of the applications initiates a request, the network data packets generated by the application during the network communication are stored in a cache module in the kernel state; The network data packets in the cache module are mapped and collected into the user state.

3. The method for processing application observable data according to claim 1, characterized in that: Based on the status information, acquiring network data information generated by the application during network communication; The network data information includes network data packets, including: When the status information indicates that a lower-level application in each of the applications is waiting to receive a request, collecting network data packets generated by the application during network communication; Associating the network data packet with the corresponding application process information and storing it in a cache unit in kernel state; The network data packet mapping associated with and stored in the cache unit is collected into the user state.

4. The method for processing application observable data according to claim 1, characterized in that: After determining the state information corresponding to the plurality of associated applications, the method further includes: In the case where the status information is a processing request of a lower-level application in each of the applications, obtaining network data information generated by the application during network communication; the network data information includes log data; The log data is associated with the corresponding link information and stored in a cache unit in kernel state; Mapping and collecting the log data associated and stored in the cache unit to the user state; The log data after parsing is converted into a protocol to obtain a corresponding second target protocol message.

5. The method for processing application observable data according to claim 1, characterized in that: Based on the status information, acquiring network data information generated by the application during network communication; The network data information includes network data packets, including: In the case where the state information is a request initiated by a lower-level application in each of the applications, determining corresponding link information according to network data packets generated by the application during network communication; When a target network data packet with the same link information is detected, the header field information of the target network data packet is modified according to the currently generated random number; Storing the modified target network data packet in a cache unit in kernel mode; The target network data packet in the cache unit is mapped and collected into the user state.

6. The method for processing application observable data according to claim 1, characterized in that: The preprocessing of the network data packet to obtain corresponding link information includes: Filtering the network data packets; Performing session aggregation on the filtered network data packets; the network data packets include request data packets and response data packets in the same communication session; The network data packets after aggregation processing are parsed to obtain link information.

7. The method for processing application observable data according to claim 6, characterized in that: The filtering of the network data packets is subject to session aggregation; the network data packets include request data packets and response data packets in the same communication session, including: Determine whether the network data packet has a stream protocol tag; When detecting that the network data packet has the stream protocol tag, obtaining application layer protocol information corresponding to the network data packet; Session aggregation is performed on network data packets with the same application layer protocol information; the network data packets include request data packets and response data packets.

8. The method for processing application observable data according to claim 1, characterized in that: After determining the status information corresponding to the plurality of associated applications, the method further includes: Acquire network data information generated by the application during network communication; the network data information includes indicator data; Analyzing the indicator data; The parsed indicator data is subjected to protocol conversion to obtain a corresponding third target protocol message.

9. An application observable data processing device, characterized in that: The device comprises: a detection module, an acquisition module, a preprocessing module and a conversion module; The detection module is used to determine the status information corresponding to the multiple associated applications; the status information is used to indicate the request status of each of the applications; The acquisition module is used to acquire network data information generated by the application during network communication based on the status information; the network data information includes network data packets; The preprocessing module is used to preprocess the network data packet to obtain corresponding link information; the conversion module is used to convert the link information protocol into a corresponding first target protocol message.

10. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the steps of the method for processing applied observable data according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for processing applied observable data according to any one of claims 1 to 8 are implemented.

Images