Network message service link generation method and system based on deep learning

Abstract

The invention discloses a network message service link generation method and system based on deep learning, and relates to the technical field of deep learning, and the method comprises the steps: obtaining source data through an eBPF, an OpenTelemetry and a tcpdump; screening out information related to network communication from the source data; different association rules are applied according to data of different types of information related to network communication, and the data are associated; mining the association rule by using an FPGA (Field Programmable Gate Array) growth / Apriori algorithm to obtain association degree data; processing the mined correlation degree data by using LSTM to generate attention information, and inputting the correlation degree data as a reference weight into the model during model training; processing the output attention information by using a graph neural network to generate a graph neural network model; a graph structure is used as a service link graph. According to the data association method and device, the accuracy of data association is improved through association rule mining.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and more particularly to a method and system for generating a network message service link based on deep learning. Background Art

[0002] With the rapid development of information technology, the complexity of network services has been increasing day by day, posing higher requirements for network message analysis and the generation of service link data. In the field of network monitoring and analysis, accurately understanding the interaction relationships between different components in the network and the execution paths of service processes is of crucial significance for optimizing system performance, troubleshooting, security detection, etc.

[0003] Traditional network message analysis methods often focus on parsing and counting the basic information of messages, such as source address, destination address, port number, and protocol type. In terms of service link generation, it usually relies on time series to associate network messages. However, due to the complexity of the network environment and the diversity of data collection, this method often has large errors and cannot meet the requirements for refined analysis of service links in modern complex network environments.

[0004] In recent years, in order to obtain service link data more comprehensively, some technical means such as using eBPF (Extended Berkeley Packet Filter) to capture system call stacks have gradually been applied. eBPF can dynamically insert code at the operating system kernel level to achieve monitoring and data collection of system events, including obtaining system call stack information. By capturing the system call stack, to a certain extent, the running trajectory of network application programs at the operating system level and the call order between different functions can be revealed, which has potential value for constructing service links. However, eBPF capturing system call stacks is not without defects. In the actual application environment, due to the complexity and dynamics of the system, the operation of eBPF programs may be interfered by various external factors. For example, kernel differences between different operating system versions may lead to compatibility issues of eBPF code, and some kernel configurations or security mechanisms may limit the normal functioning of eBPF. Even in a high-load system environment, the accuracy and integrity of eBPF captured data may also be affected, resulting in deviations or omissions in the service link data generated based on it.

[0005] Meanwhile, OpenTelemetry-related technologies are also widely used to collect call stack information and build distributed tracing systems. OpenTelemetry provides a set of standardized interfaces and tools, which facilitate developers to integrate call stack collection functions in different programming languages and frameworks and send the collected data to the backend analysis platform. However, OpenTelemetry also faces challenges from external influences. In a distributed system, factors such as network instability, version differences between different services, and resource competition may all lead to incomplete or inaccurate call stack collection. For example, network jitter may cause some call stack data to be lost during transmission, different services using different versions of the OpenTelemetry library may result in inconsistent data formats or differences in collection logic, and service instances with resource constraints may not be able to process call stack collection tasks in a timely manner, thus missing important call information. These issues will cause errors between the business links generated based on the call stack data collected by OpenTelemetry and the actual situation, affecting the accurate understanding and effective management of network services.

[0006] In summary, the existing methods of capturing system call stacks through eBPF and using OpenTelemetry-related technologies to collect call stacks to build business link data have certain limitations in terms of environmental compatibility, data accuracy, etc. due to interference from various external factors.

[0007] Therefore, it is necessary to propose a method and system for generating network packet business links based on deep learning to solve the above problems. Summary of the Invention

[0008] The purpose of the present invention is to solve the problem that the existing methods of capturing system call stacks through eBPF and using OpenTelemetry-related technologies to collect call stacks to build business link data have certain limitations in terms of environmental compatibility, data accuracy, etc. due to interference from various external factors.

[0009] The present invention specifically adopts the following technical solutions to achieve the above purpose:

[0010] A method for generating network packet business links based on deep learning includes the following steps:

[0011] S1. Obtain source data through eBPF, OpenTelemetry, and tcpdump;

[0012] S2. Screen out information related to network communication from the source data;

[0013] S3. Apply different association rules to data of different types of information related to network communication to associate the data;

[0014] S4. Use the FPGrowth / Apriori algorithm to mine the association rules and obtain the association degree data;

[0015] S5. Use LSTM to process the mined association degree data to generate attention information, and use the association degree data as the reference weight input into the model during model training;

[0016] S6. Use the graph neural network to process the output attention information to generate the graph neural network model;

[0017] S7. Use the graph structure as the business link graph.

[0018] Further, the association rules in S3 include:

[0019] For eBPF data, use the process / thread ID + time as the association rule;

[0020] For OpenTelemetry data, use the traceID as the association rule;

[0021] For tcpdump data, use the time as the association rule.

[0022] Further, the association degree data in S4 is updated daily.

[0023] Further, when using the association degree data as the weight input into LSTM in S5, the association degree data is processed by the embedding method, and the data processed by embbeing is used as the weight to participate in the training.

[0024] Further, the graph neural network in S6 includes but is not limited to GNN, GCN, GraphSAGE, and GAE.

[0025] The network packet business link generation system based on deep learning includes the above-mentioned network packet business link generation method based on deep learning, and further includes:

[0026] The data acquisition module is used to obtain the source data through eBPF, OpenTelemetry, and tcpdump;

[0027] The data screening module is used to screen out the network communication-related information from the source data;

[0028] The data association module is used to apply different association rules to the data of different types of network communication-related information to associate the data;

[0029] An association rule mining module, which is used to mine association rules using the FPGrowth / Apriori algorithm to obtain association degree data;

[0030] An LSTM processing module, which is used to process the mined association degree data using LSTM to generate attention information, and use the association degree data as a reference weight input into the model during model training;

[0031] A graph neural network processing module, which is used to process the output data using a graph neural network to generate a graph neural network model;

[0032] A business link graph generation module, which is used to further use the graph structure as a business link graph.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] 1. In the present invention, through association rule mining, the accuracy of data association is improved.

[0035] 2. In the present invention, by generating attention information using LSTM, the processing ability of the model for time series data is enhanced.

[0036] 3. In the present invention, by generating a graph structure using a graph neural network, the generation process of the business link is further optimized. Description of the Drawings

[0037] Figure 1 is a flowchart of the method for generating a network packet service link in the present invention.

[0038] Figure 2 is a flowchart of obtaining source data by eBPF, OpenTelemetry and tcpdump in the present invention. Detailed Embodiments

[0039] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] Please refer to Figure 1 , a method for generating a network packet service link based on deep learning, includes the following steps:

[0041] A data acquisition module, in combination with Figure 2As shown in the figure, for the acquisition of raw data: Use a professional network packet capture tool tcpdump or the corresponding network monitoring module to capture network packets in the target network environment, and obtain the raw packet data containing various network interaction information; Use eBPF probes deployed on the server to obtain the call stack information inside the operating system; Use the OpenTelemetry component integrated with the application to obtain the call stack information inside the application.

[0042] The data screening module is used to extract data features: For the captured network packets, system call stacks, and software call stacks, use feature extraction algorithms to extract the key features among them, such as the source address, destination address, protocol type, port number, and other information of the packets. And screen the data related to the business link according to needs.

[0043] The data association module is used to establish the association rules between data: For eBPF data, use the process / thread ID + time as the association condition; For OpenTelemetry data, use the traceID as the association condition; For the network packet data captured by tcpdump, use time as the association condition; Establish the association rules between data to form an association set of data, but this association is only preliminary and needs to be further optimized later.

[0044] The association rule mining module uses the fpgrowth / apiori algorithm to mine association rules and obtain the association degree data between data: Use the above-mentioned association degree data as the input, and use association rule mining algorithms such as fpgrowth or apiori to mine deeper association rules between network requests, calculate the numerical value of the association degree data between them, so as to quantify the tightness of the association between the two.

[0045] Update the association degree data daily: Considering the dynamic variability of network services, repeat the above-mentioned association rule mining process for the newly captured and analyzed data every day to update the association degree data between network requests, so that the association degree data can reflect the latest state of network services in real time.

[0046] The LSTM processing module uses the LSTM (Long Short-Term Memory Network) model to process the raw data and generate time-based attention: Input the data that has been feature-extracted and screened from the data warehouse into the LSTM. Utilize the advantages of the LSTM in processing sequence data and modeling long-term and short-term dependencies to further process the data and generate time-based attention information. In order to improve the accuracy of the attention information, input the latest association degree data as the weight into the LSTM.

[0047] When using the correlation data as weights and inputting them into the LSTM, the correlation data is processed by the embedding method, and the data processed by the embedding is used as weights to participate in the training.

[0048] The graph neural network processing module uses models such as GNN (graph neural network), GCN, GraphSAGE or GAE to process the output data of the LSTM: leveraging the advantages of GNN in processing graph-structured data (such as the similar graph structure formed by the requested associations in the present invention), the data is finally deeply processed and analyzed to generate accurate business link data, completing the conversion process from network packets to high-quality business link data. Finally, through the business link graph generation module, the graph structure is used as the business link graph.

[0049] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. The scope of patent protection of the present invention is subject to the claims. All equivalent structural changes made by using the description and drawings of the present invention shall, by the same token, be included within the protection scope of the present invention.

Claims

1. A method for generating a network message service link based on deep learning, characterized in that: The following steps are involved: S1. Obtain source data through eBPF, OpenTelemetry, and tcpdump. S2, filtering out information related to network communication from source data; S3, applying different association rules to data of different types of network communication related information to associate the data; S4. Use FPGrowth / Apriori algorithm to mine association rules and obtain association data; S5. Use LSTM to process the mined relevance data, generate attention information, and use the relevance data as a reference weight input into the model during model training; S6. Use the graph neural network to process the output attention information and generate a graph neural network model; S7. Use the graph structure as the business link graph.

2. The method for generating a network message service link based on deep learning according to claim 1, characterized in that: The association rules in S3 include: For eBPF data, use process / thread ID + time as the association rule; For OpenTelemetry data, use traceID as the association rule; For tcpdump data, use time as the correlation rule.

3. The method for generating a network message service link based on deep learning according to claim 1, characterized in that: The correlation data in S4 is updated daily.

4. The method for generating a network message service link based on deep learning according to claim 1, characterized in that: In the S5, when the relevance data is used as weight input to the LSTM, the relevance data is processed by the embedding method, and the data processed by embedding is used as the weight to participate in the training.

5. The method for generating a network message service link based on deep learning according to claim 1, characterized in that: The graph neural networks in S6 include but are not limited to GNN, GCN, GraphSAGE and GAE.

6. A network message service link generation system based on deep learning, comprising any one of the network message service link generation methods based on deep learning in claims 1-5, characterized in that: Also includes: Data collection module, used to obtain source data through eBPF, OpenTelemetry and tcpdump; A data filtering module is used to filter out information related to network communication from source data; A data association module, used to apply different association rules to data of different types of network communication related information to associate the data; The association rule mining module is used to mine association rules using the FPGrowth / Apriori algorithm to obtain association data; LSTM processing module, used to process the mined relevance data using LSTM, generate attention information, and use the relevance data as a reference weight input into the model during model training; A graph neural network processing module, which is used to process the output data using the graph neural network to generate a graph neural network model; The business link graph generation module is used to use the graph structure as the business link graph.

Images