Traffic data processing method and system, storage medium and computer equipment

Abstract

The invention relates to the technical field of data processing and medical health, and particularly discloses a traffic data processing method and system, a storage medium and computer equipment, and the method comprises the steps that a traffic data monitoring component monitors traffic data through a traffic data monitoring agent deployed on each first node in a distributed cluster; obtaining container flow data of each first node, and forwarding the container flow data to a distributed search engine; the distributed search engine determines a second node corresponding to the container flow data based on a preset fragmentation strategy, and sends the container flow data to the second node for storage; the distributed search engine responds to a query instruction of container flow data, recognizes query information corresponding to the query instruction, determines a target second node matched with the query information based on the preset fragmentation strategy, reads target container flow data from the target second node, and sends the target container flow data to the distributed search engine; and generating a query result corresponding to the query instruction based on the target container flow data.

Description

Technical Field

[0001] This application relates to the fields of data processing and medical health technologies, and particularly to a method and system for processing traffic data, a storage medium, and a computer device. Background Art

[0002] In modern cloud computing environments, especially with the wide application of container orchestration platforms in the medical health field, network communication between services has become more complex and dynamic. Taking a Kubernetes cluster as an example, in a medical health information system, services are usually deployed in the form of pods, and these pods communicate through a complex network topology to implement various medical business processes, such as patient information management, medical image transmission, electronic medical record recording, etc.

[0003] To effectively monitor and manage this communication, traffic data monitoring tools such as Hubble have emerged. Traffic data monitoring tools can capture and display the network traffic topology diagrams between various pods in real time in the cluster. These tools can detail the traffic flow direction and magnitude between medical services, providing an intuitive network communication view for operation and maintenance personnel. In a medical health information system, when a service (such as an electronic medical record system) malfunctions, the traffic data monitoring tool can quickly locate which pod-to-pod communications have problems, which is crucial for quickly troubleshooting and resolving network failures, ensuring patient data security, and maintaining the continuity of medical services.

[0004] However, despite the excellent performance of traffic data monitoring tools in real-time traffic monitoring, they have a significant limitation: the lack of data persistence capabilities. This means that the captured network traffic data can only be viewed in real time when captured and cannot be saved for subsequent analysis. This directly affects the maintenance of medical health information systems, resulting in a lack of historical data support during the maintenance of medical health information systems. Summary of the Invention

[0005] In view of this, this application provides a method and system for processing traffic data, a storage medium, and a computer device. The traffic data monitoring component captures the container traffic data of each node in real time through traffic data monitoring agents deployed in a distributed cluster and forwards this data to a distributed search engine for storage and query. In this way, not only is real-time traffic monitoring achieved, but also the efficient storage and query capabilities of the distributed search engine are fully utilized to achieve data persistent storage and efficient query of large-scale container traffic data, thus solving the deficiency of the traffic data monitoring component in data persistence.

[0006] According to one aspect of this application, a method for processing traffic data is provided, including:

[0007] The traffic data monitoring component obtains the container traffic data of each first node through the traffic data monitoring agents deployed on each first node in the distributed cluster, and forwards the container traffic data to the distributed search engine;

[0008] The distributed search engine determines the second node corresponding to the container traffic data based on a preset sharding strategy, and sends the container traffic data to the second node for storage;

[0009] The distributed search engine responds to a query instruction for container traffic data, identifies the query information corresponding to the query instruction, determines the target second node matching the query information based on the preset sharding strategy, reads the target container traffic data from the target second node, and generates a query result corresponding to the query instruction based on the target container traffic data.

[0010] According to another aspect of the present application, there is provided a processing system for traffic data, including:

[0011] A traffic data monitoring component, configured to obtain the container traffic data of each first node through the traffic data monitoring agents deployed on each first node in the distributed cluster, and forward the container traffic data to the distributed search engine;

[0012] The distributed search engine is configured to determine the second node corresponding to the container traffic data based on a preset sharding strategy, and send the container traffic data to the second node for storage;

[0013] The distributed search engine is further configured to respond to a query instruction for container traffic data, identify the query information corresponding to the query instruction, determine the target second node matching the query information based on the preset sharding strategy, read the target container traffic data from the target second node, and generate a query result corresponding to the query instruction based on the target container traffic data.

[0014] According to yet another aspect of the present application, there is provided a storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method for processing traffic data is implemented.

[0015] According to still another aspect of the present application, there is provided a computer device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, and when the processor executes the program, the above-mentioned method for processing traffic data is implemented.

[0016] With the above technical solutions, a method and system for processing traffic data, a storage medium, and a computer device provided by this application. The traffic data monitoring agent on each first node can forward the obtained container traffic data to the traffic data monitoring component. Then, the traffic data monitoring component can forward these container traffic data to the distributed search engine. The distributed search engine determines which second node the container traffic data should be stored on based on a preset sharding strategy and stores it. In addition to receiving container traffic data and storing it on the second node, the distributed search engine can also be used to receive a query instruction for the container traffic data input by the user. When the query instruction is received, the distributed search engine first identifies the query information specified in the query instruction. Then, it uses the preset sharding strategy to determine which second nodes store data that matches the query information, reads the matching target container traffic data from the target second nodes, and generates a query result corresponding to the query instruction based on this data. In the embodiment of this application, the traffic data monitoring component captures the container traffic data of each node in real time through the traffic data monitoring agents deployed in the distributed cluster and forwards this data to the distributed search engine for storage and query. In this way, not only real-time traffic monitoring is achieved, but also the efficient storage and query capabilities of the distributed search engine are fully utilized, realizing data persistent storage and efficient query of large-scale container traffic data, thus solving the deficiency of the traffic data monitoring component in data persistence.

[0017] The above description is only an overview of the technical solutions of this application. In order to be able to understand the technical means of this application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of this application more obvious and understandable, the following specifically gives the specific implementation manners of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of this application, and constitute a part of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0019] Figure 1 Shows a schematic flow chart of a method for processing traffic data provided by an embodiment of this application;

[0020] Figure 2 Shows a schematic structural diagram of a system for processing traffic data provided by an embodiment of this application;

[0021] Figure 3 Shows a schematic structural diagram of a device of a computer device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0023] In this embodiment, a method for processing traffic data is provided. As Figure 1 shown, the method includes:

[0024] Step 101, the traffic data monitoring component obtains the container traffic data of each first node through the traffic data monitoring agents deployed on each first node in the distributed cluster, and forwards the container traffic data to the distributed search engine.

[0025] Step 102, the distributed search engine determines the second node corresponding to the container traffic data based on a preset sharding strategy, and sends the container traffic data to the second node for storage.

[0026] Step 103, in response to a query instruction for container traffic data, the distributed search engine identifies the query information corresponding to the query instruction, determines the target second node matching the query information based on the preset sharding strategy, reads the target container traffic data from the target second node, and generates a query result corresponding to the query instruction based on the target container traffic data.

[0027] A method for processing traffic data provided by an embodiment of the present application can be applied to a distributed cluster, and can be specifically implemented through a traffic data monitoring component, a distributed search engine, and nodes in the distributed cluster. The nodes in the distributed cluster may include first nodes and second nodes. Among them, the first node is the deployment point of the traffic data monitoring agent, responsible for collecting and forwarding container traffic data; the second node is the data storage point of the distributed search engine, responsible for storing and responding to query requests. The traffic data monitoring component works through the traffic data monitoring agents deployed on each first node in the distributed cluster. The first node can be a node that needs to be focused on for container traffic data and can be set according to actual needs. The main task of the traffic data monitoring agent is to obtain the container traffic data on the corresponding first node. A container is a lightweight and portable software packaging technology that allows developers to package an application and its dependencies into a portable container and then deploy it to any environment that supports Docker. Therefore, the container traffic data here refers to the network data flowing in these containers, such as the request volume, transmission rate, etc.

[0028] Specifically, the traffic data monitoring agents on each first node can forward the obtained container traffic data to the traffic data monitoring component. Subsequently, the traffic data monitoring component can forward these container traffic data to the distributed search engine. The distributed search engine determines on which second node the container traffic data should be stored based on a preset sharding strategy. Sharding is a technology that divides a large database into multiple smaller and more manageable parts, and each part can run on different servers. This can improve the storage efficiency and access speed of data. The sharding strategy can be determined based on various factors, such as the type, source, timestamp, etc. of the data, to ensure that the data is evenly distributed across all available second nodes and to avoid overloading certain nodes.

[0029] In addition to receiving container traffic data and storing it on the second node, the distributed search engine can also be used to receive a query instruction for the container traffic data input by the user. When the query instruction is received, the distributed search engine first identifies the query information specified in the query instruction (such as time range, specific container ID, etc.). Then, it uses the preset sharding strategy to determine which second nodes store the data that matches the query information, and reads the matching target container traffic data from the target second nodes, and generates a query result corresponding to the query instruction based on these data. The query result can include a direct display of the original data, or the result after aggregating and analyzing the data, which is specifically determined according to the query information.

[0030] In a specific embodiment in the field of medical and health, traffic data monitoring agents are deployed on each first node in the distributed cluster (such as the servers corresponding to each service in the medical and health information system, etc.). These traffic data monitoring agents are used to monitor and collect traffic data generated from online consultation services, electronic medical record services, patient monitoring services, etc. The collected traffic data can be forwarded to the distributed search engine. Then, the distributed search engine can adopt a preset sharding strategy to disperse and store these traffic data on multiple second nodes. The sharding strategy can be formulated based on factors such as the type of service, timestamp, patient ID, etc., to ensure the even distribution and efficient access of the data. In the field of medical and health, the query instruction can be the access peak period of each service, etc. The distributed search engine identifies these query instructions, determines the second nodes storing the relevant data based on the sharding strategy, and after reading the target traffic data from the target second nodes, the distributed search engine can generate a query result. These results can include the access peak time of the queried service, the access volume during the peak period, etc., so as to facilitate subsequent dynamic adjustment of the access strategy to cope with the emergence of access peaks.

[0031] By applying the technical solution of this embodiment, the traffic data monitoring agent on each first node can forward the obtained container traffic data to the traffic data monitoring component. Then, the traffic data monitoring component can forward this container traffic data to the distributed search engine. The distributed search engine determines on which second node the container traffic data should be stored based on a preset sharding strategy and stores it. In addition to receiving the container traffic data and storing it on the second node, the distributed search engine can also be used to receive a query instruction for the container traffic data input by the user. When the query instruction is received, the distributed search engine first identifies the query information specified in the query instruction. Then, it uses the preset sharding strategy to determine which second nodes store the data matching the query information, reads the matching target container traffic data from the target second nodes, and generates a query result corresponding to the query instruction based on this data. In the embodiment of this application, the traffic data monitoring component captures the container traffic data of each node in real time through the traffic data monitoring agents deployed in the distributed cluster and forwards this data to the distributed search engine for storage and query. In this way, not only real-time traffic monitoring is achieved, but also the efficient storage and query capabilities of the distributed search engine are fully utilized, realizing data persistent storage and efficient query of large-scale container traffic data, thus solving the deficiency of the traffic data monitoring component in data persistence.

[0032] In the embodiment of this application, optionally, after the step of "obtaining the container traffic data of each first node" in step 101, the method further includes: the traffic data monitoring component determines the container traffic time series data within a preset time based on the container traffic data, performs anomaly detection on the container traffic time series data through a preset abnormal traffic detection model to obtain a detection result. When the detection result indicates that the container traffic time series data is abnormal traffic, it obtains the traffic transmission link corresponding to the container traffic time series data, generates a network traffic topology diagram according to the traffic transmission link, inputs the network traffic topology diagram and the container traffic time series data into a preset description information extraction model to obtain the description information corresponding to the network traffic topology diagram, marks the network traffic topology diagram with the description information, and sends the marked network traffic topology diagram to the distributed search engine to store the marked network traffic topology diagram in the second node through the distributed search engine.

[0033] In this embodiment, after the traffic data monitoring component obtains the container traffic data of each first node, it can further process this data to determine the container traffic time series data within a preset time. Time series data refers to data arranged in chronological order, which records the status or value of container traffic at different time points. Then, the traffic data monitoring component uses a preset abnormal traffic detection model to perform abnormal detection on the container traffic time series data and obtains the detection result. Among them, the abnormal traffic detection model can be constructed based on machine learning methods and can identify traffic behaviors that are significantly different from normal traffic patterns. It can be seen from the detection result whether the container traffic time series data is abnormal traffic.

[0034] When the detection result indicates that the container traffic time series data is abnormal traffic, the traffic data monitoring component can obtain the traffic transmission link corresponding to the abnormal traffic. Here, the traffic transmission link refers to the path along which data flows in the network, and specifically may involve multiple containers, nodes, or network devices. Based on the obtained traffic transmission link, the traffic data monitoring component can generate a network traffic topology map. This topology map graphically shows the propagation path of the abnormal traffic in the network and the involved nodes. For example, in the field of medical and health, the generated network traffic topology map can show the propagation path of the abnormal traffic in the medical and health information system and the involved nodes (such as servers, medical devices, workstations, etc.).

[0035] Subsequently, the traffic data monitoring component can also input the network traffic topology map and the corresponding container traffic time series data into a preset description information extraction model. This model can analyze the structure and characteristics of the network traffic topology map and extract useful description information. Among them, the description information may include key information such as the type, source, and influence range of the abnormal traffic. This information helps to understand the nature and possible causes of the abnormal traffic. Using the extracted description information, the traffic data monitoring component can mark the network traffic topology map. The marked network traffic topology map contains additional information about the abnormal traffic, which is convenient for subsequent analysis and processing.

[0036] Furthermore, the traffic data monitoring component can send the marked network traffic topology map to a distributed search engine. The distributed search engine determines the second node for storing this topology map based on a preset sharding strategy and stores it in this node. The marked network traffic topology map stored in the second node can be used for subsequent query, analysis, and visualization. Medical institutions or security teams can use this information to trace the source of the abnormal traffic, analyze attack behaviors, or optimize network configurations.

[0037] In the embodiments of the present application, by adding the functions of abnormal traffic detection and network traffic topology map generation to the traffic data monitoring component, the ability to deeply analyze and process container traffic data is improved. It can not only identify abnormal traffic behaviors in the network, but also graphically display the propagation path of abnormal traffic and the involved nodes. In addition, descriptive information for the network traffic topology map is generated and stored. Subsequently, this information is very valuable for medical institutions or security teams, which can help them better understand network traffic behaviors, timely discover potential security threats, and take corresponding measures to protect the security of patient data and medical systems.

[0038] In the embodiments of the present application, optionally, the preset descriptive information extraction model includes a first neural network, a second neural network, a gating mechanism, a feature fusion network, and a descriptive information generation network; the step of "inputting the network traffic topology map and the container traffic time series data into the preset descriptive information extraction model to obtain the descriptive information corresponding to the network traffic topology map" includes: inputting the network traffic topology map into the first neural network, and through the first neural network, respectively extracting the node features of each node and the edge features of each edge in the network traffic topology map, and generating the global topology features of the network traffic topology map according to the node features and the edge features; inputting the container traffic time series data into the second neural network, and through the second neural network, extracting the time series features of the container traffic time series data; based on the global topology features and the time series features, through the gating mechanism, respectively calculating the first weight corresponding to the global topology features and the second weight corresponding to the time series features; inputting the global topology features, the time series features, the first weight, and the second weight into the feature fusion network to obtain target fusion features; and inputting the target fusion features into the descriptive information generation network, and generating the descriptive information corresponding to the network traffic topology map based on the preset fields in the descriptive information generation network.

[0039] In this embodiment, the preset descriptive information extraction model may include a first neural network, a second neural network, a gating mechanism, a feature fusion network, and a descriptive information generation network. The preset descriptive information extraction model can be used to extract features and generate descriptive information from the network traffic topology map and the container traffic time series data.

[0040] Among them, the first neural network is used to extract the features of the network traffic topology graph. After receiving the network traffic topology graph, it extracts the node features of each node in the network traffic topology graph (such as the type of the node, the traffic size, the number of connections, etc.) and the edge features of each edge (such as the weight of the edge, the direction, the type of data transmitted, etc.). Subsequently, according to the node features and the edge features, it generates the global topology features of the network traffic topology graph. The global topology features are a high-level and comprehensive description of the entire network traffic topology graph.

[0041] The second neural network is used to extract the features of the container traffic time-series data. After receiving the container traffic time-series data, it extracts the time-series features of the container traffic time-series data, which can include the change trend of the traffic, periodicity, abnormal fluctuations, etc.

[0042] The gating mechanism is a mechanism for dynamically adjusting the importance of features. The gating mechanism can calculate the corresponding first weight and second weight according to the importance of the global topology features and the time-series features respectively. These weights reflect the importance of different features when describing the network traffic topology graph.

[0043] The feature fusion network is used to fuse the global topology features and the time-series features, and their weights can be considered simultaneously during the fusion process. The fusion process can be specifically implemented based on various methods such as weighted summation, concatenation, attention mechanism, etc. After fusion, the target fusion features can be output. The target fusion features are a comprehensive representation of the global topology features and the time-series features, and they contain the important information of both.

[0044] The description information generation network is used to convert the target fusion features into specific description information. These description information can include key information such as the type, source, and influence range of abnormal traffic, as well as the overall overview of the network traffic topology graph. These description information are presented in the form of preset fields, which is convenient for subsequent analysis and processing.

[0045] In the embodiment of the present application, optionally, after receiving the query information, the target second node queries whether there is a matching target description information in the stored marked network traffic topology graph based on the query information, and when there is a matching target description information, it returns the network traffic topology graph corresponding to the target description information to the distributed search engine, otherwise, it returns the target container traffic data matching the query information to the distributed search engine.

[0046]

[0047] ​In this embodiment, when the target second node receives the query information, it can start to execute the query operation. The query information here may include a description of the content that the user wants to query, such as finding the traffic anomaly situation of the medical and health information system within a specific time period, etc. At this time, the second node may be a node storing the traffic data of each service in the medical and health information system.

[0048] Next, the target second node can query whether there is matching target description information from the stored marked network traffic topology map. Specifically, it can calculate the similarity between the query information and the description information of each network traffic topology map, and use the description information with the largest similarity and greater than the preset similarity threshold as the target description information. If the target second node finds the target description information matching the query information in the marked network traffic topology map, it can return the corresponding network traffic topology map to the distributed search engine. If the target second node does not find the matching target description information, it can return the target container traffic data matching the query information to the distributed search engine. For example, if the user wants to query the traffic data anomaly situation of the online consultation service in the medical and health information system between March 1st and March 20th, the distributed search engine can first determine the target second node corresponding to the traffic data of the online consultation service in the medical and health information system between March 1st and March 20th, and then check whether there is target description information matching the traffic data anomaly situation of the online consultation service in the target second node. If it exists, it directly returns the network traffic topology map corresponding to the target description information. At this time, the user can intuitively see the corresponding network anomaly situation from the network traffic topology map, and at the same time, can further understand the anomaly situation from the description information corresponding to the network traffic topology map. Through this method, the embodiment of the present application can first query the result from the network traffic topology map stored in the second node. If there is a matching network traffic topology map, it directly returns the network traffic topology map to the user, which can make the query result more accurate and intuitive.

[0049] In an embodiment of the present application, optionally, after the step of "the distributed search engine responds to a query instruction for container traffic data and identifies the query information corresponding to the query instruction" in step 103, the method further includes: the distributed search engine parses the query information. If the parsing result indicates that the query information belongs to a composite query structure, it determines a query result generation tool corresponding to the composite query structure, and determines whether there is a query result corresponding to the query information from the matching cache resource pool through the query result generation tool. When there is a query result corresponding to the query information, the cached query result is used as the query result corresponding to the query instruction; otherwise, based on the preset sharding strategy, it determines the target second node(s) that the query information matches, reads the target container traffic data from the target second node(s), and generates the query result corresponding to the query instruction through the query result generation tool based on the target container traffic data.

[0050] In this embodiment, after receiving a query instruction for container traffic data, the distributed search engine can identify and parse this query instruction to understand the specific information that the user wants to query. Next, the distributed search engine determines whether the parsed query information belongs to a composite query structure. A composite query structure refers to a query structure that requires statistical analysis or operations on the basis of container traffic data, rather than directly querying the container traffic data itself. For example, if a user wants to query the high-frequency access time periods of the online consultation service in the medical and health information system, then statistical analysis needs to be performed on the basis of container traffic data, and such a query structure is a composite query structure; if a user wants to query the traffic conditions of the online consultation service in the medical and health information system during a certain period, then it does not belong to a composite query structure.

[0051] If the query is composite, then a query result generation tool corresponding to the composite query structure can be determined from multiple preset query result generation tools, and the query result generation tool is used to first attempt to obtain the result from the cache resource pool. If the cache resource pool stores the results of the same query executed previously, then the result is directly returned from the cache resource pool without having to obtain data and perform calculations again, which can greatly improve the query efficiency.

[0052] If no matching result is found in the cache resource pool, the distributed search engine will determine on which second node(s) the query information should be executed based on the preset sharding strategy. Once the target second node(s) is / are determined, the search engine reads the target container traffic data from these node(s). Finally, using the query result generation tool, the final query result is generated based on the read target container traffic data. This result can be a dataset that has been filtered, sorted, or aggregated, directly answering the user's query requirements.

[0053] In an embodiment of the present application, optionally, the traffic data monitoring component corresponds to a configuration file. Before the traffic data monitoring component forwards the container traffic data to the distributed search engine, the configuration file is configured to enable the output plugin of the distributed search engine and bind the target address and target port of the distributed search engine.

[0054] In this embodiment, the configuration file is an important part of the traffic data monitoring component, which defines the behavior and parameters of the traffic data monitoring component. Before using the traffic data monitoring component for traffic monitoring, the output plugin of the distributed search engine can be enabled in the configuration file. At the same time, the target address and target port information of the distributed search engine can also be bound in the configuration file. These information can tell the traffic data monitoring component which port of which distributed search engine the container traffic data should be sent to. The target address is usually the IP address or domain name of the server where the distributed search engine is located, while the target port is the network port where the distributed search engine listens for data. In this way, when the traffic data monitoring component captures the container traffic data, it can send the container traffic data to the specified target address and target port through the enabled output plugin of the distributed search engine according to the settings in the configuration file, ensuring that the data can reach the distributed search engine safely and accurately.

[0055] In the embodiment of the present application, by enabling the output plugin of the distributed search engine and binding the target address and target port through the configuration file, seamless docking between the traffic data monitoring component and the distributed search engine is achieved, which not only improves the efficiency and accuracy of container traffic data forwarding, but also reduces the complexity and cost of system operation and maintenance.

[0056] In an embodiment of the present application, optionally, before step 101, the method further includes: the traffic data monitoring component identifies the first field corresponding to the container traffic data and the first data type of the field value corresponding to the first field, and queries the second data type matched by the first field in the preset index structure of the distributed engine. When the first data type is different from the second data type, a preset field value conversion tool is called to convert the field value corresponding to the first field from the first data type to the second data type, and the converted container traffic data is forwarded to the distributed search engine.

[0057] In this embodiment, in order to ensure the compatibility between the container traffic data output by the traffic data monitoring component and the preset index structure of the distributed search engine, the data type corresponding to the field value can be adjusted before forwarding the container traffic data to the distributed search engine. Specifically, when the traffic data monitoring component receives the container traffic data, it first identifies the fields included in the data (referred to as the first fields here). For each first field, the traffic data monitoring component can also determine the data type of the corresponding field value (referred to as the first data type here). The data type can include integers, floating-point numbers, strings, dates, etc. Next, the traffic data monitoring component can query the information about these fields in the preset index structure of the distributed search engine. The preset index structure is used to store the mapping information between the data types of the field values that the distributed search engine can recognize and the fields. The traffic data monitoring component can find the data type (referred to as the second data type here) corresponding to the first field and recognizable by the distributed search engine in the preset index structure. It should be noted that the traffic data monitoring component can pre-store the mapping data in the preset index structure of the distributed search engine. Then, the traffic data monitoring component compares the first data type and the second data type. If they are the same, it means that the data type of the field value is consistent with the data type recorded in the preset index structure, and no conversion is required; if the first data type and the second data type are different, then data type conversion is needed. Specifically, the traffic data monitoring component can call a preset field value conversion tool. This tool can be a program or a function library, which can convert the field value from one data type to another according to the first data type and the second data type. Through the preset field value conversion tool, the traffic data monitoring component can convert the field value corresponding to the first field from the first data type to the second data type. After completing the data type conversion, the traffic data monitoring component can update the field value corresponding to the first field in the container traffic data to make it consistent with the data type recorded in the preset index structure. After conversion, it is forwarded to the distributed search engine, so that the distributed search engine can recognize these container traffic data.

[0058] Further, as Figure 1 a specific implementation of the method, an embodiment of the present application provides a processing system for traffic data, as Figure 2 shown, the system includes:

[0059] A traffic data monitoring component, configured to obtain the container traffic data of each first node through the traffic data monitoring agent deployed on each first node in the distributed cluster, and forward the container traffic data to the distributed search engine;

[0060] The distributed search engine is used to determine a second node corresponding to the container traffic data based on a preset sharding strategy, and send the container traffic data to the second node for storage;

[0061] The distributed search engine is further used to, in response to a query instruction for container traffic data, identify query information corresponding to the query instruction, determine a target second node matching the query information based on the preset sharding strategy, read target container traffic data from the target second node, and generate a query result corresponding to the query instruction based on the target container traffic data.

[0062] Optionally, the traffic data monitoring component is further used to:

[0063] After obtaining the container traffic data of each first node, based on the container traffic data, determine container traffic time series data within a preset time, perform anomaly detection on the container traffic time series data through a preset abnormal traffic detection model to obtain a detection result. When the detection result indicates that the container traffic time series data is abnormal traffic, obtain a traffic transmission link corresponding to the container traffic time series data, generate a network traffic topology diagram according to the traffic transmission link, input the network traffic topology diagram and the container traffic time series data into a preset description information extraction model to obtain description information corresponding to the network traffic topology diagram, mark the network traffic topology diagram with the description information, and send the marked network traffic topology diagram to the distributed search engine to store the marked network traffic topology diagram in the second node through the distributed search engine.

[0064] Optionally, the preset description information extraction model includes a first neural network, a second neural network, a gating mechanism, a feature fusion network, and a description information generation network; the traffic data monitoring component is further used to:

[0065] Input the network traffic topology diagram into the first neural network, and through the first neural network, extract node features of each node and edge features of each edge in the network traffic topology diagram, and generate global topology features of the network traffic topology diagram according to the node features and the edge features;

[0066] Input the container traffic time series data into the second neural network, and through the second neural network, extract time series features of the container traffic time series data;

[0067] Based on the global topology features and the time series features, through the gating mechanism, calculate a first weight corresponding to the global topology features and a second weight corresponding to the time series features respectively;

[0068] Input the global topological features, the temporal features, the first weight, and the second weight into the feature fusion network to obtain the target fusion features;

[0069] Input the target fusion features into the description information generation network, and generate the description information corresponding to the network traffic topology map based on the preset fields in the description information generation network.

[0070] Optionally, the target second node is used for:

[0071] After receiving the query information, based on the query information, query whether there is a matching target description information in the stored marked network traffic topology map, and when there is a matching target description information, return the network traffic topology map corresponding to the target description information to the distributed search engine, otherwise, return the target container traffic data matching the query information to the distributed search engine.

[0072] Optionally, the distributed search engine is further used for:

[0073] In response to a query instruction for container traffic data, after identifying the query information corresponding to the query instruction, parse the query information. If the parsing result indicates that the query information belongs to a composite query structure, determine the query result generation tool corresponding to the composite query structure, and determine whether there is a query result corresponding to the query information in the matching cache resource pool through the query result generation tool. When there is a query result corresponding to the query information, use the cached query result as the query result corresponding to the query instruction, otherwise, based on the preset sharding strategy, determine the target second node matching the query information, read the target container traffic data from the target second node, and generate the query result corresponding to the query instruction through the query result generation tool based on the target container traffic data.

[0074] Optionally, the traffic data monitoring component corresponds to a configuration file. Before the traffic data monitoring component forwards the container traffic data to the distributed search engine, the configuration file is configured to enable the distributed search engine output plugin and bind the target address and target port of the distributed search engine.

[0075] Optionally, the traffic data monitoring component is further used for:

[0076] Before forwarding the container traffic data to the distributed search engine, identify a first field corresponding to the container traffic data and a first data type of a field value corresponding to the first field, and query a second data type matched by the first field in a preset index structure of the distributed engine. When the first data type is different from the second data type, call a preset field value conversion tool to convert the field value corresponding to the first field from the first data type to the second data type, and forward the converted container traffic data to the distributed search engine.

[0077] It should be noted that for other corresponding descriptions of each functional unit involved in the traffic data processing system provided in the embodiments of the present application, reference can be made to Figure 1 the corresponding descriptions in the method, which will not be elaborated here.

[0078] The embodiments of the present application further provide a computer device, which may specifically be a personal computer, a server, a network device, etc. As Figure 3 shown, the computer device includes a bus, a processor, a memory, and a communication interface, and may further include an input / output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the method embodiments are implemented.

[0079] Those skilled in the art can understand that Figure 3 the structure shown in

[0080] merely represents a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0081] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0083] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0084] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0085] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for processing traffic data, characterized in that Including: The traffic data monitoring component obtains the container traffic data of each first node through the traffic data monitoring agents deployed on each first node in the distributed cluster, and forwards the container traffic data to the distributed search engine; The distributed search engine determines the second node corresponding to the container traffic data based on the preset sharding strategy, and sends the container traffic data to the second node for storage; The distributed search engine responds to the query instruction of the container traffic data, identifies the query information corresponding to the query instruction, and based on the preset sharding strategy, determines the target second node matching the query information, reads the target container traffic data from the target second node, and generates the query result corresponding to the query instruction based on the target container traffic data.

2. The method according to claim 1, characterized in that, After obtaining the container traffic data of each first node, the method further includes: The traffic data monitoring component determines the container traffic time series data within a preset time based on the container traffic data, performs anomaly detection on the container traffic time series data through a preset abnormal traffic detection model to obtain a detection result. When the detection result indicates that the container traffic time series data is abnormal traffic, it obtains the traffic transmission link corresponding to the container traffic time series data, generates a network traffic topology diagram according to the traffic transmission link, inputs the network traffic topology diagram and the container traffic time series data into a preset description information extraction model to obtain the description information corresponding to the network traffic topology diagram, marks the network traffic topology diagram with the description information, and sends the marked network traffic topology diagram to the distributed search engine to store the marked network traffic topology diagram in the second node through the distributed search engine.

3. The method according to claim 2, wherein The preset description information extraction model includes a first neural network, a second neural network, a gating mechanism, a feature fusion network, and a description information generation network; the step of inputting the network traffic topology diagram and the container traffic time series data into the preset description information extraction model to obtain the description information corresponding to the network traffic topology diagram includes: Input the network traffic topology diagram into the first neural network. Through the first neural network, extract the node features of each node and the edge features of each edge in the network traffic topology diagram respectively, and generate the global topology features of the network traffic topology diagram according to the node features and the edge features; Input the container traffic time series data into the second neural network. Through the second neural network, extract the time series features of the container traffic time series data; Based on the global topology features and the time series features, through the gating mechanism, calculate the first weight corresponding to the global topology features and the second weight corresponding to the time series features respectively; Input the global topology features, the time series features, the first weight, and the second weight into the feature fusion network to obtain the target fusion features; Input the target fusion feature into the description information generation network, and generate the description information corresponding to the network traffic topology diagram based on the preset fields in the description information generation network.

4. The method according to claim 2 or 3, wherein After receiving the query information, the target second node queries whether there is matching target description information in the stored marked network traffic topology diagram based on the query information, and when there is matching target description information, returns the network traffic topology diagram corresponding to the target description information to the distributed search engine; otherwise, returns the target container traffic data matching the query information to the distributed search engine.

5. The method according to claim 1, wherein After the distributed search engine identifies the query information corresponding to the query instruction in response to the query instruction for container traffic data, the method further includes: The distributed search engine parses the query information. If the parsing result indicates that the query information belongs to a composite query structure, it determines the query result generation tool corresponding to the composite query structure, and determines whether there is a query result corresponding to the query information in the matching cache resource pool through the query result generation tool. When there is a query result corresponding to the query information, the cached query result is used as the query result corresponding to the query instruction; otherwise, based on the preset sharding strategy, it determines the target second node matching the query information, reads the target container traffic data from the target second node, and generates the query result corresponding to the query instruction through the query result generation tool based on the target container traffic data.

6. The method according to claim 1, characterized in that, The traffic data monitoring component corresponds to a configuration file. Before the traffic data monitoring component forwards the container traffic data to the distributed search engine, the configuration file is configured to enable the distributed search engine output plugin and bind the target address and target port of the distributed search engine.

7. The method according to claim 1, characterized in that Before forwarding the container traffic data to the distributed search engine, the method further includes: The traffic data monitoring component identifies the first field corresponding to the container traffic data and the first data type of the field value corresponding to the first field, and queries the second data type matching the first field in the preset index structure of the distributed engine. When the first data type is different from the second data type, it calls a preset field value conversion tool to convert the field value corresponding to the first field from the first data type to the second data type, and forwards the converted container traffic data to the distributed search engine.

8. A processing system for traffic data, characterized in that, Including: A traffic data monitoring component, configured to obtain the container traffic data of each first node through the traffic data monitoring agent deployed on each first node in the distributed cluster, and forward the container traffic data to the distributed search engine; The distributed search engine is configured to determine the second node corresponding to the container traffic data based on the preset sharding strategy, and send the container traffic data to the second node for storage; The distributed search engine is further configured to, in response to a query instruction for container traffic data, identify query information corresponding to the query instruction, determine a target second node matching the query information based on the preset sharding policy, read target container traffic data from the target second node, and generate a query result corresponding to the query instruction based on the target container traffic data.

9. A storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images