A dynamically predictable data traffic detection and classification method

Abstract

The present invention discloses a dynamically predictable data flow detection and classification method, which relates to the field of data processing technology and solves the technical problem that in the prior art, a large amount of data processing and calculation are required when performing data flow monitoring and classification, and data flow detection and classification cannot be performed efficiently, resulting in low data detection and classification efficiency and poor classification effect. The present invention obtains basic features of multi-source data, and performs periodic classification according to data flow, dynamically configures detection nodes according to the classification results, obtains corresponding data labels through probabilistic analysis of the detection nodes, and then determines the classification labels of the multi-source data through a target knowledge graph, thereby realizing fast and efficient classification of the multi-source data. The present invention performs coarse classification of the multi-source data through periodic classification, and then performs fine classification through an artificial intelligence model or a domain mapping model, and finally generates classification labels in combination with knowledge graph technology, thereby realizing classification of multi-source data from different angles and improving diversion efficiency.

Description

Technical Field

[0001] The invention belongs to the field of data processing, and relates to a dynamically predictable data flow detection and classification technology, in particular to a dynamically predictable data flow detection and classification method. Background Art

[0002] Real-time analysis application scenarios for multi-source heterogeneous massive data are becoming more and more common. How to intelligently identify and classify unknown multi-source data streams to ensure subsequent business operations is a major challenge.

[0003] The prior art (invention patent with application number 2020108557201) discloses a dynamically predictable data flow detection and classification method and device, which obtains training samples by analyzing multi-source heterogeneous data, and mixes the flow session feature vector and regression model prediction vector of the training sample for deep learning, so as to realize intelligent recognition and refined classification of multi-source heterogeneous data streams. The prior art requires a large amount of data processing and calculation when performing data flow monitoring and classification, and cannot efficiently perform data flow detection and classification, resulting in low data detection and classification efficiency and poor classification effect; therefore, a dynamically predictable data flow detection and classification method is urgently needed. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a dynamically predictable data traffic detection and classification method, which is used to solve the technical problem that the prior art requires a large amount of data processing and calculation when performing data traffic monitoring and classification, and is unable to efficiently perform data traffic detection and classification, resulting in low data detection and classification efficiency and poor classification effect.

[0005] The present invention obtains basic features of multi-source data, configures detection nodes according to the basic features and data flow, analyzes the basic features through the configured detection nodes to obtain data labels, and calls the corresponding target knowledge graph to obtain classification labels of multi-source data from the target knowledge graph. The multi-source data can be dynamically detected and diverted to ensure the classification efficiency and classification effect of the multi-source data.

[0006] To achieve the above object, a first aspect of the present invention provides a dynamically predictable data traffic detection and classification method, comprising:

[0007] Acquire a number of multi-source data, and analyze the multi-source data to acquire corresponding basic features; wherein the basic features include data domain, data type and data source;

[0008] Dynamically configuring at least one detection node according to the basic features and data traffic, wherein the detection node performs a probabilistic analysis on the basic features to determine a data label;

[0009] The target knowledge graph is called according to the data label, and the classification label corresponding to the multi-source data is obtained according to the target knowledge graph; and the multi-source data is classified according to the classification label.

[0010] Preferably, configuring the detection node according to the basic characteristics and data flow includes:

[0011] Acquire the basic features corresponding to a number of the multi-source data;

[0012] Periodically classifying the multi-source data according to the basic features, and obtaining the total amount of data after periodic classification;

[0013] The detection nodes are configured in levels according to the total amount of data.

[0014] Preferably, multi-source data is periodically classified according to data flow, and a plurality of detection nodes are configured in different levels, including:

[0015] Classify the multi-source data corresponding to the basic features according to a set period, count the total amount of data of the classification results, and mark it as SZ; wherein the set period includes 1 second or 1 minute;

[0016] The configuration number JPS of the detection node is obtained according to the formula JPS=α×SZ / BS; wherein BS is the data processing amount of the detection node within a set period, and α is a proportional coefficient greater than 0, and the default value is 1;

[0017] The detection nodes are configured according to the configuration quantity JPS.

[0018] Preferably, the plurality of detection nodes configured according to the configuration quantity JPS are regarded as a detection point set, and the plurality of detection nodes in the detection point set cooperate with each other.

[0019] Preferably, the detection node performs a probabilistic analysis on the basic features based on an artificial intelligence model to obtain the corresponding data labels, including:

[0020] Converting the basic features to obtain model input data;

[0021] Input the model input data into the artificial intelligence model, obtain the output subdivision field, and obtain the data label corresponding to the subdivision field; wherein the artificial intelligence model includes a neural network model or a support vector machine model.

[0022] Preferably, the detection node performs a probabilistic analysis on the basic features based on a domain mapping model to obtain the corresponding data labels, including:

[0023] Acquire the domain mapping model; wherein the domain mapping model includes a mapping relationship between the basic features and the subdivided domains;

[0024] Based on the domain mapping model, the basic features are compared and analyzed one by one to determine the subdivided domain, and the data label corresponding to the subdivided domain is obtained.

[0025] Preferably, obtaining classification labels corresponding to the multi-source data according to the target knowledge graph includes:

[0026] Call the domain knowledge graph corresponding to the data label and mark it as the target knowledge graph;

[0027] The multi-source data content is parsed, and the parsing results are combined with the target knowledge graph to determine the corresponding classification labels.

[0028] Preferably, the domain knowledge graph is established and acquired based on a subdivided domain, or is divided and acquired from a knowledge graph established based on a data domain; wherein the coverage of the data domain is larger than that of the subdivided domain.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. After acquiring multi-source data, the present invention obtains its basic features, and performs periodic classification according to the data flow, dynamically configures the detection node according to the classification results, calls the artificial intelligence model or the domain mapping model through the detection node to obtain the corresponding data label, and then determines the classification label of the multi-source data through the target knowledge graph, thereby realizing fast and efficient classification of multi-source data.

[0031] 2. The present invention roughly classifies multi-source data through periodic classification, and then performs fine classification through artificial intelligence models or domain mapping models. Finally, it combines knowledge graph technology to generate classification labels, thereby realizing the classification of multi-source data from different angles and improving the diversion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is a schematic diagram of the working steps of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The existing technology requires a large amount of data processing and calculation when performing data flow monitoring and classification. For example, it is necessary to extract training samples from multi-source heterogeneous data, and obtain the conversation flow feature vectors and regression model prediction vectors of the training samples. It is impossible to efficiently perform data flow detection and classification, resulting in low data detection and classification efficiency and poor classification effect.

[0036] After acquiring multi-source data, the present invention obtains its basic features and performs periodic classification according to the data flow, dynamically configures the detection nodes according to the classification results, calls the artificial intelligence model or the domain mapping model through the detection node to obtain the corresponding data labels, and then determines the classification labels of the multi-source data through the target knowledge graph, thereby realizing fast and efficient classification of multi-source data.

[0037] See also Figure 1 The first aspect of the present invention provides a dynamically predictable data traffic detection and classification method, comprising:

[0038] Acquire some multi-source data, and analyze the multi-source data to obtain corresponding basic features;

[0039] At least one detection node is dynamically configured according to the basic features and data flow, and the detection node performs probabilistic analysis on the basic features to determine the data label;

[0040] The target knowledge graph is called according to the data labels, and the classification labels of the corresponding multi-source data are obtained according to the target knowledge graph; the multi-source data are classified according to the classification labels.

[0041] The main idea of ​​the present invention application is to roughly divide the multi-source data from the data field, data type, data source, etc. of the multi-source data after acquiring the multi-source data, set a reasonable number of detection nodes in combination with the data flow to obtain the data labels of the multi-source data, and then call the target knowledge graph to perform fine division of the multi-source data.

[0042] Multi-source data refers to data with different data structures obtained from different data sources. Taking the manufacturing production process as an example, the data sources include equipment maintenance systems, energy consumption management systems, production monitoring systems, quality inspection systems, etc., and the data structures include structured data, unstructured data, and semi-structured data.

[0043] One of the core is to preliminarily divert multi-source data according to basic features, so the extraction of basic features is particularly important. In the present invention application, basic features include data fields, data types, data sources, and other features that can express the overall attributes of the data, and the data field can be understood as the industry classification to which the multi-source data belongs (refer to the National Economic Industry Classification and Code (GB / T4754-2017)). Therefore, the more basic features there are, the more refined the division of multi-source data. After the basic feature division, the multi-source data can be further divided and diverted through the domain knowledge graph.

[0044] It can be understood that the multi-source data mentioned in the present application refers to all data received by the server in the case of return analysis or other overall analysis. In the specific analysis process, multi-source data refers to a single data, that is, a certain piece or a certain type of data.

[0045] It is worth noting that the data domain of multi-source data is larger than the sub-domain, that is, the data domain and the above sub-domains are dynamic. If the data domain is a primary domain, the sub-domain can be a secondary domain; if the data domain is a secondary domain, the sub-domain can be a tertiary domain.

[0046] The present invention configures detection nodes according to basic characteristics and data flow, including:

[0047] Obtain basic features corresponding to a number of multi-source data; periodically classify the multi-source data according to the basic features, and obtain the total amount of data after periodic classification; and configure detection nodes by level according to the total amount of data.

[0048] The detection node is equivalent to the data processing node, which is equivalent to the data computing power allocated by the server. It can be understood that the detection node always exists, but it is supplemented with the increase of the total amount of data corresponding to the multi-source data within the set period to meet the data processing needs.

[0049] When classifying multi-source data according to basic features, the multi-source data can be classified only according to the corresponding data field. If the data field cannot effectively divide the multi-source data, it can be classified in combination with other basic features.

[0050] In an optional embodiment, multi-source data is classified in real time, that is, a detection node is configured by default to analyze and process a type of multi-source data, and the total amount of data of this type of multi-source data is counted in real time. If an existing detection node cannot meet the data processing requirements, the detection node is reasonably configured according to the total amount of data. It can be understood that this embodiment has high requirements for dynamic adjustment capabilities.

[0051] In another optional embodiment, multi-source data is periodically classified according to data flow, and a plurality of detection nodes are configured in different levels, including:

[0052] The multi-source data corresponding to the basic features are classified according to the set period, the total amount of data of the classification results is counted and marked as SZ; the configuration number JPS of the detection node is obtained according to the formula JPS=α×SZ / BS; the detection node is configured according to the configuration number JPS.

[0053] This embodiment uses a set period as a boundary, counts the total amount of multi-source data within a set period, and reasonably configures the detection nodes based on the data processing capacity of each detection node in a set period. For example: assuming that the total amount of data of a certain type of multi-source data within 1 second is 10M, and each detection node can process 1M of data within 1 second, 10 detection nodes need to be configured to efficiently process the data.

[0054] It is worth noting that according to the configuration number, several detection nodes configured by JPS are regarded as a detection point set, and several detection nodes in the detection point set cooperate with each other to ensure efficient data processing. After the data processing within the set period is completed, detection nodes are added or deleted according to the total amount of data in the next set period to achieve dynamic adjustment of server computing power.

[0055] The detection node in the present invention performs probabilistic analysis on the basic features based on the artificial intelligence model to obtain the corresponding data labels, including:

[0056] Convert the basic features to obtain model input data; input the model input data into the artificial intelligence model to obtain the output segmentation field and the data label corresponding to the segmentation field.

[0057] After configuring the detection node, the basic features are converted into model input data that can be recognized by the artificial intelligence model, and the subdivision field with the highest probability of the model output result is determined as the subdivision field corresponding to the multi-source data. The artificial intelligence model actually determines which subdivision field the corresponding multi-source data belongs to based on the basic features. The more basic features, the easier it is to determine the subdivision field.

[0058] The artificial intelligence model is obtained through training with historical experience data, which includes several sets of training data, each of which includes basic features and corresponding subdivisions (or the probability value of belonging to a certain subdivision). It is understandable that the fewer basic features there are, the more difficult it is to determine the subdivision, so the corresponding subdivision can be determined by probability.

[0059] The sub-segment in the historical experience data can be a single value, such as the number corresponding to a certain sub-segment, or a sequence composed of several sub-segments and corresponding probabilities. The historical experience data is mainly obtained through statistics.

[0060] Assume that the basic features include data type 1 and data source 1; in historical data, when the basic features include both data type 1 and data source 1, the corresponding probability of segmentation field 1 is GL1, and the corresponding probability of segmentation field 2 is GL2. Then the set of training data includes [data type 1, data source 1, segmentation field 1, GL1, data source 2, GL2]. When training the artificial intelligence model, [data type 1, data source 1] is used as input data, and [segmentation field 1, GL1, data source 2, GL2] is used as output data.

[0061] Here is an example of how the AI ​​model works:

[0062] Assume that there are multi-source data 1 and multi-source data 2. The basic features of multi-source data 1 include data type and data type, and the basic features of multi-source data 2 only include data source;

[0063] When the artificial intelligence model analyzes multi-source data 1, it determines that the probability of it belonging to subdivision field 1 is 0.8, and the probability of it belonging to subdivision field 2 is 0.4, so the subdivision field of the multi-source data is determined to be subdivision field 1;

[0064] When the artificial intelligence model analyzes multi-source data 2, it determines that the probability of it belonging to a subdivision is 0.5, and the probability of it belonging to subdivision 2 is 0.5. Therefore, the subdivision of the multi-source data may be subdivision 1 or subdivision 2.

[0065] In another preferred embodiment, the detection node performs probabilistic analysis on the basic features based on the domain mapping model to obtain corresponding data labels, including:

[0066] Obtain a domain mapping model; compare and analyze basic features one by one based on the domain mapping model, determine the subdivision domain, and obtain the data labels corresponding to the subdivision domain.

[0067] The domain mapping model includes the mapping relationship between basic features and subdivided domains. As mentioned above, the more basic features there are, the more detailed the subdivided domains can be. Its working principle is as follows:

[0068] Obtain all basic features, and search all basic feature groups as a whole in the domain mapping model. If the searched subdivision field can meet the requirements with corresponding probability (for example, the probability of the two matching is above 0.8), then the subdivision field is the subdivision field corresponding to the multi-source data;

[0069] If no suitable subdivision is found through all basic features, the basic features are deleted and the search is repeated to ensure that the most suitable subdivision is obtained from multi-source data.

[0070] In the present invention, the classification labels of corresponding multi-source data are obtained according to the target knowledge graph, including:

[0071] Call the domain knowledge graph corresponding to the data label and mark it as the target knowledge graph; parse the multi-source data content, and combine the parsing results with the target knowledge graph to determine the corresponding classification label.

[0072] The corresponding target knowledge graph is called according to the data label, and then the content of the multi-source data is parsed to obtain keywords. According to the keywords, several entities corresponding to the multi-source data and the relationships between several entities are retrieved in the target knowledge graph.

[0073] The domain knowledge graph is established based on the subdivided domain, that is, a number of entities and the relationship between several entities are analyzed and extracted under the subdivided domain, and then the domain knowledge graph corresponding to the subdivided domain is established. It is also possible to establish an overall knowledge graph based on the data field, and then divide the domain knowledge graph according to the subdivided domain.

[0074] It can be understood that the classification labels corresponding to multi-source data can be represented by digital sequences. The classification labels include data fields, sub-fields, and entities involved. Based on the classification labels, multi-sources can be divided according to entities.

[0075] The technical solution of the present invention is illustrated by examples:

[0076] Assuming that multi-source data is obtained, the basic features corresponding to the multi-source data are the power field, structured data, and smart meters;

[0077] The multi-source data is classified according to the power field, and then analyzed by the configured detection node combined with other basic features to determine that the power subdivision field corresponding to the multi-source data is power maintenance;

[0078] The content of the multi-source data is parsed, and according to the parsed content, it is matched in the target knowledge graph corresponding to power maintenance, the corresponding entities (such as jurisdiction units, maintenance personnel, etc.) are determined, and analysis labels are comprehensively generated.

[0079] It is worth noting that whether it is periodic classification of multi-source data or analysis of multi-source data based on artificial intelligence models or domain mapping models, classification is based on the basic characteristics of multi-source data. If the amount of data is small and the basic characteristics are sufficient to determine the sub-sectors, the analysis steps through artificial intelligence models or domain mapping models can be omitted.

[0080] Some of the data in the above formula are calculated by removing the dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0081] Working principle of the present invention:

[0082] Acquire some multi-source data, analyze the multi-source data to acquire corresponding basic features; dynamically configure at least one detection node according to the basic features and data flow, and the detection node performs probabilistic analysis on the basic features to determine data labels.

[0083] The target knowledge graph is called according to the data labels, and the classification labels of the corresponding multi-source data are obtained according to the target knowledge graph; the multi-source data are classified according to the classification labels.

[0084] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A dynamically predictable data traffic detection and classification method, characterized in that: include: Acquire a number of multi-source data, and analyze the multi-source data to acquire corresponding basic features; wherein the basic features include data domain, data type and data source; Dynamically configuring at least one detection node according to the basic features and data flow, the detection node performs probabilistic analysis on the basic features to determine the data label; wherein the detection node is equivalent to a data processing node; Calling a target knowledge graph according to the data label, and obtaining classification labels corresponding to the multi-source data according to the target knowledge graph; classifying the multi-source data according to the classification labels; Configuring the detection node according to the basic characteristics and data flow includes: Acquire the basic features corresponding to a number of the multi-source data; Periodically classifying the multi-source data according to the basic features, and obtaining the total amount of data after periodic classification; The detection nodes are configured in different levels according to the total amount of data; The detection node performs a probabilistic analysis on the basic features based on the artificial intelligence model to obtain the corresponding data labels, including: Converting the basic features to obtain model input data; Inputting the model input data into the artificial intelligence model, obtaining the output subdivision field, and obtaining the data label corresponding to the subdivision field; wherein the artificial intelligence model includes a neural network model or a support vector machine model; The detection node performs a probabilistic analysis on the basic features based on the domain mapping model to obtain the corresponding data labels, including: Acquire the domain mapping model; wherein the domain mapping model includes a mapping relationship between the basic features and the subdivided domains; Based on the domain mapping model, the basic features are compared and analyzed one by one to determine the subdivided domain, and the data label corresponding to the subdivided domain is obtained.

2. A dynamically predictable data traffic detection and classification method according to claim 1, characterized in that: Multi-source data is periodically classified according to data traffic, and most detection nodes are configured in different levels, including: Classify the multi-source data corresponding to the basic features according to a set period, count the total amount of data of the classification results, and mark it as SZ; wherein the set period includes 1 second or 1 minute; The configuration number JPS of the detection node is obtained according to the formula JPS=α×SZ / BS; wherein BS is the data processing amount of the detection node within a set period, and α is a proportional coefficient greater than 0, and the default value is 1; The detection node is configured according to the configuration quantity JPS.

3. A dynamically predictable data flow detection and classification method according to claim 2, characterized in that: The detection nodes configured according to the configuration quantity JPS are regarded as a detection point set, and the detection nodes in the detection point set cooperate with each other.

4. The method for dynamically predicting data flow detection and classification according to claim 1, characterized in that: Acquiring classification labels corresponding to the multi-source data according to the target knowledge graph includes: Call the domain knowledge graph corresponding to the data label and mark it as the target knowledge graph; The multi-source data content is parsed, and the parsing results are combined with the target knowledge graph to determine the corresponding classification labels.

5. A dynamically predictable data flow detection and classification method according to claim 4, characterized in that: The domain knowledge graph is established and obtained based on the subdivided domain, or is divided and obtained from the knowledge graph established based on the data domain; wherein the coverage of the data domain is larger than the subdivided domain.

Images