Network data classification method, system and equipment

By establishing a data classification rule library, recurrent neural network model and Qwen natural language processing model, combined with multiple classification rules, the problem that traditional methods are difficult to ensure timeliness and accuracy is solved, and efficient and intelligent classification of network data is achieved to meet the needs of the system. The enterprise needs for in-depth data insights and security.

CN119089247APending Publication Date: 2024-12-06NORTHWESTERN POLYTECHNICAL UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411236375.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-04
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional data classification methods are difficult to ensure the timeliness and accuracy of classification rules, especially when facing massive, heterogeneous, and rapidly changing data, it is difficult to meet the needs of enterprises for in-depth insights and efficient use of data.

Method used

A network data classification method is adopted to collect and preprocess target network data, establish a data classification rule library, recurrent neural network model and Qwen natural language processing model, and comprehensively use a variety of classification rules and deep learning algorithms to determine the optimal classification results. .

Benefits of technology

It realizes comprehensive, accurate and efficient classification of network data, improves the level of intelligence and automation, can cope with complex and changeable network environments and data types, and enhances the intelligibility and security of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089247A_ABST
    Figure CN119089247A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a network data classification method, which comprises the following steps of: acquiring target network data, preprocessing the target network data, and determining first information data; establishing a data classification rule base, and determining a first classification result corresponding to the first information data according to the data classification rule base; establishing a recurrent neural network model, training the recurrent neural network model by using the first sample data until the training is completed, and classifying the first information data according to the trained recurrent neural network model to obtain a second classification result; establishing a Qwen natural language processing model, training the Qwen natural language processing model by using the second sample data until the training is completed, classifying the first information data according to the Qwen natural language processing model after the training is completed, and obtaining a third classification result; and determining an optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a network data classification method, system and device. Background Art

[0002] In today's wave of digitalization, data has become an important part of the core competitiveness of enterprises. The value it contains is constantly being explored and amplified with the rapid development of technologies such as cloud computing, big data, natural language processing (NLP), artificial intelligence (AI) and blockchain; however, maximizing the value of data depends on effective data classification and management mechanisms. Faced with massive, heterogeneous and rapidly changing data, traditional management methods can no longer meet the needs of enterprises for deep insight into and efficient use of data.

[0003] Data classification is the core link of data governance. It divides data into different categories or levels through scientific methods, which not only helps to improve the comprehensibility and availability of data, but also provides enterprises with accurate data analysis tools to help them explore the business value behind the data, optimize operational strategies, and promote product and service innovation. At the same time, as data compliance and privacy protection become increasingly important, an effective data classification mechanism is also the key to ensuring that enterprises operate legally and in compliance with regulations and reduce risks.

[0004] The data classification methods commonly used in the prior art mostly rely on keyword matching and rule matching technology. This method uses a preset keyword library or rule set to perform pattern recognition during the data processing process to determine the sensitivity level or content attributes of the data. However, as the amount of data increases, this method is difficult to ensure the timeliness and accuracy of the classification rules. Summary of the invention

[0005] Based on this, it is necessary to propose a network data classification method to address the above problems.

[0006] A network data classification method, the method comprising the following steps:

[0007] Collecting target network data and preprocessing it to determine first information data;

[0008] Establishing a data classification rule base, and determining a first classification result corresponding to the first information data according to the data classification rule base;

[0009] Establishing a recurrent neural network model, training the recurrent neural network model using the first sample data until the training is completed, and classifying the first information data according to the trained recurrent neural network model to obtain a second classification result;

[0010] Establishing a Qwen natural language processing model, training the Qwen natural language processing model using the second sample data until the training is completed, and classifying the first information data according to the trained Qwen natural language processing model to obtain a third classification result;

[0011] An optimal classification result of the target network data is determined according to the first classification result, the second classification result, the third classification result and corresponding weights.

[0012] In the above solution, the collecting target network data and preprocessing the data to determine the first information data specifically includes:

[0013] Monitor the target network environment and obtain target network data;

[0014] Cleaning and formatting the target network data to obtain first target network data;

[0015] Perform missing value processing and outlier processing on the first target network data to obtain first information data.

[0016] In the above solution, the step of establishing a data classification rule base and determining a first classification result corresponding to the first information data according to the data classification rule base specifically includes:

[0017] The data classification rule base includes several rule matching methods, including: regular expression matching method, keyword group matching method and common identification matching method;

[0018] Acquire the first information data, analyze it, and determine a corresponding rule matching method;

[0019] The first information data is processed according to a rule matching method corresponding to the first information data to obtain a first classification result.

[0020] In the above scheme, different weight ratios are preset for the several rule matching methods. When the first information data corresponds to multiple industries and the multiple industries have different classification rules, the first information data is classified according to the weight ratios, and the rule matching method is used to obtain a first classification result;

[0021] When the first classification result meets the preset rule, the first classification result is marked and output.

[0022] In the above scheme, the establishing of a recurrent neural network model, training the recurrent neural network model using the first sample data until the training is completed, and classifying the first information data according to the recurrent neural network model after the training to obtain a second classification result specifically includes:

[0023] Collect raw network data and preprocess it;

[0024] Dividing the preprocessed original network data into a training set, a validation set and a test set according to corresponding proportions, wherein the training set includes the first sample data;

[0025] Constructing a recurrent neural network model, and updating model parameters of the recurrent neural network model according to a back propagation algorithm;

[0026] Preset an iteration number threshold, train the recurrent neural network model after the model parameters are updated according to the first sample data, and when the iteration number of the model reaches the iteration number threshold, the training of the recurrent neural network model is completed;

[0027] The first information data is classified according to the trained recurrent neural network model to obtain a second classification result.

[0028] In the above solution, the first information data is classified according to the trained recurrent neural network model to obtain the second classification result, and further includes:

[0029] Predict the validation set according to the trained recurrent neural network model, convert the second classification result into the prediction probability of each category using the softmax function, and determine the corresponding probability distribution;

[0030] Preset a corresponding prediction threshold range to determine whether the prediction probability of each category is valid;

[0031] If the predicted probability of each category is valid, the category with the highest predicted probability is output as the second classification result.

[0032] In the above solution, determining the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights specifically includes:

[0033] The classification result set of the several rule matching methods is preset as R1={R 11 ,R 12 ,…,R 1n}, where R 1i is the classification result of the ith rule, and the weight set corresponding to the rule classification result is W1={W 11 ,W 12 ,…,W 1n}, where W 1i The weights corresponding to several rule matching methods;

[0034] The first classification result is determined according to the following formula:

[0035]

[0036] Where n is the number of classification results of the rule base;

[0037] The classification result set of the recurrent neural network model is preset to be R2={R 21 ,R 22 ,…,R 2n}, where R 2i is the i-th classification result of the recurrent neural network model, and the preset weight set corresponding to the classification result is W2={W 21 ,W 22 ,…,W 2n}, where W 2i is the weight corresponding to the i-th classification result of the recurrent neural network model;

[0038] The second classification result is determined according to the following formula:

[0039]

[0040] Where m is the number of classification results of the recurrent neural network model;

[0041] The classification result set of the Qwen natural language processing model is preset as R3={R 31 ,R 32 ,…,R 3n}, where R 3i is the i-th classification result of the Qwen natural language processing model, and the preset weight set corresponding to the classification result is W3={W 31 ,W 32 ,…,W 3n}, where W 3i is the weight corresponding to the i-th classification result of the Qwen natural language processing model;

[0042] The third classification result is determined according to the following formula:

[0043]

[0044] Where p is the number of classification results of the Qwen natural language processing model.

[0045] In the above solution, determining the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights specifically includes:

[0046]

[0047] Among them, r(rule), r(DL) and r(NLP) are the first classification result, the second classification result, and the third classification result respectively, w(rule), w(DL) and w(NLP) are the weight values ​​corresponding to the first classification result, the second classification result, and the third classification result respectively, and Final_result is the optimal classification result.

[0048] The present application also proposes a network data classification system, the system comprising: a data acquisition unit, a data classification unit and an optimal classification result determination unit;

[0049] The data acquisition unit is used to collect target network data and pre-process it to determine the first information data;

[0050] The data classification unit is used to establish a data classification rule base, and input the first information data into the data classification rule base to determine a first classification result corresponding to the first information data; establish a recurrent neural network model, use the first sample data to train the recurrent neural network model until the training is completed, and classify the first information data according to the trained recurrent neural network model to obtain a second classification result; establish a Qwen natural language processing model, use the second sample data to train the Qwen natural language processing model until the training is completed, and classify the first information data according to the trained Qwen natural language processing model to obtain a third classification result;

[0051] The optimal classification result determination unit is used to determine the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights.

[0052] The present application also proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the following steps of the computer program:

[0053] Collecting target network data and preprocessing it to determine first information data;

[0054] Establishing a data classification rule base, and determining a first classification result corresponding to the first information data according to the data classification rule base;

[0055] Establishing a recurrent neural network model, training the recurrent neural network model using the first sample data until the training is completed, and classifying the first information data according to the trained recurrent neural network model to obtain a second classification result;

[0056] Establishing a Qwen natural language processing model, training the Qwen natural language processing model using the second sample data until the training is completed, and classifying the first information data according to the trained Qwen natural language processing model to obtain a third classification result;

[0057] An optimal classification result of the target network data is determined according to the first classification result, the second classification result, the third classification result and corresponding weights.

[0058] The embodiment of the present invention has the following beneficial effects: firstly, the target network data is collected and preprocessed to determine the first information data; a data classification rule base is established, and the first classification result corresponding to the first information data is determined according to the data classification rule base; a recurrent neural network model is established, and the recurrent neural network model is trained with the first sample data until the training is completed, and the first information data is classified according to the recurrent neural network model after the training is completed to obtain the second classification result; a Qwen natural language processing model is established, and the Qwen natural language processing model is trained with the second sample data until the training is completed, and the first information data is classified according to the Qwen natural language processing model after the training is completed to obtain the third classification result; the optimal classification result of the target network data is determined according to the first classification result, the second classification result, the third classification result and the corresponding weight. The method realizes the comprehensive, accurate and efficient classification of network data by comprehensively using data collection and preprocessing, rule base, recurrent neural network model and natural language processing model, has a higher level of intelligence and automation, and can cope with complex and changeable network environments and data types. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] in:

[0061] Figure 1 The figure is a flowchart of a network data classification method in one embodiment. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0063] In the following description, a large number of specific details are given in order to provide a more thorough understanding of the present invention; however, it is obvious to those skilled in the art that the present invention can be implemented without one or more of these details; in other examples, in order to avoid confusion with the present invention, some technical features known in the art are not described, and it should be understood that the present invention can be implemented in different forms and should not be interpreted as limited to the embodiments presented here; on the contrary, providing these embodiments will make the disclosure thorough and complete and fully convey the scope of the present invention to those skilled in the art.

[0064] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present invention. When used herein, the singular forms "a", "an" and "said / the" are also intended to include the plural forms, unless the context clearly indicates otherwise, and it should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0065] In order to thoroughly understand the present invention, a detailed structure will be presented in the following description to illustrate the technical solution proposed by the present invention; optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementation methods.

[0066] like Figure 1 As shown, in one embodiment, a network data classification method is provided, and the network data classification method includes steps S101 to S105, which are described in detail as follows:

[0067] S101, collecting target network data and preprocessing it to determine first information data;

[0068] Preferably, a traffic data collection module is built in the target network environment, and a high-speed network packet capture module is constructed using DPDK technology. CPU affinity is used to bind each execution thread to a different CPU logical core, and a CPU logical core is allocated to each thread separately. The messages received by the network card are then distributed to each CPU for processing, thereby reducing the processing pressure on a single CPU core. The captured data packets are sorted out by a preprocessing module, and data formatting is performed. The target information carrying the data content that needs to be analyzed is restored and extracted, thereby providing analyzable data support for data classification and analysis.

[0069] Collecting target network data and preprocessing it can ensure the quality and consistency of the input data. The preprocessing steps (such as data formatting, information extraction, etc.) provide a reliable data basis for subsequent classification work and help improve the accuracy and efficiency of classification.

[0070] In some embodiments, collecting target network data and preprocessing the target network data to determine the first information data specifically includes:

[0071] Monitor the target network environment and obtain target network data;

[0072] Cleaning and formatting the target network data to obtain first target network data;

[0073] Perform missing value processing and outlier processing on the first target network data to obtain first information data.

[0074] S102, establishing a data classification rule base, and determining a first classification result corresponding to the first information data according to the data classification rule base;

[0075] Establishing a data classification rule library and formulating compliant classification rules in accordance with national laws and regulations, industry standards and local standards will help to achieve standardization and regularization of data classification. The establishment of a rule library can reduce the subjectivity of human judgment, improve the accuracy and repeatability of classification, and establish multiple rule matching methods for complex network data to ensure the comprehensiveness of rule identification.

[0076] In some embodiments, establishing a data classification rule base, and determining a first classification result corresponding to the first information data according to the data classification rule base specifically includes:

[0077] The data classification rule base includes several rule matching methods, including: regular expression matching method, keyword group matching method and common identifier matching method;

[0078] Acquire first information data, analyze it, and determine a corresponding rule matching method;

[0079] The first information data is processed according to a rule matching method corresponding to the first information data to obtain a first classification result.

[0080] Specifically, the regular expression matching method includes: designing a regular expression recognition formula to match texts through a preset pattern to identify data that meets a specific pattern;

[0081] Keyword group matching methods include: establishing rules to identify keyword groups, and determining whether the data conforms to specific rules by finding whether the text contains these keyword groups;

[0082] Common identifier matching methods include: establishing rules to identify common identifiers, such as field names, etc., and identifying data by checking whether these identifiers exist in the text.

[0083] In some embodiments, different weight ratios are preset for several rule matching methods. When the first information data corresponds to multiple industries and different classification rules exist for multiple industries, the first information data is sorted according to the size of the weight ratio and classified using the rule matching method to obtain a first classification result; when the first classification result meets the preset rules, the first classification result is marked and output to achieve priority classification.

[0084] S103, establishing a recurrent neural network model, using the first sample data to train the recurrent neural network model until the training is completed, and classifying the first information data according to the trained recurrent neural network model to obtain a second classification result;

[0085] The first information data is classified using a recurrent neural network (RNN) model, which can capture the temporal characteristics and complex relationships in the data and is suitable for processing sequence data. By training the model, the RNN can automatically learn the features in the data and use it to predict new data classifications. This method is more flexible and intelligent than traditional rule matching methods and can handle more complex and diverse data types.

[0086] In some embodiments, a recurrent neural network model is established, and the recurrent neural network model is trained using the first sample data until the training is completed, and the first information data is classified according to the recurrent neural network model after the training is completed to obtain a second classification result, which specifically includes:

[0087] Collect raw network data and preprocess it;

[0088] Dividing the preprocessed original network data into a training set, a validation set, and a test set according to corresponding proportions, wherein the training set includes the first sample data;

[0089] Construct a recurrent neural network model and update the model parameters of the recurrent neural network model according to the back propagation algorithm;

[0090] Preset an iteration number threshold, train the recurrent neural network model after the model parameters are updated according to the first sample data, and when the iteration number of the model reaches the iteration number threshold, the recurrent neural network model training is completed;

[0091] The first information data is classified according to the trained recurrent neural network model to obtain a second classification result.

[0092] Specifically, the above preprocessing includes processing missing values ​​and outliers in the original network data to ensure data quality; standardizing or normalizing the original network data to facilitate faster model convergence; and dividing the data set into a training set, a validation set, and a test set, with a ratio of 70% training set, 15% validation set, and 15% test set.

[0093] Preferably, a recurrent neural network structure is selected for sequence data processing. The recurrent neural network model is constructed using one hidden layer with 50 neurons in each layer. The logarithmic cross entropy loss function is used for classification tasks, and the mean square error is used for regression tasks, etc. The stochastic gradient descent (SGD) optimization algorithm is selected, and indicators for evaluating model performance are selected, including accuracy, precision, recall, etc. The hyperparameters of the model are adjusted according to the performance of the validation set, including the learning rate and regularization parameters. The regularization term is increased to prevent the model from overfitting, and finally an independent test set is used to evaluate the generalization ability and performance of the model.

[0094] Preferably, batch training is adopted, using a small portion of data each time the parameters are updated. After the number of training iterations is determined, a fixed number of iterations can be used or training can be stopped when the model performance converges.

[0095] In some embodiments, classifying the first information data according to the trained recurrent neural network model to obtain the second classification result also includes:

[0096] The validation set is predicted based on the trained recurrent neural network model, and the second classification result is converted into the prediction probability of each category using the softmax function to determine the corresponding probability distribution;

[0097] Preset the corresponding prediction threshold range to determine whether the prediction probability of each category is valid;

[0098] If the predicted probability of each category is valid, the category with the highest predicted probability is output as the second classification result.

[0099] Specifically, the trained model is integrated into practical applications to perform real-time data classification tasks. At the model output layer, it is ensured that the predicted probability of each category can be obtained. The output of the neural network is converted into a probability distribution through the softmax function. A prediction threshold of 0.5 is set to determine which categories have a high enough predicted probability to be considered a valid classification result. For each input sample, the model will output a probability distribution, indicating the possibility that the sample belongs to each category. According to the needs of the actual application, the category with the highest probability can be selected as the final classification result. The predicted probability of each category is recorded and output so that further processing or decision-making can be performed based on these weight ratios in actual applications.

[0100] S104, establishing a Qwen natural language processing model, using the second sample data to train the Qwen natural language processing model until the training is completed, and classifying the first information data according to the trained Qwen natural language processing model to obtain a third classification result;

[0101] The introduction of the Qwen natural language processing model (assuming it is some kind of pre-trained language model) provides strong support for the classification of text data. Natural language processing technology can understand and analyze the semantic information in the text, thereby improving the accuracy and depth of classification. Compared with the RNN model, the Qwen model may focus more on the understanding and representation of text semantics and is suitable for scenarios that require in-depth understanding of text content.

[0102] First, data collection and preprocessing are carried out. In addition to obtaining data types from relevant national laws and French standards, relevant data are automatically obtained from websites, social media, forums, etc. through crawler technology to ensure the breadth and representativeness of the data. Then, the data is cleaned. In addition to removing special symbols, irrelevant information such as emoticons and links also need to be considered to maintain the purity of the data.

[0103] Specifically, when vectorizing data, word embedding technology is used to capture the semantic relationship between words. When extracting and selecting features, in addition to traditional word frequency and text length, it is also necessary to extract richer features such as part of speech, named entities, and sentiment analysis to improve classification accuracy. At the same time as using statistical methods, a model-based method (LASSO) is used for feature selection to automatically select the most representative features.

[0104] Preferably, a recurrent neural network (RNN) model is used to capture complex features in the data; when training the model, a transfer learning method is used, and a pre-trained language model (Qwen) is used for fine-tuning to improve the generalization ability and effect of the model. In addition to conventional indicators such as accuracy and recall, more comprehensive evaluation indicators such as AUC-ROC curve and confusion matrix are used to better evaluate the performance of the model; when tuning the model, a hyperparameter search algorithm (random search) is used to automatically search for the best hyperparameter combination to improve the performance of the model; in terms of named entity recognition, sentiment analysis, and text classification, the natural language processing model result weights are set to obtain the classification results of the model and the corresponding weight ratios, as follows:

[0105] (1) Named Entity Recognition (NER):

[0106] Use the named entity recognition function in the model to extract key entity information from the text.

[0107] Application scenarios: Legal terms, names, place names, etc. can be extracted from legal documents.

[0108] Weight ratio: 30%

[0109] (2) Sentiment Analysis:

[0110] Use the sentiment analysis function in the model to determine the sentiment tendency of the text (positive, negative or neutral).

[0111] Application scenario: Analyze user comments on social media to understand the public's attitude towards specific events or products.

[0112] Weight ratio: 30%

[0113] (3) Text Classification:

[0114] Using the model's main features, classify text.

[0115] Application scenario: Classify news articles into different subject categories.

[0116] Weight ratio: 40%

[0117] Named entity recognition and sentiment analysis each account for 30%, and these two tasks provide an additional layer of information, increasing the value and application scope of the model. Named entity recognition is very useful for scenarios where specific information needs to be extracted from text, while sentiment analysis provides an understanding of the emotional tendency of the text.

[0118] S105. Determine the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights.

[0119] By combining the classification results of the rule base, RNN model and Qwen natural language processing model and fusing them according to their respective weights to determine the optimal classification result, this multi-model fusion method can make full use of the advantages of various models to improve the accuracy and robustness of classification. The weight distribution can be adjusted according to actual needs and model performance to achieve the best effect.

[0120] In some embodiments, determining the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights specifically includes:

[0121] The classification result set of several rule matching methods is preset as R1={R 11 ,R 12 ,…,R 1n}, where R 1i is the classification result of the ith rule, and the weight set corresponding to the rule classification result is W1={W 11 ,W 12 ,…,W 1n}, where W 1i The weights corresponding to several rule matching methods;

[0122] The first classification result is determined according to the following formula:

[0123]

[0124] Where n is the number of classification results of the rule base;

[0125] The classification result set of the preset recurrent neural network model is R2 = {R 21 ,R 22 ,…,R 2n}, where R 2i is the i-th classification result of the recurrent neural network model, and the preset weight set corresponding to the classification result is W2={W 21 ,W 22 ,…,W 2n}, where W 2i is the weight corresponding to the i-th classification result of the recurrent neural network model;

[0126] The second classification result is determined according to the following formula:

[0127]

[0128] Where m is the number of classification results of the recurrent neural network model;

[0129] The classification result set of the preset Qwen natural language processing model is R3 = {R 31 ,R 32 ,…,R 3n}, where R 3i is the i-th classification result of the Qwen natural language processing model, and the preset weight set corresponding to the classification result is W3={W 31 ,W 32 ,…,W 3n}, where W 3i is the weight corresponding to the i-th classification result of the Qwen natural language processing model;

[0130] The third classification result is determined according to the following formula:

[0131]

[0132] Where p is the number of classification results of the Qwen natural language processing model.

[0133] In some embodiments, determining the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights specifically includes:

[0134]

[0135] Among them, r(rule), r(DL) and r(NLP) are the first classification result, the second classification result, and the third classification result respectively, w(rule), w(DL) and w(NLP) are the weight values ​​corresponding to the first classification result, the second classification result, and the third classification result respectively, and Final_result is the optimal classification result.

[0136] Specifically, if the classification result of the rule base is "industry", the classification result of the recurrent neural network model is "medical", and the classification result of the Qwen natural language processing model is "education", the corresponding weights are 0.3, 0.5, and 0.2. The specific calculation process is as follows:

[0137]

[0138] Since the final result is a numerical value, we need to map it back to the classification label. We need to select the closest classification label. Although Medical has the highest weight (0.5), it is more important to see which weighted value is closer. In this example, 1.9 is closer to 2 (the numerical value of Medical), so we can map Final_Result back to "Medical" as the classification label.

[0139] In summary, the network data classification method proposed in the present invention can first formulate relevant data classification supporting standard guidelines according to national, industrial and local conditions, and then use deep learning algorithms to more accurately classify data by business attributes, thereby improving the precision and accuracy of classification; through natural language processing technology, it can automatically process a large amount of network data and classify it quickly and efficiently, reducing the burden of manual processing and improving data processing efficiency; finally, a real-time detection mechanism is introduced to timely monitor changes in network data and discover potential security risks, so that timely measures can be taken to respond and ensure data security; effectively reduce the risk of data leakage and unauthorized access, and protect the data asset security of enterprises and organizations.

[0140] The present application also proposes a network data classification system, the system comprising: a data acquisition unit, a data classification unit and an optimal classification result determination unit;

[0141] A data acquisition unit, used for collecting target network data and preprocessing the data to determine first information data;

[0142] A data classification unit is used to establish a data classification rule base, input the first information data into the data classification rule base, and determine a first classification result corresponding to the first information data; establish a recurrent neural network model, use the first sample data to train the recurrent neural network model until the training is completed, and classify the first information data according to the trained recurrent neural network model to obtain a second classification result; establish a Qwen natural language processing model, use the second sample data to train the Qwen natural language processing model until the training is completed, and classify the first information data according to the trained Qwen natural language processing model to obtain a third classification result;

[0143] The optimal classification result determination unit is used to determine the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights.

[0144] The present application also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to perform the following steps:

[0145] Collecting target network data and preprocessing it to determine first information data;

[0146] Establishing a data classification rule base, and determining a first classification result corresponding to the first information data according to the data classification rule base;

[0147] Establishing a recurrent neural network model, training the recurrent neural network model using the first sample data until the training is completed, and classifying the first information data according to the trained recurrent neural network model to obtain a second classification result;

[0148] Establishing a Qwen natural language processing model, training the Qwen natural language processing model using the second sample data until the training is completed, and classifying the first information data according to the trained Qwen natural language processing model to obtain a third classification result;

[0149] The optimal classification result of the target network data is determined according to the first classification result, the second classification result, the third classification result and the corresponding weights.

[0150] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0151] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present application. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. The above disclosure is only a preferred embodiment of the present invention, and of course it cannot be used to limit the scope of the rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope covered by the present invention.

Claims

1. A network data classification method, characterized in that: The method comprises: Collecting target network data and preprocessing it to determine first information data; Establishing a data classification rule base, and determining a first classification result corresponding to the first information data according to the data classification rule base; Establishing a recurrent neural network model, training the recurrent neural network model using the first sample data until the training is completed, and classifying the first information data according to the trained recurrent neural network model to obtain a second classification result; Establishing a Qwen natural language processing model, training the Qwen natural language processing model using the second sample data until the training is completed, and classifying the first information data according to the trained Qwen natural language processing model to obtain a third classification result; An optimal classification result of the target network data is determined according to the first classification result, the second classification result, the third classification result and corresponding weights.

2. The network data classification method according to claim 1, characterized in that: The collecting target network data and preprocessing the data to determine the first information data specifically includes: Monitor the target network environment and obtain target network data; Cleaning and formatting the target network data to obtain first target network data; Perform missing value processing and outlier processing on the first target network data to obtain first information data.

3. The network data classification method according to claim 2, characterized in that: The step of establishing a data classification rule base and determining a first classification result corresponding to the first information data according to the data classification rule base specifically includes: The data classification rule base includes several rule matching methods, including: regular expression matching method, keyword group matching method and common identification matching method; Acquire the first information data, analyze it, and determine a corresponding rule matching method; The first information data is processed according to a rule matching method corresponding to the first information data to obtain a first classification result.

4. The network data classification method according to claim 3, characterized in that: Presetting different weight ratios for the plurality of rule matching methods, when the first information data corresponds to multiple industries and the multiple industries have different classification rules, sorting the first information data according to the weight ratios, classifying the first information data using the rule matching method, and obtaining a first classification result; When the first classification result meets the preset rule, the first classification result is marked and output.

5. The network data classification method according to claim 4, characterized in that: The establishing of the recurrent neural network model, training the recurrent neural network model using the first sample data until the training is completed, and classifying the first information data according to the recurrent neural network model after the training to obtain a second classification result specifically includes: Collect raw network data and preprocess it; Dividing the preprocessed original network data into a training set, a validation set and a test set according to corresponding proportions, wherein the training set includes the first sample data; Constructing a recurrent neural network model, and updating model parameters of the recurrent neural network model according to a back propagation algorithm; Preset an iteration number threshold, train the recurrent neural network model after the model parameters are updated according to the first sample data, and when the iteration number of the model reaches the iteration number threshold, the training of the recurrent neural network model is completed; The first information data is classified according to the trained recurrent neural network model to obtain a second classification result.

6. The network data classification method according to claim 5, characterized in that: The classifying the first information data according to the trained recurrent neural network model to obtain a second classification result also includes: Predict the validation set according to the trained recurrent neural network model, convert the second classification result into the prediction probability of each category using the softmax function, and determine the corresponding probability distribution; Preset a corresponding prediction threshold range to determine whether the prediction probability of each category is valid; If the predicted probability of each category is valid, the category with the highest predicted probability is output as the second classification result.

7. The network data classification method according to claim 6, characterized in that: Determining the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights specifically includes: The classification result set of the several rule matching methods is preset as R1={R 11 ,R 12 ,…,R 1n }, where R 1i is the classification result of the ith rule, and the weight set corresponding to the rule classification result is W1={W 11 ,W 12 ,…,W 1n }, where W 1i The weights corresponding to several rule matching methods; The first classification result is determined according to the following formula: Where n is the number of classification results of the rule base; The classification result set of the recurrent neural network model is preset to be R2={R 21 ,R 22 ,…,R 2n }, where R 2i is the i-th classification result of the recurrent neural network model, and the preset weight set corresponding to the classification result is W2={W 21 ,W 22 ,…,W 2n }, where W 2i is the weight corresponding to the i-th classification result of the recurrent neural network model; The second classification result is determined according to the following formula: Where m is the number of classification results of the recurrent neural network model; The classification result set of the Qwen natural language processing model is preset as R3={R 31 ,R 32 ,…,R 3n }, where R 3i is the i-th classification result of the Qwen natural language processing model, and the preset weight set corresponding to the classification result is W3={W 31 ,W 32 ,…,W 3n }, where W 3i is the weight corresponding to the i-th classification result of the Qwen natural language processing model; The third classification result is determined according to the following formula: Where p is the number of classification results of the Qwen natural language processing model.

8. The network data classification method according to claim 7, characterized in that: Determining the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights specifically includes: Among them, r(rule), r(DL) and r(NLP) are the first classification result, the second classification result, and the third classification result respectively, w(rule), w(DL) and w(NLP) are the weight values ​​corresponding to the first classification result, the second classification result, and the third classification result respectively, and Final_result is the optimal classification result.

9. A network data classification system, characterized in that: The system comprises: a data acquisition unit, a data classification unit and an optimal classification result determination unit; The data acquisition unit is used to collect target network data and pre-process it to determine the first information data; The data classification unit is used to establish a data classification rule base, and input the first information data into the data classification rule base to determine a first classification result corresponding to the first information data; establish a recurrent neural network model, use the first sample data to train the recurrent neural network model until the training is completed, and classify the first information data according to the trained recurrent neural network model to obtain a second classification result; establish a Qwen natural language processing model, use the second sample data to train the Qwen natural language processing model until the training is completed, and classify the first information data according to the trained Qwen natural language processing model to obtain a third classification result; The optimal classification result determination unit is used to determine the optimal classification result of the target network data according to the first classification result, the second classification result, the third classification result and the corresponding weights.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Electrocardiogram signal detection method based on belief rule base and deep neural network

    CN108095716A

  • Financial data sample classification method and device, computer equipment and storage medium

    CN117493940A

  • Image classification method and device, electronic equipment and medium

    CN117788904A

  • System and methods for performing NLP related tasks using contextualized word representations

    US20190197109A1

  • Adversarial example identification method and apparatus

    WO2022063076A1