GB35114 traffic classification and grading methods, equipment, media and products based on reinforcement learning

Abstract

This application relates to the field of traffic classification technology, and more particularly to a method, device, medium, and product for classifying and grading GB35114 traffic based on reinforcement learning. The method comprises: obtaining raw video traffic data and classifying GB35114 traffic data from the raw video traffic data; performing feature extraction and feature preprocessing on the GB35114 traffic data to obtain preprocessed feature data; inputting the feature data into a pre-trained reinforcement learning model to obtain a traffic classification result for the GB35114 traffic data output by the reinforcement learning model; wherein the critic network in the reinforcement learning model has a hierarchical structure, with each layer corresponding to a traffic level. This application can achieve accurate classification and grading of GB35114 traffic.

Description

Technical Field

[0001] The present application relates to the technical field of traffic classification, and in particular to a GB35114 traffic classification and grading method, equipment, medium and product based on reinforcement learning. Background Art

[0002] With the rapid development of the information society, video surveillance systems are increasingly being used in areas such as urban public safety, traffic management, and financial security. The data processed and transmitted by these systems often contains a large amount of sensitive information. Ensuring that this information is protected from illegal access, tampering, or misuse has become a pressing issue. To address this, the "GB35114-2017 Technical Requirements for Information Security of Public Security Video Surveillance Networks" was developed. Led by the Ministry of Public Security and jointly developed with numerous research institutions, this standard aims to provide comprehensive technical support for information security in the public security video surveillance sector. GB35114 strengthens the information security capabilities of security video surveillance systems, ensuring the confidentiality, integrity, and availability of video data, and safeguarding the safety of the nation and its people.

[0003] GB35114 divides encryption levels into three levels: A, B, and C. Level A is based on two-way device authentication using digital certificates to confirm the legitimacy of the identity. In other words, the new GB35114 certificate verification logic is added to the traditional GB / T28181 identity registration and authentication process, and the legitimacy of users and access devices is authenticated based on certificates and keys to ensure that only legitimate devices can access the management platform. Level B is based on the video data signing capability of digital certificates to verify the authenticity of the video source and prevent video data from being tampered with during transmission. Level C is the highest level. Based on Level B, it encrypts video transmission to effectively prevent video data leakage.

[0004] With the implementation of the GB35114 standard, more and more users are deploying GB35114-compliant terminal devices at key locations in accordance with national policies and their own security requirements. This effectively prevents security issues such as unauthorized access, video tampering, and video leaks. However, the proportion of GB35114-compliant terminal devices in user networks is typically very low. The challenge is to quickly identify GB35114-compliant traffic from video traffic and automatically categorize its security level. This will facilitate the assessment of the number of GB35114-compliant devices in user networks and facilitate the security capability assessment and rating of video surveillance systems.

[0005] Related technologies for classifying and grading the GB35114 standard primarily include rule-based matching, cluster analysis, and machine learning. However, these methods face numerous challenges in practical application. First, they rely on expert experience and knowledge to formulate rules, which is not only subjective but also can lead to inconsistent classification results depending on the rules. Second, GB35114 still adopts the GB / T28181 standard for its cascading and interconnection methods. However, during networking, GB35114's security algorithm mechanisms are incorporated based on device access requirements. This can make it difficult to distinguish between GB / T28181 and GB35114 during cluster analysis, thus affecting the accuracy of classification results. Furthermore, while machine learning methods can automatically learn features, they still require researchers to manually construct feature sets based on expert experience, and the quality of these feature sets directly impacts classification results. Furthermore, network traffic labeling is time-consuming and labor-intensive, further increasing the complexity of the classification task.

[0006] Therefore, how to effectively and accurately classify and grade traffic under the GB35114 standard has become a technical problem that needs to be solved urgently. Summary of the Invention

[0007] In order to solve the problem that the existing technology cannot accurately classify and grade GB35114 traffic, the present application provides a GB35114 traffic classification and grading method, equipment, medium and product based on reinforcement learning.

[0008] In the first aspect, this application provides a GB35114 traffic classification and grading method based on reinforcement learning, which adopts the following technical solutions:

[0009] A GB35114 traffic classification and grading method based on reinforcement learning, comprising:

[0010] Obtaining original video traffic data, and classifying GB35114 traffic data from the original video traffic data;

[0011] Performing feature extraction and feature preprocessing on the GB35114 flow data to obtain preprocessed feature data;

[0012] The feature data is input into a pre-trained reinforcement learning model to obtain a traffic classification result for the GB35114 traffic data output by the reinforcement learning model; wherein the critic network in the reinforcement learning model is a hierarchical structure, and each layer corresponds to a traffic level.

[0013] By adopting the above technical solution, the original video traffic data is obtained and GB35114 traffic data is classified. Then, feature extraction and preprocessing are performed on it. Finally, the reinforcement learning model of the hierarchical critic network is used to output the traffic classification results, thereby achieving effective classification and grading of GB35114 traffic data. The hierarchical critic network can more accurately correspond to different traffic levels, thereby improving the accuracy and efficiency of traffic classification.

[0014] In a preferred example, the present application may be further configured as follows: the construction process of the reinforcement learning model includes:

[0015] Initialize the Actor network, Critic network, target network, and experience replay pool;

[0016] Define reward function and objective function;

[0017] Acquire training sample data, generate quintuple data corresponding to the training sample data using the Actor network and the reward function, and store the quintuple data in the experience replay pool;

[0018] Repeat the model training steps until the model training stopping condition is met.

[0019] By adopting the above technical solution, each network and the experience replay pool are initialized to build a basic framework. The reward and objective functions are defined to provide direction and optimization goals for model training. The Actor network is combined with the reward function to generate five-tuple data and store it in the experience replay pool to accumulate training materials. Training is repeated until the stopping condition is met. This process enables the reinforcement learning model to effectively learn data features, optimize network parameters, and ultimately achieve the ability to accurately classify and grade GB35114 traffic data.

[0020] In a preferred example, the present application can be further configured as follows: the model training step includes:

[0021] Randomly sampling experience data from the experience replay pool, calculating a target Q value based on the target network and the experience data, and updating the predicted Q value of the critic network;

[0022] Calculating a loss function value of the critic network based on the predicted Q value and the target Q value, and updating parameters of the critic network based on the loss function value;

[0023] Calculate the function value of the objective function and update the parameters of the Actor network;

[0024] Update parameters of the target network.

[0025] By adopting the above technical solution, randomly sampling experience data from the experience replay pool can effectively utilize past experience and reduce data correlation. The target Q value is calculated based on the target network and experience data to update the predicted Q value of the Critic network, making the prediction more accurate; by calculating the loss function value to update the Critic network parameters, the network's estimate of value can be optimized; calculating the objective function value to update the Actor network parameters can enable the Actor network to learn better strategies; updating the target network parameters helps stabilize the training process and improve the convergence speed and stability of the model.

[0026] In a preferred example, the present application may be further configured as follows: updating the predicted Q value of the critic network includes:

[0027] For each layer of the critic network, obtaining a Q value of the layer, determining a matching degree between the empirical data and a preset layer feature of the layer, and using the sum of the Q value of the layer and the matching degree as a sub-predicted Q value of the layer;

[0028] The predicted Q value of the critic network is calculated based on the sub-predicted Q values ​​of each layer in the critic network.

[0029] By adopting the above technical solution, when updating the predicted Q value of the Critic network, the sum of the Q value of each layer and the matching degree with the empirical data and the preset layer features is first calculated as the sub-predicted Q value. This can combine the characteristics of each layer itself and the fit with the empirical data to more carefully evaluate the value. The overall predicted Q value is then calculated based on the sub-predicted Q values ​​of each layer. It can be dynamically adjusted according to the security level of the current state, avoiding the fuzzy processing of complex security constraints by a single Q value.

[0030] In a preferred example, the present application may be further configured as follows: the function value of the calculated objective function includes:

[0031] determining a security index and a communication delay index based on the empirical data;

[0032] The function value of the objective function is calculated based on the predicted Q value, the security index, and the communication delay index.

[0033] By adopting the above technical solutions, an innovative multi-objective optimization strategy was designed in the optimization strategy. In addition to the value estimation of traffic data, additional objective functions (security and communication delay) were introduced. By designing a weighted optimal objective function for fusion calculation, multi-objective optimization was achieved to balance performance and security.

[0034] In a preferred example, the present application may be further configured as follows: classifying the GB35114 traffic data from the original video traffic data includes:

[0035] Performing a feature classification step on each piece of traffic data in the original video traffic data, and classifying the GB35114 traffic data from the original video traffic data based on the classification result of each piece of traffic data;

[0036] Among them, the feature classification step includes: matching the current flow data with relevant features of GB / T28181 flow to obtain a comprehensive feature matching degree, and judging whether the current flow data is the GB35114 flow data based on the comprehensive feature matching degree.

[0037] By adopting the above technical solution, the feature classification step is performed on the original video traffic data one by one. The comprehensive feature matching degree is obtained by matching the current traffic data with the GB / T28181 traffic-related features to determine whether it is GB35114 traffic data. This method can carefully and accurately filter out GB35114 traffic data from a large amount of original video traffic data, thereby improving the accuracy and efficiency of data classification.

[0038] In a preferred example, the present application may be further configured as follows: performing feature extraction and feature preprocessing on the GB35114 flow data to obtain preprocessed feature data includes:

[0039] Performing multi-dimensional feature extraction on the GB35114 traffic data to obtain initial feature data;

[0040] The initial feature data is sequentially subjected to outlier elimination and standardization processing to obtain intermediate feature data;

[0041] Extracting time series features of the intermediate feature data;

[0042] The intermediate feature data and the time series features are subjected to format normalization processing to obtain preprocessed feature data.

[0043] By adopting the above technical solution, multi-dimensional feature extraction is first performed on the GB35114 traffic data to fully obtain data features. Then, intermediate feature data is obtained through outlier removal and standardization, which can improve data quality and model training efficiency. Time series features are then extracted and formatted together with the intermediate feature data to make the data format unified and regular. The final pre-processed feature data can be better suitable for subsequent model training and analysis, thereby improving the accuracy and stability of the model.

[0044] In a second aspect, the present application provides an electronic device, which adopts the following technical solution:

[0045] one or more processors;

[0046] Memory;

[0047] At least one application, wherein at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the GB35114 traffic classification and grading method based on reinforcement learning as described in any one of the first aspects.

[0048] In a third aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0049] A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the GB35114 traffic classification and grading method based on reinforcement learning as described in any one of the first aspects.

[0050] In a fourth aspect, the present application provides a computer program product that adopts the following technical solution:

[0051] A computer program product includes a computer program. When the computer program is executed by a processor, it implements the GB35114 traffic classification and grading method based on reinforcement learning as described in any one of the first aspects.

[0052] In summary, this application has the following beneficial technical effects:

[0053] This application obtains the original video traffic data and classifies the GB35114 traffic data, then extracts and preprocesses the features, and finally uses the reinforcement learning model of the hierarchical Critic network to output the traffic classification results, thereby achieving effective classification and grading of the GB35114 traffic data. The hierarchical Critic network can more accurately correspond to different traffic levels, thereby improving the accuracy and efficiency of traffic classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a structural diagram of a GB35114 traffic classification and grading system based on reinforcement learning provided in an embodiment of the present application;

[0055] Figure 2 This is a flow chart of a GB35114 traffic classification and grading method based on reinforcement learning provided in an embodiment of the present application;

[0056] Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following is combined with Figure 1 -Attached Figure 3 This application is described in further detail.

[0058] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed, but as long as they are within the scope of the claims of the present application, they are protected by the patent law.

[0059] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0061] It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained through the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0062] The purpose of this application is to automatically extract traffic data that complies with the GB35144 standard from video traffic data, effectively and automatically classify GB35144 A, B, and C traffic, and facilitate the security capability assessment and rating of video surveillance systems. This not only improves the accuracy and efficiency of traffic classification and grading under the GB35144 standard, but also significantly reduces the time and effort spent by security personnel on traffic classification and grading, thereby significantly improving the information security protection capabilities of public security video surveillance systems.

[0063] Based on this, the embodiment of the present application provides a GB35114 traffic classification and grading system based on reinforcement learning, which can be loaded on electronic devices, such as Figure 1As shown, the system includes: a data collection and coarse-grained classification module 101, a traffic data preprocessing module 102, a reinforcement learning model construction module 103, a model training and optimization module 104, and a real-time fine-grained classification module 105.

[0064] The data collection and coarse-grained classification module 101 is used to perform coarse-grained classification on the identifiable features extracted from the original video traffic data by building a string matching mechanism, so as to classify the GB35114 traffic data from the original video traffic data.

[0065] The traffic data preprocessing module 102 is used to extract traffic behavior features from GB35114 traffic data, construct multi-dimensional feature vectors, and clean and preprocess the extracted features to remove outliers and noise interference to ensure high data quality.

[0066] The reinforcement learning model construction module 103 is used to construct a deep neural network model, allowing the model to automatically learn the complex mapping relationship between traffic behavior characteristics and the three-level standards A, B, and C of GB35144.

[0067] The model training and optimization module 104 is used to train the model through a large amount of labeled traffic sample data, continuously optimize the model parameters, and improve the model's generalization ability and classification accuracy.

[0068] The real-time fine-grained classification module 105 is used to input the collected real-time video traffic data into the trained reinforcement learning model in actual applications, and receive the classification results of the GB35114 traffic output by the model, which can significantly improve the accuracy and efficiency of traffic classification and grading under the GB35114 standard.

[0069] The present application embodiment provides a GB35114 traffic classification and grading method based on reinforcement learning, such as Figure 1 As shown, the method provided in the embodiment of the present application is applied to a GB35114 traffic classification and grading system based on reinforcement learning, and is executed by an electronic device, which can be a server or a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smart phone, tablet computer, laptop computer, desktop computer, etc., but is not limited to this. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. The method includes steps S201 to S203, wherein:

[0070] S201. Obtain original video traffic data, and classify GB35114 traffic data from the original video traffic data.

[0071] Specifically, traffic data from the video surveillance network can be collected through probes or network interfaces to obtain the original video traffic data of the video surveillance network. Due to the high similarity between GB / T28181 and GB35114, misjudgment is prone to occur. This application proposes a coarse-grained clustering method to achieve rapid classification of GB / T28181 and GB35114 traffic. Raw video traffic data may contain multiple traffic data items. Coarse-grained clustering analysis can be performed on each traffic data item to determine whether the traffic data item is GB / T28181 or GB35114 traffic.

[0072] Specifically, relevant features of GB / T28181 traffic are pre-defined based on the characteristics of GB / T28181. There can be multiple relevant features, and a matching threshold is defined. Any traffic data in the original video traffic data is used as the current traffic data. The current traffic data is matched with the relevant features of GB / T28181 traffic to obtain a comprehensive feature matching degree. The greater the comprehensive feature matching degree, the higher the similarity between the relevant features of the current traffic data and GB / T28181 traffic. Furthermore, the comprehensive feature matching degree is compared with the matching threshold. If the comprehensive feature matching degree does not exceed the matching threshold, the current traffic data is determined to be GB35114 traffic. If the comprehensive feature matching degree exceeds the matching threshold, the current traffic data is determined to be GB / T28181 traffic.

[0073] S202: Extract and preprocess the GB35114 traffic data to obtain preprocessed feature data.

[0074] Specifically, for each piece of traffic data identified as GB35114 traffic data, multi-dimensional features are extracted as initial feature data. Feature dimensions include transmission behavior features, connection behavior features, data packet features, and encryption features. Specific features include: average data packet rate and rate fluctuation standard deviation in the transmission behavior dimension; connection duration in the connection behavior dimension; data packet size distribution entropy in the data packet dimension; and encryption algorithm identification, key negotiation frequency, signaling security, identity authentication, signature control, encryption control, and data encryption in the encryption dimension.

[0075] Furthermore, the initial feature data is sequentially processed to remove outliers and normalize to obtain intermediate feature data. Time series features are extracted from the intermediate feature data, and the intermediate feature data and time series features are format-normalized to obtain preprocessed feature data. The preprocessed feature data is a 16-dimensional feature vector.

[0076] S203. Input the feature data into a pre-trained reinforcement learning model to obtain a traffic classification result for GB35114 traffic data output by the reinforcement learning model; wherein the critic network in the reinforcement learning model is a hierarchical structure, and each layer corresponds to a traffic level.

[0077] This embodiment classifies GB / T28181 and GB35114, two highly similar protocols, and accurately extracts GB35114 traffic data. A reinforcement learning model is then used to classify real-time traffic into the three levels of A, B, and C specified in GB35114. The system monitors the accuracy of traffic classification and the security status of the system in real time. If classification errors or system anomalies are detected, the relevant information is fed back to the reinforcement learning model. The model then learns and adjusts online based on the new feedback and real-time traffic data, continuously optimizing the classification strategy to adapt to dynamic changes in the network environment and ensure the effective protection of the information security of the video surveillance system.

[0078] A possible implementation of the embodiment of the present application is to classify GB35114 traffic data from raw video traffic data, including:

[0079] Performing a feature classification step on each piece of traffic data in the original video traffic data, and classifying GB35114 traffic data from the original video traffic data based on the classification result of each piece of traffic data;

[0080] Among them, the feature classification step includes: matching the current flow data with the relevant features of GB / T28181 flow to obtain a comprehensive feature matching degree, and judging whether the current flow data is GB35114 flow data based on the comprehensive feature matching degree.

[0081] In this embodiment, the relevant characteristics of GB / T28181 traffic are the specific SIP methods defined for implementing video surveillance system functions, including: the INVITE method for establishing a video call session, the ACK method for confirming session establishment, the BYE method for terminating a session, the OPTIONS method for querying device capabilities, the REGISTER method for device registration, specific 401 Unauthorized responses, 20-digit encoding rules for device IDs, and content of the application / sdp type in the message body. If these specific SIP methods are detected in traffic and the message content is related to video surveillance services, it can be determined to be GB / T28181 traffic.

[0082] Construct a fusion coarse-grained classification algorithm based on GB / T28181 and GB35114: Among them, it represents the comprehensive matching degree between the current traffic data and the relevant characteristics of GB / T28181 traffic; A indicates whether the INVITE method is matched; B indicates whether the ACK method and BYE method are matched; C indicates whether the OPTIONS method and REGISTER method are matched; D indicates whether the specific 401 Unauthorized response is matched; E indicates whether the 20-digit encoding rule that conforms to the device ID is matched; and F indicates whether the message weight contains content of the application / sdp type. For A, B, C, D, E, and F, if the match result is a match, the corresponding value is set to 1; if not, the corresponding value is set to 0. are the weights of their respective parameters, which can be set according to actual experience.

[0083] Furthermore, the obtained comprehensive feature matching degree is compared with the preset comprehensive feature matching degree threshold. When the comprehensive feature matching degree exceeds the preset comprehensive feature matching degree threshold, it means that the relevant feature matching degree between the current traffic data and the GB / T28181 traffic is too high, and the current traffic data is judged to be GB / T28181 traffic; when the comprehensive feature matching degree does not exceed the preset comprehensive feature matching degree threshold, the current traffic data is judged to be GB35114 traffic data.

[0084] This embodiment integrates a coarse-grained matching algorithm to classify the original video traffic and accurately extract the GB35114 traffic data therein, so as to facilitate the next step of analyzing and processing the GB35114 traffic data and classifying it into Class A, Class B, and Class C.

[0085] A possible implementation of the embodiment of the present application is to perform feature extraction and feature preprocessing on GB35114 traffic data to obtain preprocessed feature data, including:

[0086] Perform multi-dimensional feature extraction on GB35114 traffic data to obtain initial feature data;

[0087] The initial feature data is sequentially processed by removing outliers and standardizing to obtain intermediate feature data;

[0088] Extracting time series features of intermediate feature data;

[0089] The intermediate feature data and time series features are format-normalized to obtain preprocessed feature data.

[0090] In this embodiment, the extracted initial feature data is subjected to outlier elimination, including: using the IsolationForest algorithm to identify and eliminate abnormal traffic, , where E(h(x)) is the average path length of data point x in all trees, n is the sample size of training data, and c(n) is the average path length of the tree, which is used for standardization. Then the data is standardized and Z-score normalization is performed on each dimension. The normalization formula is: , Z is the data point after standardization, X is the original data point before standardization, μ is the mean of the original data, and σ is the standard deviation of the original data.

[0091] Extracting time series features of intermediate feature data includes: processing the intermediate feature data through a sliding window to obtain time series features.

[0092] Furthermore, format planning and processing include dimensionality reduction and format integration. The intermediate feature data and time series features are reduced in dimensionality, and PCA is used to compress the original 32-dimensional features to 16 dimensions. The destination IP address is used as the key value (key) to integrate the reduced dimensionality features into JSON format to obtain preprocessed feature data for subsequent data processing and analysis.

[0093] This embodiment first performs multi-dimensional feature extraction on the GB35114 traffic data to comprehensively obtain data features. Then, intermediate feature data is obtained through outlier removal and standardization, which can improve data quality and model training efficiency. Time series features are then extracted and formatted together with the intermediate feature data to make the data format unified and regular. The final pre-processed feature data can be better suited for subsequent model training and analysis, thereby improving the accuracy and stability of the model.

[0094] The Deep Deterministic Policy Gradient (DDPG) algorithm is an algorithm based on deep reinforcement learning that aims to optimize a deterministic policy rather than a random policy through gradient descent. In the field covered by this application, network traffic can be divided into different levels according to the GB35114 standard. The traffic has many characteristics and high dimensionality. In most cases, a single traffic feature cannot play a decisive role in the classification result. Instead, multiple features need to be fused and calculated. This makes the reinforcement learning process very unstable and the convergence speed will be very slow. After many experiments, this embodiment selects the Deep Deterministic Policy Gradient algorithm. This algorithm has advantages in handling high-dimensional state problems. At the same time, it has good stability and convergence, and can effectively learn complex strategies.

[0095] In one possible implementation of the embodiment of the present application, the process of building a reinforcement learning model includes:

[0096] Initialize the Actor network, Critic network, target network, and experience replay pool;

[0097] Define reward function and objective function;

[0098] Obtain training sample data, use the Actor network and reward function to generate quintuple data corresponding to the training sample data, and store the quintuple data in the experience replay pool;

[0099] Repeat the model training steps until the model training stopping condition is met.

[0100] In this embodiment, the deep deterministic policy gradient algorithm adopts a dual network architecture. After reducing the dimensionality of the traffic features, a 16-dimensional core feature vector is extracted, covering key security indicators such as traffic load, connection timing, protocol characteristics, signaling security, identity authentication, signature control, encryption control, and data encryption. The resulting state space S is a 16-dimensional feature vector. The Actor network is responsible for generating actions, and the Critic network is responsible for evaluating the quality of actions. The Critic network of traditional DDPG outputs a single Q value and cannot perform differentiated evaluations for security levels. In the field of GB35114 traffic classification, Class A includes certificate verification logic, Class B adds signature verification logic on the basis of Class A, and Class C encrypts data on the basis of Class B.

[0101] This embodiment innovatively discretizes the action space A of the actor network into three security level decisions {0, 1, 2}, corresponding to levels A, B, and C of GB35114 traffic, respectively. The critic network is then layered into a three-layer structure, corresponding to levels A, B, and C of GB35114 traffic, respectively. This not only enables more precise assessment of action quality and optimizes action selection strategies for different security levels, but also improves the model's adaptability to complex environments. The target network includes a target actor network corresponding to the actor network and a target critic network corresponding to the critic network.

[0102] Based on the characteristics of GB35114 traffic classification, a reward function R is defined to better match the requirements of GB35114 traffic classification, including: a reward score of +100 for correct classification, a reward score of -50 for incorrect classification, a reward score of -20 when an adversarial network is detected (an independent forest or autoencoder can be used to detect the adversarial network), and a reward score of -10 when the action delay exceeds the threshold.

[0103] Initialize the relevant parameters of the reinforcement learning model, including the weights and biases of the neural network. The training sample data is consistent with the data format of the feature data after preprocessing the GB35114 traffic data in this application. The training sample data includes training traffic data and corresponding graded annotations. For each training sample data, input it into the reinforcement learning model. The constructed model selects an action according to the current strategy and grades the traffic. Calculate the reward value based on the selected action and the actual labeled GB35114 graded result. Store the current state, action, reward value and next state in the experience replay pool. Specifically, in state Under the deterministic behavior strategy µ, the action is generated through the Actor network θ , perform the action , get the new state and rewards , and whether it is in the terminal state ,Will The five-tuple data is stored in the experience replay pool.

[0104] The conventional optimization strategy is to use the predicted Q value as the objective function and maximize the predicted Q value of the Critic network for optimization: , is the performance index of the Actor network, expressed in parameters The goal is to maximize the average predicted Q value evaluated by the Critic network. Then, the gradient backpropagation of the neural network is used to update all the parameters of the Actor network.

[0105] This embodiment innovatively adopts a multi-objective optimization strategy. The objective function not only includes maximizing the Q value but also introduces additional security indicators and communication delay indicators to balance performance and security through multi-objective optimization. The objective function is: .

[0106] in, is the performance indicator of the Actor network, and the goal is to maximize this performance indicator; Parameters of the Actor network; Indicates the samples; Indicates the The status of the samples; Indicates Output action of the Actor network in the state; Represents the predicted Q value of the Critic network; Represents the parameters of the Critic network; N is the batch size, that is, the number of samples of experience data sampled from the experience replay pool; Indicates safety indicators; Indicates the communication delay indicator; is a weight parameter used to balance the importance of each objective.

[0107] When the number of samples in the experience replay pool reaches a certain threshold, small batches of experience data are randomly sampled from the pool. After each sampling, a round of model training is performed using the sampled experience data. The model's neural network parameters are updated by minimizing a loss function (such as the mean squared error loss function). At the same time, neural network hyperparameters such as the learning rate, number of hidden layer neurons, and batch size are adjusted. The optimal hyperparameter combination is selected through methods such as cross-validation to ensure that the Q-value function more closely approximates the true state-action-value function. This process is repeated repeatedly, and as training progresses, the policy is gradually optimized, allowing the model to select more optimal actions based on Q-values ​​and improving classification accuracy. This continues until a termination condition is met, such as reaching the maximum number of training rounds.

[0108] This embodiment introduces reinforcement learning technology into the security classification scenario of GB35114 video surveillance traffic, breaking through the traditional passive classification mode based on rule matching or static feature analysis (such as the existing technology that relies on manually preset thresholds or fixed risk indicators). It realizes the automatic and adaptive determination of risk levels through the dynamic interaction between the reward and punishment mechanism of the reinforcement learning algorithm model and the traffic environment, providing a new technical path for highly dynamic and multi-dimensional security assessment.

[0109] To address the problem that a single layer of the value assessment network in traditional reinforcement learning cannot accurately match complex security levels, this invention innovatively designs a layered design for the Critic network of the deep deterministic policy gradient algorithm. Each layer corresponds to Level A / Level B / Level C of GB35114, and is dynamically adjusted according to the security level of the current state. This not only allows for a more precise evaluation of action quality and optimizes action selection strategies for different security levels, but also improves the model's adaptability to complex environments.

[0110] In a possible implementation of the embodiment of the present application, the model training step includes:

[0111] Randomly sample experience data from the experience replay pool, calculate the target Q value based on the target network and the experience data, and update the predicted Q value of the critic network;

[0112] Calculate the loss function value of the Critic network based on the predicted Q value and the target Q value, and update the parameters of the Critic network based on the loss function value;

[0113] Calculate the function value of the objective function and update the parameters of the Actor network;

[0114] Update the parameters of the target network.

[0115] In this embodiment, for each training sample in the empirical data, the training sample is input into the Actor network to generate an action, and the next state is input into the target Actor network. The target Actor network receives the next state in the training sample and outputs the target action. The target Critic network evaluates the value of the current state-action pair, the next state and the target action. The target Q value is an estimate of the true value of the current state-action pair. The target Q value of the Critic network is updated using the Bellman equation: ;

[0116] Among them, y is the target Q value, and R is the function value of the reward function; is the target Actor network, Parameters of the target Actor network; is the target Critic network, are the parameters of the target Critic network; The discount factor indicates the degree of attenuation of future rewards. Its value range is (0, 1). The closer it is to 1, the more emphasis is placed on long-term benefits. For the next state; Target action.

[0117] For each layer of the Critic network, the Q value of the layer is obtained, and the matching degree between the training sample and the preset layer features of the layer is determined. The sum of the Q value of the layer and the matching degree is used as the sub-predicted Q value of the layer; the predicted Q value of the Critic network is calculated based on the sub-predicted Q values ​​of each layer in the Critic network.

[0118] Traverse each training sample in the empirical data and obtain the target Q value and predicted Q value corresponding to each training sample. Based on the target Q value and predicted Q value corresponding to each training sample in the empirical data, the loss function value is calculated using the mean square error. , is the loss function value of the Critic network; are the parameters of the Critic network; is the target Q value; N is the batch size, that is, the number of training samples of experience data sampled from the experience replay pool; Indicates the The state of the training sample; Indicates the Based on the calculated loss function value, the gradient back propagation of the neural network is used to update all parameters of the critic network.

[0119] Based on the defined objective function and the predicted Q value, security index and communication delay index of the current round of training, the function value of the objective function is calculated. The parameters of the target network are updated using the soft update mechanism. ; is the soft update coefficient.

[0120] This embodiment can effectively utilize past experience and reduce data correlation by randomly sampling experience data from the experience replay pool, and calculate the target Q value based on the target network and experience data to update the predicted Q value of the critic network, making the prediction more accurate; by calculating the loss function value to update the critic network parameters, the network's estimate of value can be optimized; calculating the objective function value to update the actor network parameters, the actor network can learn a better strategy; updating the target network parameters helps to stabilize the training process and improve the convergence speed and stability of the model.

[0121] A possible implementation of the embodiment of the present application is to update the predicted Q value of the critic network, including:

[0122] For each layer of the critic network, obtain the Q value of the layer, determine the matching degree between the empirical data and the preset layer features of the layer, and use the sum of the Q value of the layer and the matching degree as the sub-predicted Q value of the layer;

[0123] The predicted Q value of the Critic network is calculated based on the sub-predicted Q values ​​of each layer in the Critic network.

[0124] The original Critic network outputs a single Q value: ， The modified hierarchical critic network in this embodiment outputs multiple Q values, which correspond to different security levels. Each layer of the network can learn the decision preference under a specific security level. Level A matches the GB35114 traffic authorization feature and calculates the similarity. , Represents the multidimensional feature vector representing the training sample, Represents the Authorization feature vector. Level B matches the GB35114 traffic VideoSignatureControl feature and calculates the similarity. , Represents the multidimensional feature vector representing the training sample, Indicates the VideoSignatureControl feature. Level C matches the GB35114 traffic VideoEncryptionControl, GB35114 traffic VideoPlayEncryption, and the degree of encryption obfuscation, and calculates the similarity. , Represents the multidimensional feature vector representing the training sample, Represents the VideoEncryptionControl feature, Represents the VideoPlayEncryption feature, The correlation coefficient indicating the degree of encryption obfuscation, The preset weights.

[0125] Calculate the decision preference for each layer and obtain the sub-prediction Q value corresponding to each layer: ,in, is the sub-prediction Q value of the k-th layer, represents the Q-value function of the kth layer, 3 is the total number of security levels, is the similarity of the kth layer.

[0126] The predicted Q value of the critic network is obtained by weighted summing of the sub-predicted Q values ​​of each layer. , is the weight of the kth layer.

[0127] When updating the predicted Q value of the critic network, this embodiment first calculates the sum of the Q value of each layer and the matching degree of the empirical data and the preset layer features as the sub-predicted Q value. This can combine the characteristics of each layer itself and the fit with the empirical data to more carefully evaluate the value. Then, the overall predicted Q value is calculated based on the sub-predicted Q values ​​of each layer. It can be dynamically adjusted according to the security level of the current state, avoiding the fuzzy processing of complex security constraints by a single Q value.

[0128] A possible implementation of the embodiment of the present application is to calculate the function value of the objective function, including:

[0129] Determine security indicators and communication delay indicators based on empirical data;

[0130] Based on the predicted Q value, security index and communication delay index, the function value of the objective function is calculated.

[0131] In this embodiment, encryption features and transmission behavior features may be first extracted from the empirical data, a security index may be determined based on the encryption features, and a communication delay index may be determined based on the transmission behavior features.

[0132] For security indicators, scoring rules are set. Encryption features may include: encryption control field, encryption algorithm identifier, key length, certificate validity, etc. Corresponding encryption strength scores are set for unencrypted, encrypted and using a certain encryption algorithm, and validity scores are set for certificate validity. The security index is obtained by weighted summation of the encryption strength and validity scores.

[0133] Scoring rules are set for communication delay metrics. Transmission behavior characteristics can include packet arrival time interval, network congestion flags, and protocol type. A delay baseline is set based on the protocol type and adjusted based on network congestion. For example, when network congestion occurs, the delay baseline increases, indicating a greater maximum allowable delay. The difference between the packet arrival time interval (current delay) and the delay baseline is calculated, and the ratio of this difference to the delay baseline is used as a safety indicator.

[0134] The objective function is: , based on the predicted Q value, security index and communication delay index, and their corresponding weights, the function value of the objective function is calculated.

[0135] This embodiment innovatively designs a multi-objective optimization strategy in the optimization strategy, introduces additional objective functions (security and communication delay), and implements multi-objective optimization to balance performance and security by designing a weighted optimal objective function for fusion calculation.

[0136] An electronic device is provided in an embodiment of the present application, such as Figure 3 As shown, Figure 3 The electronic device 300 shown includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may further include a transceiver 304. It should be noted that in actual applications, the number of transceivers 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present application.

[0137] Processor 301 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0138] Bus 302 may include a path for transmitting information between the above components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 3 Only one thick line is used in the diagram, but it does not mean that there is only one bus or one type of bus.

[0139] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0140] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the embodiment of the GB35114 traffic classification and grading method based on reinforcement learning.

[0141] Figure 3 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0142] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the contents shown in the aforementioned embodiment of the GB35114 traffic classification and grading method based on reinforcement learning.

[0143] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0144] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the contents shown in the aforementioned embodiment of the GB35114 traffic classification and grading method based on reinforcement learning are implemented.

[0145] The above are only some of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A GB35114 traffic classification and grading method based on reinforcement learning, characterized in that: include: Obtaining original video traffic data, and classifying GB35114 traffic data from the original video traffic data; Performing feature extraction and feature preprocessing on the GB35114 flow data to obtain preprocessed feature data; Inputting the feature data into a pre-trained reinforcement learning model to obtain a traffic classification result for the GB35114 traffic data output by the reinforcement learning model; wherein the critic network in the reinforcement learning model has a hierarchical structure, with each layer corresponding to a traffic grade; The classifying GB35114 traffic data from the original video traffic data includes: Performing a feature classification step on each piece of traffic data in the original video traffic data, and classifying the GB35114 traffic data from the original video traffic data based on the classification result of each piece of traffic data; The feature classification step includes: matching the current flow data with relevant features of the GB / T28181 flow to obtain a comprehensive feature matching degree, and determining whether the current flow data is the GB35114 flow data based on the comprehensive feature matching degree; The construction process of the reinforcement learning model includes: Initialize the Actor network, Critic network, target network, and experience replay pool; Define reward function and objective function; Acquire training sample data, generate quintuple data corresponding to the training sample data using the Actor network and the reward function, and store the quintuple data in the experience replay pool; Repeat the model training steps until the model training stopping condition is met.

2. The GB35114 traffic classification and grading method based on reinforcement learning according to claim 1 is characterized in that: The model training step includes: Randomly sampling experience data from the experience replay pool, calculating a target Q value based on the target network and the experience data, and updating the predicted Q value of the critic network; Calculating a loss function value of the critic network based on the predicted Q value and the target Q value, and updating parameters of the critic network based on the loss function value; Calculate the function value of the objective function and update the parameters of the Actor network; Update parameters of the target network.

3. The GB35114 traffic classification and grading method based on reinforcement learning according to claim 2 is characterized in that: The updating of the predicted Q value of the Critic network includes: For each layer of the critic network, obtaining a Q value of the layer, determining a matching degree between the empirical data and a preset layer feature of the layer, and using the sum of the Q value of the layer and the matching degree as a sub-predicted Q value of the layer; The predicted Q value of the critic network is calculated based on the sub-predicted Q values ​​of each layer in the critic network.

4. The GB35114 traffic classification and grading method based on reinforcement learning according to claim 2 is characterized in that: The function value of the target function is calculated, comprising: determining a security index and a communication delay index based on the empirical data; The function value of the objective function is calculated based on the predicted Q value, the security index, and the communication delay index.

5. The GB35114 traffic classification and grading method based on reinforcement learning according to claim 1 is characterized in that: The feature extraction and feature preprocessing of the GB35114 flow data to obtain preprocessed feature data includes: Performing multi-dimensional feature extraction on the GB35114 traffic data to obtain initial feature data; The initial feature data is sequentially subjected to outlier elimination and standardization processing to obtain intermediate feature data; Extracting time series features of the intermediate feature data; The intermediate feature data and the time series features are subjected to format normalization processing to obtain preprocessed feature data.

6. An electronic device, characterized in that: include: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the GB35114 traffic classification and grading method based on reinforcement learning as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the GB35114 traffic classification and grading method based on reinforcement learning as described in any one of claims 1 to 5.

8. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the GB35114 traffic classification and grading method based on reinforcement learning described in any one of claims 1 to 5.

Images