DQN-based detection method and apparatus for anomalous traffic of database, and device

Abstract

The present disclosure relates to the technical field of database security, and provides a DQN-based detection method and apparatus for anomalous traffic of a database, and a device, for use in improving the security of the database. In the method, at least one traffic detection model separately uses a Q network and a target Q network built by neural networks. During training, risk prediction is performed on behavior features learned by the two neural networks from traffic training samples, thereby avoiding cumbersome steps such as protocol parsing and rule matching, and thus improving the efficiency of detection of the anomalous traffic; moreover, the deficiency of detection of a limited number of risk types is solved, and the security and reliability of the database are improved. In addition, a behavior feature of a current traffic training sample, a behavior feature of the next traffic training sample, and a reward value corresponding to a predicted risk type of the current training sample are used as experience data to calculate a detection loss, so that the neural networks can fully utilize context information in a database session, thereby improving the accuracy of the detection of the anomalous traffic.

Description

Database abnormal traffic detection method, device and equipment based on DQN

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202311760153.1, filed with the State Intellectual Property Office of China on December 20, 2023, and entitled “Database Abnormal Traffic Detection Method, Device and Equipment Based on DQN,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the field of database security technology, and in particular to a method, apparatus, and device for detecting abnormal database traffic based on DQN. Background Art

[0004] Database abnormal traffic detection refers to monitoring the traffic in the database system to detect any unusual or abnormal database operation behavior.

[0005] Currently, traditional methods for detecting abnormal database traffic typically require first collecting traffic from the database port, then parsing the traffic into SQL statements, and then matching potentially risky statements using pre-defined rules and regular expression techniques. However, this process requires extremely high levels of expertise in network security, whether in terms of protocol parsing, rule formulation, or regular expression matching, and presents certain technical barriers. Moreover, fixed rule bases are difficult to generalize to new, previously unseen attacks. Therefore, without uploading encryption certificates, it is impossible to monitor encrypted traffic for risks. Furthermore, traditional methods for detecting abnormal database traffic require significant manual intervention to maintain protocol parsing and rule base updates for different databases, even for different database versions, resulting in high labor costs.

[0006] Summary of the Invention

[0007] The embodiments of the present disclosure provide a method, apparatus, and device for detecting abnormal database traffic based on DQN, so as to improve the security and reliability of database access.

[0008] In one aspect, an embodiment of the present disclosure provides a method for detecting abnormal database traffic based on a DQN, comprising:

[0009] Based on the traffic training sample set generated by the database, at least one initialized traffic detection model is iteratively trained. Each traffic detection model includes a Q network built by a neural network and a target Q network. The training process of each round is as follows:

[0010] Using the Q network, based on the behavioral characteristics of each traffic training sample, a first evaluation value corresponding to the predicted risk type of the corresponding traffic training sample is obtained, and combined with the marked real risk types, a reward value for each traffic training sample is determined;

[0011] The behavior characteristics, predicted risk type, reward value of the current traffic training sample and the behavior characteristics of the next traffic training sample are stored as a piece of experience data in the preset experience recovery pool;

[0012] For each batch of experience data uniformly sampled in the preset experience recovery pool, the target Q network is used to determine the second evaluation value corresponding to the predicted risk type of the next traffic training sample based on the behavioral characteristics of the next traffic training sample. The reward value and the first evaluation value of the current traffic training sample are combined to determine the detection loss of the traffic detection model, and the network parameters of the Q network and the target Q network are adjusted according to the detection loss.

[0013] On the other hand, an embodiment of the present disclosure provides a database abnormal traffic detection device based on DQN, comprising:

[0014] The model training module is used to iteratively train at least one initialized traffic detection model based on the traffic training sample set generated by the database. Each traffic detection model includes a Q network built by the neural network and a target Q network. During each round of training, each unit of the model training module performs the following operations:

[0015] A risk prediction unit is configured to use the Q network to obtain a first evaluation value corresponding to the predicted risk type of each traffic training sample based on the behavioral characteristics of the corresponding traffic training sample, and determine a reward value for each traffic training sample in combination with the labeled real risk types; and, for each batch of experience data uniformly sampled from the preset experience recovery pool, use the target Q network to determine a second evaluation value corresponding to the predicted risk type of the next traffic training sample based on the behavioral characteristics of the next traffic training sample;

[0016] An experience recovery unit is used to store the behavior characteristics, predicted risk type, reward value of the current traffic training sample and the behavior characteristics of the next traffic training sample as a piece of experience data into a preset experience recovery pool;

[0017] a loss calculation unit, configured to determine the detection loss of the traffic detection model according to the second evaluation value corresponding to the predicted risk type of the next traffic training sample, the reward value of the current traffic training sample, and the first evaluation value;

[0018] A parameter adjustment unit is used to adjust network parameters of the Q network and the target Q network according to the detection loss.

[0019] Optionally, the risk prediction unit is specifically used to:

[0020] Generate a random number within a preset range for each traffic training sample based on a preset random seed, where the random seeds for different traffic detection models are different;

[0021] For the traffic training sample whose random number is less than a preset probability threshold, randomly select one from each risk type as a predicted risk type, and use the Q network to determine a first evaluation value of the traffic training sample based on the behavioral characteristics of the traffic training sample and the predicted risk type;

[0022] For the traffic training samples whose random numbers are not less than the preset probability threshold, the Q network is used to obtain the probability that the traffic training samples belong to each risk type based on the behavioral characteristics of the traffic training samples, and each obtained probability is multiplied by the vector after the one-hot encoding of each risk type to obtain a first evaluation value for each risk type, and the risk type corresponding to the maximum first evaluation value is used as the predicted risk type of the traffic training sample;

[0023] The preset probability threshold decreases as the number of iterations increases.

[0024] Optionally, the risk prediction unit is specifically used to:

[0025] Using the target Q network, based on the behavioral characteristics of the next traffic training sample in each piece of empirical data and the one-hot encoded vector of each risk type, a second evaluation value of each risk type is obtained, and the risk type corresponding to the maximum second evaluation value is used as the predicted risk type of the next traffic training sample;

[0026] The loss calculation unit is specifically used for:

[0027] The second evaluation value corresponding to the predicted risk type of the next traffic training sample in each piece of experience data, the sum of the reward value of the current traffic training sample, and the difference between the sum and the first evaluation value corresponding to the current traffic training sample are used as the detection loss of the traffic detection model.

[0028] Optionally, the parameter adjustment unit is specifically configured to:

[0029] Adjusting the network parameters of the Q network according to the detection loss obtained in each round of training;

[0030] After the network parameters of the Q network are adjusted C times, part of the network parameters of the Q network are used as the network parameters of the target Q network, where C is an integer greater than 1.

[0031] Optionally, the model training module further includes a preprocessing unit, configured to:

[0032] Eliminating invalid traffic training samples and invalid data parts in the traffic training samples in the traffic training sample set to obtain valid traffic samples;

[0033] For valid traffic samples whose packet length is less than a preset byte threshold, pad the end with 0 to make its packet length equal to the preset byte threshold; and, for valid traffic samples whose packet length is greater than the preset byte threshold, intercept the valid traffic samples starting from the head according to the preset byte threshold;

[0034] The valid traffic samples whose packet length is equal to the preset byte threshold are normalized.

[0035] Optionally, the preprocessing unit is specifically configured to:

[0036] Eliminating the traffic training samples that are concentrated on handshake messages generated during the network connection establishment process; and

[0037] Eliminating at least one invalid data portion among Ethernet head and tail headers, IP datagram headers and transport protocol headers in the traffic training samples captured by the traffic training sample concentration at the data link layer.

[0038] Optionally, the model training module further includes a testing unit, which is used to:

[0039] For at least one candidate traffic detection model that reaches the preset number threshold, testing the target Q network in the candidate traffic detection model based on the traffic test sample set generated by the database, obtaining a reward value for each traffic test sample set, and obtaining a test result of the target Q network based on the reward value of each traffic test sample;

[0040] At least one target traffic detection model is selected from the at least one candidate traffic detection model in descending order of the test result of at least one target Q network.

[0041] Optionally, the preprocessing unit is specifically configured to:

[0042] Acquiring initial flow data from a target database, and preprocessing the initial flow data to obtain target flow data;

[0043] The risk prediction unit is specifically used for:

[0044] inputting the target traffic data into at least one target traffic detection model respectively, and predicting at least one initial risk type of the target traffic data according to the behavioral characteristics of the target traffic data;

[0045] fusing the obtained at least one initial risk type to obtain a target risk type of the target traffic data;

[0046] When the target risk type is abnormal, an alarm message corresponding to the target risk type is sent.

[0047] On the other hand, an embodiment of the present disclosure provides an electronic device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0048] The communication interface is used to send and receive data;

[0049] The memory is used to store computer instructions;

[0050] The processor is configured to read the computer instructions in the memory and execute the steps of any one of the above-mentioned methods for detecting abnormal database traffic based on DQN.

[0051] On the other hand, the present disclosure provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of any of the above-mentioned DQN-based database abnormal traffic detection methods.

[0052] The DQN-based database abnormal traffic detection method, device, and apparatus provided by the embodiments of the present disclosure have the following beneficial effects:

[0053] The traffic detection model utilizes a Q-network and a target Q-network constructed using neural networks, trained using a database-generated traffic training sample set. During training, the two neural networks use behavioral features learned from the traffic training samples to predict risks, eliminating tedious steps such as protocol parsing and rule matching. This improves the efficiency of abnormal traffic detection and addresses the limitations of detecting a limited number of risk types, enhancing the security and reliability of the database. Furthermore, the detection loss is calculated by using the behavioral features of the current traffic training sample, the behavioral features of the next traffic training sample, and the reward value corresponding to the predicted risk type of the current training sample as empirical data. This allows the neural network to fully utilize the contextual information in the database session, improving the accuracy of abnormal traffic detection.

[0054] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0056] FIG1 is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0057] FIG2 is an overall architecture diagram of database abnormal traffic detection provided by an embodiment of the present disclosure;

[0058] FIG3A is a schematic diagram of a data packet structure of a traffic training sample provided by an embodiment of the present disclosure;

[0059] FIG3B is a flow chart of a method for preprocessing traffic training samples according to an embodiment of the present disclosure;

[0060] FIG3C is a logic diagram of a preprocessing method provided by an embodiment of the present disclosure;

[0061] FIG4 is a deep reinforcement learning process provided by an embodiment of the present disclosure;

[0062] FIG5 is a flow chart of a method for training a flow detection model according to an embodiment of the present disclosure;

[0063] FIG6 is a flowchart of a Q network risk prediction according to an embodiment of the present disclosure;

[0064] FIG7 is a flowchart of network parameter adjustment according to an embodiment of the present disclosure;

[0065] FIG8A is a partial structural diagram of a Q network and a target Q network provided by an embodiment of the present disclosure;

[0066] FIG8B is a complete structural diagram of a Q network and a target Q network provided by an embodiment of the present disclosure;

[0067] FIG9 is a flowchart of a multi-model selection process according to an embodiment of the present disclosure;

[0068] FIG10 is a complete multi-model training flow chart provided by an embodiment of the present disclosure;

[0069] FIG11 is a flowchart of abnormal database traffic detection according to an embodiment of the present disclosure;

[0070] FIG12 is a schematic diagram of multi-model fusion detection provided by an embodiment of the present disclosure;

[0071] FIG13 is a flow chart of a model application method provided by an embodiment of the present disclosure;

[0072] FIG14 is a flow chart of another model application method provided by an embodiment of the present disclosure;

[0073] FIG15 is a functional structure diagram of a database abnormal traffic detection device provided by an embodiment of the present disclosure;

[0074] FIG16 is a hardware structure diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0075] To make the objectives, technical solutions, and advantages of this disclosure more clear, the disclosure will be further described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are merely some, not all, of the embodiments of this disclosure. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this disclosure without inventive effort are intended to fall within the scope of protection of this disclosure.

[0076] All other embodiments derived by persons of ordinary skill in the art based on the exemplary embodiments described in this disclosure without inventive effort are within the scope of protection of this disclosure. Furthermore, while the disclosure herein is presented based on one or more exemplary embodiments, it should be understood that each aspect of the disclosure may constitute a complete technical solution.

[0077] It should be understood that the terms "first," "second," and the like in the specification, claims, and drawings of this disclosure are used to distinguish similar objects, and are not necessarily used to describe a particular order or precedence. It should be understood that such terms are interchangeable where appropriate, such that, for example, the embodiments of this disclosure can be implemented in an order other than that shown or described in the drawings.

[0078] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0079] Abnormal database traffic detection can help detect and address potential security threats and attacks early on, including but not limited to SQL injection, denial of service attacks, unauthorized and malicious access, and data leaks. However, traditional abnormal traffic detection methods, due to technical limitations and complex risk identification and analysis, require high cybersecurity expertise and are only applicable to known attack patterns, resulting in low security.

[0080] For abnormal traffic detection, deep learning (DL) can directly input database traffic data into a model to obtain detection results. Compared to the cumbersome traditional detection techniques, deep learning offers higher computational efficiency and requires less manual maintenance. However, these methods typically model abnormal traffic detection as a classification problem and fail to fully utilize the continuous information of database sessions. Reinforcement learning (RL), a machine learning method, is primarily used to solve continuous control and decision-making problems. It typically involves an agent learning through interaction with its environment to maximize a reward signal for a certain objective function. Deep learning models can be used in RL, forming deep reinforcement learning (DRL). This approach leverages the perception capabilities of deep learning to solve the problem of policy and function modeling, then uses the error backpropagation algorithm to optimize the objective function. Furthermore, it leverages the decision-making capabilities of RL to define the problem and optimize the objective, addressing the challenges of deep learning in abnormal traffic detection.

[0081] In view of this, the embodiment of the present disclosure provides a database abnormal traffic detection method based on DQN. The method builds an end-to-end traffic detection model based on deep reinforcement learning. It can directly learn behavioral features from the traffic data obtained from the database, and to a certain extent, it can utilize the context information in the database connection session to make better decisions, thereby improving the accuracy of abnormal traffic detection, avoiding the tedious steps such as protocol parsing and rule matching in traditional detection methods, and significantly improving detection efficiency. At the same time, the Q network and target Q network in the model use neural networks for feature learning, which effectively overcomes the deficiency of traditional detection based on rule bases that can only detect a limited number of attacks, improves the security and reliability of database access, and has irreplaceable important significance in the field of database security technology. In addition, the embodiment of the present disclosure integrates the detection results of multiple traffic detection models, further improving the security and reliability of abnormal traffic detection.

[0082] The preferred embodiments of the present disclosure are described below in conjunction with the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present disclosure and are not used to limit the present disclosure. In addition, the embodiments and features in the embodiments of the present disclosure may be combined with each other if there is no conflict.

[0083] FIG1 exemplarily shows a scenario diagram of a method for dynamic abnormal data detection based on DQN provided by an embodiment of the present disclosure. The application scenario diagram includes a terminal device 110 and a server 120 .

[0084] In the embodiment of the present disclosure, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, car terminals and other devices; the client on the terminal device 110 can access the database of the target business to generate traffic data. The client can be software (such as a browser, instant messaging software, etc.), or a web page, mini-program, etc. The server 120 is a background server that maintains the security of the database. It can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0085] In an optional implementation, the terminal device 110 and the server 120 may communicate via a communication network.

[0086] In an optional implementation, the communication network is a wired network or a wireless network.

[0087] It should be noted that what is shown in FIG1 is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present disclosure.

[0088] In the embodiment of the present disclosure, when there are multiple servers, the multiple servers can be combined into a blockchain, and the servers are nodes on the blockchain; as disclosed in the embodiment of the present disclosure, the database abnormal traffic detection method based on DQN, in which the traffic training sample set, traffic test sample set, behavior characteristics, reward value, evaluation value, etc. involved can be saved on the blockchain.

[0089] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the implementation methods of the present disclosure are not limited in this regard.

[0090] See Figure 2, which is a diagram of the overall architecture of abnormal database traffic detection provided by the embodiment of the present disclosure, which mainly includes three parts: traffic data preprocessing, traffic detection model training, and multi-model fusion.

[0091] In the traffic data preprocessing phase, the raw traffic data generated by the database captured by the data link layer is annotated to generate a traffic training sample set and a traffic test sample set. Considering that the quality of network model training depends on high-quality training and test samples, if the raw traffic data captured by the data link layer is directly used for training and testing, the raw traffic data contains some invalid information (such as packets), which will affect the performance of the traffic detection model. Therefore, the traffic training sample set and the traffic test sample set are preprocessed to ensure the quality of the traffic data and help improve the accuracy of model detection.

[0092] In the training phase of the traffic detection model, the Deep Q-network (DQN) algorithm is used for reinforcement learning. The Q-network and target Q-network are built based on neural networks and can handle more complex problems than traditional Q-tables. When referencing the DQN algorithm, a reinforcement learning environment is established for the database abnormal traffic detection scenario, using traffic data as the environment state s, the risk type output by the agent as the action a, and the correctness of the detection result as the reward value r. In this environment, the Q-network obtains the current environment state and current action from each piece of data in the experience recycling pool, while the target Q-network obtains the next environment state and next action from each piece of data in the experience recycling pool. This fully utilizes the contextual information in the database connection session and improves the accuracy of model detection. Furthermore, during the training process, after a certain number of iterations, the network parameters of the Q-network are used to update the network parameters of the target Q-network to ensure learning stability.

[0093] In the multi-model fusion part, since the reinforcement learning algorithm is very sensitive to the setting of random seeds, the random seeds can be reinitialized for multi-model training. Finally, the detection results output by the target Q network in the n models with better detection results are fused to determine the final detection result.

[0094] In the embodiments of the present disclosure, a deep reinforcement learning algorithm DQN, which is good at processing sequential decision problems, is adopted to input the preprocessed traffic data into the Q network and the target Q network in the traffic detection model to obtain the current environment state and current action as well as the next environment state and next action, thereby fully capturing the contextual timing information in the database connection session and improving the accuracy of model detection. In addition, the detection results of n models with better training quality are fused to further improve the accuracy of model detection.

[0095] In order to detect abnormal database traffic, the embodiment of the present disclosure generates a traffic training sample set for model training and a traffic training sample set for model testing based on traffic data captured from the database port. The generation process of the two sample sets is similar.

[0096] Taking the generation of traffic training sample sets as an example, the generation process is as follows:

[0097] First, collect the original traffic data generated by the database.

[0098] In one example, raw traffic data is obtained through a pacp file at the database port. The raw traffic data includes abnormal traffic and normal traffic. To protect database security as much as possible, abnormal traffic needs to be detected. Therefore, abnormal traffic is richer than normal traffic.

[0099] In one example, the abnormal traffic in the original traffic data mainly comes from the alarm data audited by the database audit system. This part of the alarm data is usually harmful to the database.

[0100] For example, by triggering a privilege escalation vulnerability in Oracle MySQL Server, an attacker can use this vulnerability to add a new administrative user, who can then control the affected database system, triggering permission and access control issues; an attacker can also use this vulnerability to execute waiting code with superuser privileges.

[0101] It should be noted that the collected file type is only an example and may also be other types of files, and the embodiments of the present disclosure do not impose any restrictive requirements.

[0102] Then, the original traffic data is labeled to obtain the real risk type of the original traffic data.

[0103] In one example, most of the raw traffic data can be input into a database audit system, which uses traditional detection methods for automatic labeling, and a small part is manually labeled by professionals to obtain the true risk type of the raw traffic data.

[0104] Among them, the real risk types of the marked raw traffic data include no risk, low risk, medium risk and high risk.

[0105] For example, for common database attacks, comment bypass in SQL injection is classified as high risk, database type determination is classified as low risk, and OR injection is classified as medium risk.

[0106] It should be noted that the embodiments of the present disclosure do not impose any restrictive requirements on the classification method of risk types. For example, risk types can be divided into three types: no risk, low risk, and high risk.

[0107] Finally, a traffic training sample set is generated based on the labeled original traffic data.

[0108] In the embodiment of the present disclosure, a database audit system is used to automatically label most of the original traffic data, thereby improving the labeling efficiency. However, since the database audit system adopts traditional detection methods, rule customization and data updates may not be timely. Manual labeling of a small portion of the original traffic data by professionals can solve the defects of the database audit system, improve the accuracy of data labeling, and improve the accuracy of model training.

[0109] After obtaining the traffic training sample set, at least one traffic detection model using the DQN algorithm may be iteratively trained, wherein different traffic detection models may have different random seeds.

[0110] As shown in Figure 3A, a schematic diagram of the data packet structure of a traffic training sample provided in an embodiment of the present disclosure includes a 14-byte Ethernet header, a 20-byte IP header, a 20-byte transport protocol header, a 4-byte Ethernet tail and application data. Data other than the application data is invalid information in abnormal traffic detection and may have a negative impact on model training.

[0111] Therefore, in one example, in order for the Q network and the target Q network in the traffic detection model to more accurately extract features and standardize the input of the neural network, after obtaining the traffic training sample set, before iteratively training at least one traffic detection model, the traffic training samples can also be preprocessed to eliminate invalid information that may have a negative impact on the model training, thereby improving the accuracy of the model training.

[0112] 3B , the preprocessing process of the traffic training sample provided by the embodiment of the present disclosure mainly includes the following steps:

[0113] S301: Eliminate invalid traffic training samples in the traffic training sample set and invalid data parts in the traffic training samples to obtain valid traffic samples.

[0114] As shown in Figure 3C, in the traffic training samples captured at the data link layer, the information stored in the Ethernet header, IP header, transport protocol header and Ethernet tail plays an important role in network forwarding, but is invalid information in the database abnormal traffic detection. Therefore, in order to reduce the interference of these invalid information, the invalid data parts in the traffic training samples can be deleted. These invalid data parts include at least one of the Ethernet head and tail headers, IP datagram headers and transport protocol headers (such as TCP headers or UDP headers, etc.).

[0115] In one example, the network connection process typically generates handshake messages. However, these messages typically do not contain any payload and therefore provide no useful information for abnormal traffic detection in the database. Therefore, if a traffic training sample in the traffic training sample set is a handshake message generated during the network connection process, it can be directly removed.

[0116] As shown in FIG3C , taking the TCP three-way handshake as an example, the SYN, ACK, or FIN flag in the captured message is usually set to 1, which does not contain valid information required for abnormal traffic detection and can be deleted.

[0117] After removing the invalid data part from the handshake message and the traffic training samples captured by the data link layer, the remaining valid traffic samples mainly contain the payload of the traffic data, that is, the traffic generated by the SQL operation statement.

[0118] S302: For valid traffic samples whose packet length is less than the preset byte threshold, fill the end with 0 to make its packet length equal to the preset byte threshold; and, for valid traffic samples whose packet length is greater than the preset byte threshold, start from the head of the valid traffic sample and cut according to the preset byte threshold.

[0119] In one example, in order to standardize the input of the neural network, the packet length of the valid traffic samples can be processed for consistency so that the valid traffic samples input to the neural network are all vectors of a preset byte threshold.

[0120] As shown in FIG3C , assuming that the preset byte threshold is 1024, for valid traffic samples with a packet length less than 1024, 0 is padded at the end to make the packet length equal to 1024. For valid traffic samples with a packet length greater than 1024, the first 1024 parts are cut off from the head of the valid traffic sample.

[0121] S303: Normalize the valid traffic samples whose packet length is equal to the preset byte threshold.

[0122] As shown in Figure 3C, after the packet length of the valid traffic samples is processed for consistency, the byte length of each valid traffic sample is consistent, and every 8 bits is a byte, the maximum value of each byte is 255, and every 8 bits divided by 255 is added to the vector as a new value. A vector with a length of the preset byte threshold and a value range of [0,1] can be generated, thereby improving the processing performance of the neural network.

[0123] In the embodiments of the present disclosure, by eliminating invalid data portions in handshake messages and traffic training samples, the accuracy of feature extraction is improved, thereby reducing the interference of invalid information on abnormal traffic detection. In addition, through length consistency and normalization processing, the standardization of neural network input can be guaranteed, which helps to improve the processing performance of the network.

[0124] In the database abnormal traffic detection scenario, the model output is a discrete action. Therefore, the traffic detection model is trained using the DQN algorithm, which combines deep learning and Q learning technologies to solve complex problems with a large number of states and actions.

[0125] Figure 4 illustrates the deep reinforcement learning process. Given a reinforcement learning agent and an environment, the agent, leveraging deep learning's perception capabilities, observes the state of the environment and then performs an action, which in turn provides the agent with a reward. During the learning process, the agent's goal is to maximize its cumulative reward.

[0126] In order to effectively apply reinforcement learning to abnormal database traffic detection, the embodiment of the present disclosure uses the behavioral characteristics of traffic data as the environmental state, the risk type predicted by the intelligent agent as the action, and the correctness of the prediction as the reward, to build a reinforcement learning environment.

[0127] According to the number of risk types involved in the embodiments of the present disclosure, actions can be encoded using 0-3 to represent different risk levels, where 0 represents no risk, 1 represents low risk, 2 represents medium risk, and 3 represents high risk. In addition, the design of the reward function is very simple. The agent is rewarded with 1 point for correct prediction and -1 point for incorrect prediction, thereby guiding the agent in reinforcement learning to learn to distinguish traffic data of different risk levels.

[0128] In the DQN-based traffic detection model, the Q network constructed by the neural network and the target Q network have the same network structure but different network parameters. Furthermore, the DQN algorithm uses a fixed-size experience recycling pool to store empirical data. Initially, the pool contains no data, and training begins after a sufficient amount of empirical data has accumulated. Therefore, after building the traffic detection model, the network parameters of the Q network and the target Q network can be initialized, as well as the capacity of the experience recycling pool.

[0129] Next, based on the established reinforcement learning environment, at least one initialized traffic detection model is iteratively trained according to the generated traffic training sample set. The training process of each traffic detection model is shown in Figure 5 and mainly includes the following steps:

[0130] S501: Using the Q network, according to the behavioral characteristics of each traffic training sample, obtain a first evaluation value corresponding to the predicted risk type of the corresponding traffic training sample, and determine the reward value of each traffic training sample in combination with the marked real risk types.

[0131] In the DQN algorithm's experience replay mechanism, a risk type can be determined using a greedy strategy based on the behavioral characteristics of the current traffic training sample. The corresponding reward value and next behavioral characteristics are then obtained for use in training the Q network. The specific process is shown in Figure 6 and includes the following steps:

[0132] S5011: Generate a random number within a preset range for each traffic training sample according to a preset random seed.

[0133] In one example, different random seeds generate different random numbers. By comparing the random numbers with a preset probability threshold, the risk type of each traffic training sample can be predicted in different ways.

[0134] Optionally, the default range of random numbers is [0-1].

[0135] S5012: For traffic training samples whose random numbers are less than a preset probability threshold, randomly select one from each risk type as the predicted risk type of the traffic training sample.

[0136] In one example, when the random number of the traffic training sample is less than a preset probability threshold, one is randomly selected from no risk, low risk, medium risk, and high risk as the predicted risk type.

[0137] For example, assuming that the preset probability threshold is 0.5 and the random number of the traffic training sample is 0.32, if the random number is less than the preset probability threshold, the randomly selected medium risk is used as the predicted risk type of the traffic training sample.

[0138] S5013: Using the Q network, determine a first evaluation value of the traffic training sample according to the behavioral characteristics and predicted risk type of the traffic training sample.

[0139] In one example, for a traffic training sample whose random number is less than a preset probability threshold, the Q network calculates a first evaluation value based on the input behavior characteristics and the predicted risk type.

[0140] S5014: For traffic training samples whose random numbers are not less than a preset probability threshold, a Q network is used to obtain the probability that the traffic training samples belong to each risk type based on the behavioral characteristics of the traffic training samples.

[0141] In one example, when the random number of a traffic training sample is greater than or equal to a preset probability threshold, after the traffic training sample is input into the Q network, the Q network calculates the probability of it belonging to the four risk types of no risk, low risk, medium risk and high risk based on the behavioral characteristics.

[0142] S5015: Multiply each obtained probability by the one-hot-encoded vector of each risk type to obtain a first evaluation value of each risk type.

[0143] If the behavioral characteristics and the first assessment value of each risk type are calculated directly, each risk type corresponding to the behavioral characteristics needs to be forwarded once, which requires a large amount of calculation.

[0144] In one example, in order to reduce the computational cost of the first evaluation value, the four risk types of no risk, low risk, medium risk and high risk are one-hot encoded to obtain a vector represented by a binary system. The vector is then dot-multiplied with the probabilities corresponding to the four risk types and summed to obtain a scalar containing the first evaluation values ​​corresponding to the four risk types respectively.

[0145] S5016: The risk type corresponding to the maximum first evaluation value is used as the predicted risk type of the traffic training sample.

[0146] In one example, the formula for predicting risk type is as follows:

[0147] Among them, i=1,2,3,4, a i They represent the four risk types: no risk, low risk, medium risk and high risk respectively, θ is the network parameter of the Q network in this round of training, s is the behavioral feature, and a is the predicted risk type.

[0148] In the embodiment of the present disclosure, the preset probability threshold decreases as the number of iterations increases. This is because at the beginning of training, a larger probability threshold is set to make the network more exploratory. As the number of training times increases, the network becomes more and more stable. The exploratory nature can be reduced by appropriately reducing the probability threshold, making it more inclined to the network's own judgment.

[0149] It should be noted that the embodiment of the present disclosure does not impose any restrictive requirements on the decreasing method of the preset probability threshold. It can be a linear decrease, such as 0.5 during the first training, 0.4 during the second training, and 0.3 during the third training; it can also be a nonlinear decrease, such as 0.5 during the first training, 0.4 during the second training, and 0.25 during the third training.

[0150] S5017: Determine a reward value for each traffic training sample based on the predicted risk type and the labeled actual risk type of the traffic training sample.

[0151] For example, the predicted risk type of a traffic training sample is no risk, while its corresponding true risk type is low risk, that is, the Q network prediction is wrong and the reward value is -1.

[0152] For another example, the predicted risk type of a traffic training sample is medium risk, while its corresponding true risk type is medium risk, that is, the Q network prediction is correct and the reward value is 1.

[0153] S502: The behavior characteristics, predicted risk type, reward value of the current traffic training sample and the behavior characteristics of the next traffic training sample are stored as a piece of experience data in a preset experience recovery pool.

[0154] In practical applications, neural networks can perform batch processing, but if a single sample is used for training, each sample and the corresponding gradient will have a large variance and convergence will be slow. Therefore, after obtaining the reward value corresponding to the predicted risk type of each traffic training sample and the next traffic training sample through the Q network, the four elements of the behavior characteristics of the current traffic training sample, the predicted risk type, the reward value, and the behavior characteristics of the next traffic training sample are stored as a piece of experience data in a preset experience recycling pool. This allows for random sampling of several pieces of data from the experience recycling pool for batch processing during training.

[0155] In the embodiment of the present disclosure, by using the behavioral characteristics of the current traffic training sample, the behavioral characteristics of the next traffic training sample and the reward value corresponding to the predicted risk type of the current training sample as an experience data for training, the neural network can make full use of the context information in the database session, thereby improving the accuracy of abnormal traffic detection. At the same time, by storing the data in the experience recycling pool for sampling, the correlation between the data is broken, which facilitates the reuse of previously collected data and allows the network to converge faster.

[0156] S503: For each batch of experience data uniformly sampled in the preset experience recovery pool, the target Q network is used to determine the second evaluation value corresponding to the predicted risk type of the next traffic training sample according to the behavioral characteristics of the next traffic training sample. The reward value and the first evaluation value of the current traffic training sample are combined to determine the detection loss of the traffic detection model, and the network parameters of the first neural network and the second neural network are adjusted according to the detection loss.

[0157] In one example, after uniformly sampling a batch of experience data from the experience replay pool, backpropagation is performed using mean square error loss, and the network parameters are updated using gradient descent. The specific process is shown in Figure 7 and mainly includes the following steps:

[0158] S5031: Using the target Q network, according to the behavioral characteristics of the next traffic training sample in each piece of experience data and the vector after the one-hot encoding of each risk type, the second evaluation value of each risk type is obtained, and the risk type corresponding to the maximum second evaluation value is used as the predicted risk type of the next traffic training sample.

[0159] In the traffic detection model built by the DQN algorithm, the Q network obtains the current predicted risk type and the corresponding first initial value from the behavioral characteristics of the current traffic training sample in each sampled experience data, and the target Q network obtains the next predicted risk type and the corresponding second initial value from the behavioral characteristics of the next traffic training sample in each sampled experience data.

[0160] Since the network structures of the Q network and the target Q network are consistent, the prediction process of the target Q network is consistent with that of the Q network. First, the target Q network obtains the probabilities corresponding to no risk, low risk, medium risk and high risk according to the behavioral characteristics of the next traffic training sample, and then multiplies these four probabilities with the vectors of the four risk types after one-hot encoding to obtain the second evaluation values ​​of the four risk types at once. Then, the risk type corresponding to the maximum second evaluation value is used as the predicted risk type of the next traffic training sample.

[0161] S5032: The sum of the second evaluation value corresponding to the predicted risk type of the next traffic training sample in each piece of experience data and the reward value of the current traffic training sample, and the difference between the sum and the first evaluation value corresponding to the current traffic training sample, is used as the detection loss of the traffic detection model.

[0162] The formula for detection loss is as follows:

[0163] Among them, r j represents the reward value, Q′ represents the second evaluation value, Q represents the first evaluation value, s j+1 Indicates the behavioral characteristics of the next traffic training sample, a j+1 Indicates the predicted risk type of the next traffic training sample, s j Indicates the behavioral characteristics of the current traffic training sample, a j represents the predicted risk type of the current traffic training sample, θ′ is the network parameter of the target Q network, θ represents the network parameter of the Q network, and γ represents the network coefficient.

[0164] S5033: Adjust the network parameters of the Q network according to the detection loss obtained in each round of training.

[0165] After each round of training, the network parameters of the Q network are adjusted using the gradient descent method according to the detection loss.

[0166] S5034: After the network parameters of the Q network are adjusted C times, part of the network parameters of the Q network are used as the network parameters of the target Q network.

[0167] Where C is an integer greater than 1. During training, updates to the Q network cause the target to constantly change. However, copying the network parameters of the target Q network to a target Q network with the same network structure every C times ensures that the second evaluation value remains stable in a short period of time, thereby improving the stability of model training.

[0168] In one example, in order to improve the feature extraction capability, the Q network and the target Q network can adopt convolutional neural networks (CNN), deep neural networks (DNN), multilayer perceptrons (MLP), etc. in deep learning, and the embodiments of the present disclosure do not make any restrictive requirements.

[0169] Taking CNN as an example, the partial structure of the Q network and target Q network is shown in Figure 8A. The first half of the network consists of two consecutive convolutional layers, which are used to extract features from the payload of the preprocessed traffic training samples. The extracted features are then passed through a pooling layer to reduce information redundancy. The pooled two-dimensional tensor is then compressed into a one-dimensional vector and fed into the second half of the fully connected network. The fully connected network consists of two layers, the last of which uses a Softmax classifier to implement the multi-classification task.

[0170] Optionally, the convolution kernel sizes of the two convolutional layers are 1*5 and 4*1 respectively, and the size of the last fully connected layer is 512*4.

[0171] When detecting abnormal database traffic, there are multiple risk types that can be set. If the Q network directly outputs the first assessment value of a risk type corresponding to each behavioral feature, then all risk types corresponding to each behavioral feature need to be forwarded once, and the predicted risk type with the largest first assessment value is found through comparison. This results in the amount of calculation increasing linearly with the number of risk types.

[0172] Therefore, to reduce the computational cost of the first assessment value, the embodiments of this disclosure make partial improvements to the Q network and target Q network. Figure 8B shows a schematic diagram of the complete improved network structure. In this improvement, the Q network first outputs the probabilities of all possible risk types given a given behavioral feature. It then performs one-hot encoding on all possible risk types given that behavioral feature. Furthermore, the encoded vector is dot-multiplied by the probabilities of all possible risk types and summed to calculate the first assessment value for each risk type, thus avoiding repeated calculations.

[0173] In the embodiments of the present disclosure, the Q network and the target Q network in the traffic detection model use convolutional neural networks for feature extraction, and output the probability value of each risk type through a fully connected layer and a Softmax function. The network structure is simple; at the same time, the Q network and the target Q network can take into account both the prediction of risk types and the output of evaluation values, thereby effectively reducing the computational complexity.

[0174] Generally speaking, the stability and generalization ability of a single network model are relatively weak. In order to obtain higher performance and accuracy, the embodiment of the present disclosure adopts a multi-model fusion strategy.

[0175] In one example, the multi-model selection process is shown in Figure 9. It mainly includes the following steps:

[0176] S504: For at least one candidate traffic detection model that reaches a preset number of times, test the target Q network in the candidate traffic detection model according to the traffic test sample set generated by the database to obtain a reward value for each traffic test sample set.

[0177] Since reinforcement learning is very sensitive to the setting of random seeds, different traffic detection models can be trained by reinitializing different random seeds.

[0178] During the training process of each traffic detection model, when the training times of the traffic detection model reach the preset times threshold, a candidate traffic detection model is obtained and tested with the traffic test sample set generated by the database to obtain a model with better training results.

[0179] In one example, for each traffic detection model, 512 training times constitute one round, and the candidate traffic detection model is tested once every 100 rounds. The amount of data in the traffic test sample set is 1,000 data items.

[0180] In specific implementation, after obtaining the candidate traffic detection model, the preprocessed traffic test sample set is input into the target Q network in the candidate traffic detection model to obtain the predicted risk type of each traffic test sample, and combined with the marked real risk type to obtain the reward value of each traffic test sample.

[0181] It should be noted that the number of training times corresponding to the candidate traffic detection model can be adjusted according to actual conditions, and the embodiments of the present disclosure do not impose any restrictive requirements. In addition, the preprocessing process of the traffic test sample set is consistent with the preprocessing process of the traffic training sample set, and will not be repeated here.

[0182] S505: Obtain the test result of the target Q network according to the reward value of each traffic test sample.

[0183] In one example, a correct risk type prediction is scored as 1 point, and an incorrect prediction is scored as -1 point. Therefore, the sum of the reward values ​​of each traffic test sample can be used as the test result of the target Q network. The larger the sum of the reward values, the more accurate the model training.

[0184] S506: Select at least one target traffic detection model from at least one candidate traffic detection model in descending order of the test result of at least one target Q network.

[0185] In one example, the test results of each target Q network are sorted from high to low, and then the candidate traffic detection models corresponding to the first Z target Q networks are selected as the target traffic detection model, where Z is an integer greater than 1.

[0186] See Figure 10 for the complete model training process, which mainly includes the following steps:

[0187] S1001: Label the collected database traffic data to generate a traffic training sample set and a traffic test sample set.

[0188] S1002: Perform preprocessing operations such as cleaning, length consistency, and normalization on the traffic training sample set and the traffic test sample set.

[0189] S1003: Perform iterative training on the current traffic detection model corresponding to the current random seed according to the traffic training sample set.

[0190] S1004: When the number of training times reaches a preset threshold, the candidate traffic detection model obtained through training is tested according to traffic test samples.

[0191] S1005: Update the random seed to obtain a new traffic detection model, use the new random seed as the current seed, use the new traffic detection model as the current traffic detection model, and return to S1003 for iterative training.

[0192] S1006: Determine whether the number of updates of the random seed meets the predicted number threshold. If so, execute S1007; otherwise, return to S1005.

[0193] S1007: Stop random seed updating, perform test results on each candidate traffic detection model, and select at least one target traffic detection model.

[0194] In the embodiments of the present disclosure, a deep reinforcement learning approach is employed to address the problem of abnormal database traffic detection. A complete process is presented, encompassing problem modeling, dataset annotation, data preprocessing, network model design, and multi-model integration, laying the foundation for abnormal database traffic detection. Furthermore, deep reinforcement learning overcomes the technical limitations of traditional database abnormal traffic detection, avoiding tedious steps such as traffic parsing and rule matching, and reducing manual intervention. Through learning, detection is performed directly from features extracted from encrypted or unencrypted traffic data, resulting in higher generalization capabilities. Furthermore, it effectively addresses the drawback of a limited rule base that can only detect a limited number of attacks. It can also detect new, previously unseen attacks, thereby better maintaining database security and allowing the model to automatically update to adapt to new changes. Furthermore, deep reinforcement learning improves the utilization of contextual information in database connection sessions to a certain extent, further enhancing the accuracy of model detection. Furthermore, the embodiments of the present disclosure eliminate invalid traffic data and invalid data portions within traffic data, thereby reducing the interference of invalid information on traffic features and improving the accuracy of model detection. Furthermore, the real-time response capability of the model is enhanced, enabling rapid identification of potential threats, thereby strengthening the protection of sensitive data and improving the security and reliability of the database.

[0195] Based on at least one trained target traffic detection model, the traffic generated by the database can be detected in real time, thereby maintaining the security of the database.

[0196] In practical applications, the detection method for abnormal database traffic based on DQN is shown in Figure 11, which mainly includes the following steps:

[0197] S1101: Acquire initial traffic data of a target database, and pre-process the initial traffic data to obtain target traffic data.

[0198] Among them, the preprocessing operations include data cleaning (such as the removal of handshake messages, Ethernet headers, IP datagram headers, TCP / UDP headers, etc.), length consistency processing (such as unification to 1024 bytes) and normalization processing. The specific process is shown in Figure 3B and will not be repeated here.

[0199] S1102: Input the target traffic data into at least one target traffic detection model respectively, and predict at least one initial risk type of the target traffic data according to the behavioral characteristics of the target traffic data.

[0200] During specific implementation, for each target traffic detection model, the target Q network determines the initial risk type of the target traffic data based on the behavioral characteristics of the target traffic data.

[0201] S1103: Fusing the at least one obtained initial risk type to obtain a target risk type of the target traffic data.

[0202] As shown in Figure 12, which is a schematic diagram of multi-model fusion, after the target traffic data is input into Z target traffic detection models, each target Q network will output an initial risk type represented by 0-3. By averaging these initial risk types according to the preset weights and then rounding them, a numerical value is obtained, and the final target risk type is determined based on this numerical value.

[0203] S1104: When the target risk type is abnormal, an alarm message corresponding to the target risk type is sent.

[0204] When the target risk type of the target traffic data is determined to be an abnormal type such as low risk, medium risk, or high risk, the data security is maintained by issuing an alarm message.

[0205] Among them, the prompting methods of warning information corresponding to different risk levels can be different.

[0206] In the disclosed embodiment, the actual behavior pattern of database network traffic is analyzed through a trained model, thereby detecting abnormal traffic data and determining the risk level, thereby issuing an abnormality alarm to maintain data security.

[0207] For organizations in regulated industries and under regulatory requirements, abnormal traffic detection can be used to address aspects such as abnormal records and data privacy, thereby improving regulatory requirements and compliance construction, and has important research significance and value.

[0208] The target flow detection model provided by the embodiments of the present disclosure can be implemented and applied in different scenarios.

[0209] Scenario 1: Database protection systems for governments and financial institutions

[0210] In this scenario, governments and financial institutions need to ensure the security of sensitive information, and abnormal traffic detection can monitor access to government and financial institution databases, thereby detecting unauthorized data queries or leaks and other operations, and maintaining public safety and privacy security.

[0211] In practical applications, before using the target traffic detection model of the disclosed embodiments, a traffic data collector must be deployed on the client to send the collected traffic data to the server address where the model is deployed. Once the model obtains the client's traffic data, it automatically determines whether the traffic data is abnormal and triggers an alarm if an anomaly is found, thereby improving the system's security protection capabilities.

[0212] In specific implementation, the application process of the target traffic detection model is shown in Figure 13, which mainly includes the following steps:

[0213] S1301: After deploying at least one target traffic detection model on the server, the server's IP address and port are sent to the client to receive database traffic reported by the client.

[0214] S1302: The database port of the client is used to install the collector corresponding to the server.

[0215] S1303: The client collects all traffic data of the database port through the collector and performs pre-processing.

[0216] S1304: The client sends all pre-processed traffic data to the server via the received IP address and port number.

[0217] S1305: The server uses at least one target traffic detection model to obtain a target risk type for each traffic data.

[0218] S1306: Send an alarm message to the client for traffic data with abnormal target risk type.

[0219] S1307: The client checks the database according to the alarm information.

[0220] It should be noted that the preprocessing can be performed by the client or the server.

[0221] For the database protection systems of governments and financial institutions, the client checks whether there are any abnormal operations through the alarm information returned by the server, thereby avoiding security losses.

[0222] Scenario 2: Detection systems in industrial production testing

[0223] In industrial manufacturing scenarios, databases monitoring production lines can help detect equipment failures or abnormal operating modes. When equipment generates abnormal flow or abnormal sensor data, it can automatically shut down to avoid damaging products.

[0224] Before applying the model of the embodiment of the present disclosure in the field of industrial production detection, it is necessary to retrain the model using the method of the embodiment of the present disclosure so that it can identify abnormal traffic in the industrial production field.

[0225] The specific application process is shown in Figure 14, which mainly includes the following steps:

[0226] S1401: The server retrains or fine-tunes the model based on data from the industrial production field to obtain at least one target detection model.

[0227] Through retraining or fine-tuning, the model can identify abnormal data in the industrial production field.

[0228] S1402: The server sends the IP address and port to the client to receive the database data reported by the client.

[0229] S1403: The database port of the client is used to install the collector corresponding to the server.

[0230] S1404: The server collects all traffic data of the database port of the client through the collector and performs pre-processing.

[0231] S1405: The client sends all pre-processed traffic data to the server via the received IP address and port number.

[0232] S1406: The server uses at least one target detection model to obtain the target risk type of each industrial data.

[0233] S1407: Send an alarm message to the client for industrial data with abnormal target risk type.

[0234] S1408: The client performs device maintenance based on the alarm information.

[0235] For detection systems in the field of industrial production inspection, the client performs equipment maintenance (such as shutdown inspection, component replacement, etc.) based on the alarm information returned by the server, thereby avoiding economic losses.

[0236] Scenario 3: Security Systems in E-commerce

[0237] In this scenario, e-commerce websites are vulnerable to distributed denial of service (DDoS) attacks. When a system is under a DDoS attack, the database may become unstable or unresponsive due to the large number of requests, resulting in database access issues. Abnormal traffic detection can identify anomalous requests and then utilize other devices to divert these requests to a dedicated firewall or CDN to mitigate the attack.

[0238] In actual application, a flow data collector is deployed on the database port of the client, and then the collected flow data is sent to the server where the model is deployed, so as to automatically issue early warnings for the flow data. The specific embodiment process is consistent with the application of the model in the field of industrial production detection and will not be repeated here.

[0239] Based on the same technical concept, the embodiment of the present disclosure also provides a DQN-based database abnormal traffic detection device, which can implement the steps of any of the above-mentioned DQN-based database abnormal traffic detection methods and achieve the same technical effects.

[0240] Referring to FIG15 , the detection device includes a model training module 15 for iteratively training at least one initialized flow detection model based on a flow training sample set generated by a database, wherein each flow detection model includes a Q network constructed by a neural network and a target Q network;

[0241] The model training module includes a risk prediction unit 1501, an experience recovery unit 1502, a loss calculation unit 1503, and a parameter adjustment unit 1504. During each round of training, each unit of the model training module performs the following operations:

[0242] The risk prediction unit 1501 is configured to use the Q network to obtain a first evaluation value corresponding to the predicted risk type of each traffic training sample based on the behavioral characteristics of the corresponding traffic training sample, and determine a reward value for each traffic training sample in combination with the labeled real risk types; and, for each batch of experience data uniformly sampled from the preset experience recovery pool, use the target Q network to determine a second evaluation value corresponding to the predicted risk type of the next traffic training sample based on the behavioral characteristics of the next traffic training sample;

[0243] The experience recovery unit 1502 is used to store the behavior characteristics, predicted risk type, reward value of the current traffic training sample and the behavior characteristics of the next traffic training sample as a piece of experience data in a preset experience recovery pool;

[0244] a loss calculation unit 1503, configured to determine the detection loss of the traffic detection model according to the second evaluation value corresponding to the predicted risk type of the next traffic training sample, the reward value of the current traffic training sample, and the first evaluation value;

[0245] The parameter adjustment unit 1504 is configured to adjust network parameters of the Q network and the target Q network according to the detection loss.

[0246] Optionally, the risk prediction unit 1501 is specifically configured to:

[0247] Generate a random number within a preset range for each traffic training sample based on a preset random seed, where the random seeds for different traffic detection models are different;

[0248] For the traffic training sample whose random number is less than a preset probability threshold, randomly select one from each risk type as a predicted risk type, and use the Q network to determine a first evaluation value of the traffic training sample based on the behavioral characteristics of the traffic training sample and the predicted risk type;

[0249] For the traffic training samples whose random numbers are not less than the preset probability threshold, the Q network is used to obtain the probability that the traffic training samples belong to each risk type based on the behavioral characteristics of the traffic training samples, and each obtained probability is multiplied by the vector after the one-hot encoding of each risk type to obtain a first evaluation value for each risk type, and the risk type corresponding to the maximum first evaluation value is used as the predicted risk type of the traffic training sample;

[0250] The preset probability threshold decreases as the number of iterations increases.

[0251] Optionally, the risk prediction unit 1501 is specifically configured to:

[0252] Using the target Q network, based on the behavioral characteristics of the next traffic training sample in each piece of empirical data and the one-hot encoded vector of each risk type, a second evaluation value of each risk type is obtained, and the risk type corresponding to the maximum second evaluation value is used as the predicted risk type of the next traffic training sample;

[0253] The loss calculation unit 1503 is specifically used to:

[0254] The second evaluation value corresponding to the predicted risk type of the next traffic training sample in each piece of experience data, the sum of the reward value of the current traffic training sample, and the difference between the sum and the first evaluation value corresponding to the current traffic training sample are used as the detection loss of the traffic detection model.

[0255] Optionally, the parameter adjustment unit 1504 is specifically configured to:

[0256] Adjusting the network parameters of the Q network according to the detection loss obtained in each round of training;

[0257] After the network parameters of the Q network are adjusted C times, part of the network parameters of the Q network are used as the network parameters of the target Q network, where C is an integer greater than 1.

[0258] Optionally, the model training module further includes a preprocessing unit 1505, configured to:

[0259] Eliminating invalid traffic training samples and invalid data parts in the traffic training samples from the traffic training sample set to obtain valid traffic samples;

[0260] For valid traffic samples whose packet length is less than a preset byte threshold, pad the end with 0 to make its packet length equal to the preset byte threshold; and, for valid traffic samples whose packet length is greater than the preset byte threshold, intercept the valid traffic samples starting from the head according to the preset byte threshold;

[0261] The valid traffic samples whose packet length is equal to the preset byte threshold are normalized.

[0262] Optionally, the preprocessing unit 1505 is specifically configured to:

[0263] Eliminating the traffic training samples that are concentrated on handshake messages generated during the network connection establishment process; and

[0264] Eliminating at least one invalid data portion among Ethernet head and tail headers, IP datagram headers and transport protocol headers in the traffic training samples captured by the traffic training sample concentration at the data link layer.

[0265] Optionally, the model training module further includes a testing unit 1506, configured to:

[0266] For at least one candidate traffic detection model that reaches the preset number threshold, testing the target Q network in the candidate traffic detection model based on the traffic test sample set generated by the database, obtaining a reward value for each traffic test sample set, and obtaining a test result of the target Q network based on the reward value of each traffic test sample;

[0267] At least one target traffic detection model is selected from the at least one candidate traffic detection model in descending order of the test result of at least one target Q network.

[0268] Optionally, the preprocessing unit 1505 is specifically configured to:

[0269] Acquiring initial flow data from a target database, and preprocessing the initial flow data to obtain target flow data;

[0270] The risk prediction unit 1501 is specifically used to:

[0271] inputting the target traffic data into at least one target traffic detection model respectively, and predicting at least one initial risk type of the target traffic data according to the behavioral characteristics of the target traffic data;

[0272] fusing the obtained at least one initial risk type to obtain a target risk type of the target traffic data;

[0273] When the target risk type is abnormal, an alarm message corresponding to the target risk type is sent.

[0274] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing the present disclosure, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0275] After introducing the method and apparatus for detecting abnormal database traffic based on DQN according to an exemplary embodiment of the present disclosure, an electronic device according to another exemplary embodiment of the present disclosure is introduced next.

[0276] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0277] Based on the same inventive concept as the above method embodiment, the electronic device provided in the embodiment of the present disclosure may be the server in Figure 1. In this embodiment, the structure of the electronic device may be as shown in Figure 16, including a memory 1601, a communication interface 1603, and one or more processors 1602.

[0278] Memory 1601 is used to store computer programs executed by processor 1602. Memory 1601 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0279] Memory 1601 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing a desired computer program in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1601 may be a combination of the aforementioned memories.

[0280] The processor 1602 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1602 is configured to implement the above-mentioned DQN-based database abnormal traffic detection method when calling the computer program stored in the memory 1601 .

[0281] The communication interface 1603 is used to communicate with terminal devices and other servers.

[0282] The specific connection medium between the memory 1601, communication interface 1603, and processor 1602 is not limited in the embodiments of the present disclosure. In FIG16 , the memory 1601 and processor 1602 are connected via bus 1604, which is depicted by a bold line in FIG16 . The connection methods between other components are merely illustrative and are not intended to be limiting. Bus 1604 can be divided into an address bus, a data bus, a control bus, and the like. For ease of description, FIG16 depicts only a single bold line, but this does not imply that there is only one bus or only one type of bus.

[0283] Memory 1601 stores a computer storage medium, which stores computer-executable instructions for implementing the DQN-based database abnormal traffic detection method according to an embodiment of the present disclosure. Processor 1602 is configured to execute the DQN-based database abnormal traffic detection method, as shown in FIG11 .

[0284] In some possible implementations, various aspects of the DQN-based database abnormal traffic detection method provided by the present disclosure can also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the DQN-based database abnormal traffic detection method according to various exemplary embodiments of the present disclosure described above in this specification. For example, the electronic device can execute the steps shown in Figures 5 and 11.

[0285] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0286] The program product of the embodiment of the present disclosure may be a portable compact disc read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with a command execution system, apparatus, or device.

[0287] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with a command execution system, apparatus, or device.

[0288] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0289] The computer program for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program may be executed entirely on the user electronic device, partially on the user electronic device, as a stand-alone software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving a remote electronic device, the remote electronic device may be connected to the user electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0290] It should be noted that although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0291] Furthermore, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0292] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0293] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0294] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0295] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0296] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A database abnormal traffic detection method based on DQN, comprising: According to the traffic training sample set generated by the database, at least one initialized traffic detection model is iteratively trained, wherein each traffic detection model includes a Q network constructed by a neural network and a target Q network, and each round of training process is as follows: Using the Q network, according to the behavioral characteristics of each traffic training sample, a first evaluation value corresponding to the predicted risk type of the corresponding traffic training sample is obtained, and the reward value of each traffic training sample is determined in combination with the marked real risk types; The behavior characteristics, predicted risk type, reward value of the current traffic training sample and the behavior characteristics of the next traffic training sample are stored as a piece of experience data in a preset experience recycling pool; For each batch of experience data uniformly sampled in the preset experience recovery pool, the target Q network is used to determine the second evaluation value corresponding to the predicted risk type of the next traffic training sample according to the behavioral characteristics of the next traffic training sample, and the detection loss of the traffic detection model is determined in combination with the reward value and the first evaluation value of the current traffic training sample, and the network parameters of the Q network and the target Q network are adjusted according to the detection loss.

2. The method according to claim 1, wherein the Q network is used to obtain a first evaluation value corresponding to the predicted risk type of each traffic training sample according to the behavior characteristics of each traffic training sample, comprising: Generate a random number within a preset range for each traffic training sample according to a preset random seed, wherein the random seeds of different traffic detection models are different; For the traffic training sample whose random number is less than the preset probability threshold, randomly select one from each risk type as the predicted risk type, and use the Q network to determine the first evaluation value of the traffic training sample according to the behavior characteristics of the traffic training sample and the predicted risk type; For the traffic training samples whose random numbers are not less than the preset probability threshold, the Q network is used to obtain the probability that the traffic training samples belong to each risk type according to the behavioral characteristics of the traffic training samples, and the obtained probabilities are multiplied by the vectors after the one-hot encoding of each risk type to obtain the first evaluation value of each risk type, and the risk type corresponding to the maximum first evaluation value is used as the predicted risk type of the traffic training sample; The preset probability threshold decreases as the number of iterations increases.

3. The method according to claim 1, wherein the target Q network is used to determine the second evaluation value corresponding to the predicted risk type of the next traffic training sample according to the behavior characteristics of the next traffic training sample, and the detection loss of the traffic detection model is determined by combining the reward value and the first evaluation value of the current traffic training sample, including: Using the target Q network, according to the behavior characteristics of the next traffic training sample in each piece of experience data and the vector after the one-hot encoding of each risk type, the second evaluation value of each risk type is obtained, and the risk type corresponding to the maximum second evaluation value is used as the predicted risk type of the next traffic training sample; The sum of the second evaluation value corresponding to the predicted risk type of the next traffic training sample in each piece of experience data and the reward value of the current traffic training sample and the difference between the sum and the first evaluation value corresponding to the current traffic training sample are used as the detection loss of the traffic detection model.

4. The method of claim 1, wherein adjusting network parameters of the Q network and the target Q network according to the detection loss comprises: Adjusting the network parameters of the Q network according to the detection loss obtained in each round of training; After the network parameters of the Q network are adjusted C times, part of the network parameters of the Q network are used as the network parameters of the target Q network, where C is an integer greater than 1.

5. The method according to any one of claims 1 to 4, characterized in that Before iteratively training at least one traffic detection model based on the traffic training sample set generated by the database, the method further includes: Eliminate invalid traffic training samples in the traffic training sample set and invalid data parts in the traffic training samples to obtain valid traffic samples; For valid traffic samples whose packet length is less than a preset byte threshold, fill the end with 0 to make its packet length equal to the preset byte threshold, and, for valid traffic samples whose packet length is greater than the preset byte threshold, intercept from the head of the valid traffic sample according to the preset byte threshold; The valid traffic samples whose packet length is equal to the preset byte threshold are normalized.

6. The method according to claim 5, characterized in that The step of removing invalid traffic training samples in the traffic training sample set and invalid data parts in the traffic training samples to obtain valid traffic samples includes: Eliminating the traffic training samples and concentrating on the handshake messages generated during the network connection establishment process; and, At least one invalid data portion among Ethernet head and tail headers, IP datagram headers and transport protocol headers in the traffic training samples captured by the traffic training sample concentration at the data link layer is eliminated.

7. The method according to any one of claims 1 to 4, when the number of training times of the at least one traffic detection model reaches a preset number threshold, the method further comprises: For at least one candidate traffic detection model that reaches the preset number of times threshold, test the target Q network in the candidate traffic detection model according to the traffic test sample set generated by the database, obtain the reward value of each traffic test sample set, and obtain the test result of the target Q network according to the reward value of each traffic test sample; At least one target traffic detection model is selected from the at least one candidate traffic detection model in descending order of the test result of at least one target Q network.

8. The method according to claim 7, after selecting at least one target traffic detection model, the method further comprises: Acquire initial flow data of a target database, and preprocess the initial flow data to obtain target flow data; Inputting the target flow data into at least one target flow detection model respectively, and predicting at least one initial risk type of the target flow data according to the behavior characteristics of the target flow data; Fusing the obtained at least one initial risk type to obtain a target risk type of the target traffic data; When the target risk type is abnormal, an alarm message corresponding to the target risk type is sent.

9. A database abnormal traffic detection device based on DQN, comprising: The model training module is used to iteratively train at least one initialized traffic detection model according to the traffic training sample set generated by the database, wherein each traffic detection model includes a Q network constructed by a neural network and a target Q network. During each round of training, each unit of the model training module performs the following operations: A risk prediction unit, configured to use the Q network to obtain a first evaluation value corresponding to the predicted risk type of the corresponding traffic training sample according to the behavioral characteristics of each traffic training sample, and determine the reward value of each traffic training sample in combination with the marked real risk types; and, for each batch of experience data uniformly sampled in the preset experience recovery pool, use the target Q network to determine a second evaluation value corresponding to the predicted risk type of the next traffic training sample according to the behavioral characteristics of the next traffic training sample; An experience recovery unit is used to store the behavior characteristics, predicted risk type, reward value of the current traffic training sample and the behavior characteristics of the next traffic training sample as a piece of experience data into a preset experience recovery pool; a loss calculation unit, configured to determine the detection loss of the traffic detection model according to the second evaluation value corresponding to the predicted risk type of the next traffic training sample, the reward value of the current traffic training sample, and the first evaluation value; A parameter adjustment unit is used to adjust network parameters of the Q network and the target Q network according to the detection loss.

10. An electronic device comprising a processor, a memory and a communication interface, wherein the communication interface, the memory and the processor are connected via a bus; The communication interface is used to send and receive data; The memory is used to store computer instructions; The processor is used to read the computer instructions in the memory and execute the steps of any one of claims 1 to 8.

11. A computer-readable storage medium, comprising a computer program, wherein when the computer program is executed on an electronic device, the computer program is used to cause the electronic device to execute the steps of any one of the methods of claims 1 to 8.

Images