Malicious code escape detection method and device

Abstract

The present invention relates to the field of computer technology, and provides a method and device for detecting malicious code escapes, the method comprising: obtaining different types of target detection data related to malicious code escape detection in a network system; extracting feature vectors of different types of target detection data respectively; fusing different feature vectors to obtain a fused multimodal feature vector; inputting the multimodal feature vector into a malicious code escape detection model constructed based on a reinforcement learning network to obtain a detection result output by the malicious code escape detection model, and outputting a corresponding response action when the detection result is that there is a malicious code escape. In the present invention, the features of target detection data of multiple different types of network systems are fused, and the reinforcement learning network can gradually optimize the malicious code escape detection model, improve the accuracy and timeliness of detection, and provide corresponding response measures according to the detection results, thereby achieving accurate and efficient malicious code escape detection and response.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a malicious code escape detection method and device. Background Art

[0002] At present, there are four main detection strategies for malicious code escape: memory monitoring, behavior analysis, network traffic monitoring and static analysis. Among them, memory monitoring can detect and respond to malicious code escape behavior through multiple methods such as memory forensic analysis, memory integrity check, real-time monitoring and anomaly detection. Behavioral analysis monitors and analyzes system behavior through sandbox technology, behavior fingerprint, behavior analysis tools, advanced behavior analysis technology, multi-level behavior analysis and automated response and defense mechanisms to identify and respond to malicious code escape behavior. Network traffic monitoring is a method of identifying and blocking malicious behavior by monitoring and analyzing network traffic. Malicious code escape usually involves communication with external control servers, data leakage or other network activities. By monitoring these network traffic, these malicious activities can be detected and blocked. Static analysis does not execute program code, but directly analyzes source code, binary files or intermediate codes to identify potential malicious behavior and escape. However, none of the above four methods can accurately and efficiently detect malicious code escape behavior. Summary of the invention

[0003] The present invention provides a malicious code escape detection method and device, which are used to solve the problem that the malicious code escape behavior cannot be accurately and efficiently detected in the prior art.

[0004] The present invention provides a malicious code escape detection method, which comprises the following steps.

[0005] Obtain different types of target detection data related to malicious code evasion detection in network systems.

[0006] The feature vectors of the different types of target detection data are extracted respectively.

[0007] Different feature vectors are fused to obtain a fused multimodal feature vector.

[0008] The multimodal feature vector is input into a malicious code escape detection model constructed based on a reinforcement learning network to obtain a detection result output by the malicious code escape detection model, and when the detection result is that a malicious code escape exists, a corresponding response action is output.

[0009] Among them, the malicious code escape detection model is trained based on the sample data tuple (state, action, reward, next_state), and the training is completed when the preset training times are reached or the loss function converges. The state state is represented by the multimodal feature vector, the action action is one of the actions in the action set, and the action set includes the corresponding response action of the detection result. The reward value reward is determined based on the accuracy of the detection result and the execution result of the response action. The next state next_state represents the multimodal feature vector corresponding to the network system after the action is executed.

[0010] According to a malicious code escape detection method provided by the present invention, the different types of target detection data include: network traffic data, system log data, memory snapshot data and file system change data.

[0011] The step of respectively extracting feature vectors of the different types of target detection data comprises the following steps.

[0012] The flow feature vector of the network flow data is extracted based on a bidirectional long short-term memory network.

[0013] A log feature vector of the system log data is extracted based on an attention model.

[0014] A memory snapshot feature vector of the memory snapshot data is extracted based on a first graph convolutional neural network.

[0015] A file system change feature vector of the file system change data is extracted based on a second graph convolutional neural network.

[0016] According to a malicious code escape detection method provided by the present invention, different feature vectors are fused to obtain a fused multimodal feature vector, including: inputting a traffic feature vector, a log feature vector, a memory snapshot feature vector and a file system change feature vector into a multi-layer perceptron model to obtain the multimodal feature vector fused and output by the multi-layer perceptron model.

[0017] According to a malicious code escape detection method provided by the present invention, a reward value reward is determined based on the accuracy of the detection result and the execution result of the response action, including: correctly identifying normal behavior and assigning reward as a first positive reward value.

[0018] Correctly identify the malicious code escape behavior and assign the reward as the second positive reward value.

[0019] Correctly identify the malicious code escape behavior and output effective response actions, and assign reward as the third positive reward value.

[0020] Misjudging normal behavior as malicious code escape behavior, the reward is assigned the first negative reward value.

[0021] Misjudged malicious code escape behavior as normal behavior, and assigned a reward of the second negative reward value.

[0022] The third positive reward value is greater than the second positive reward value, the second positive reward value is greater than the first positive reward value, and the first negative reward value is greater than the second negative reward value.

[0023] According to a malicious code escape detection method provided by the present invention, the malicious code escape detection model training process is as follows.

[0024] Construct reinforcement learning network structure, loss function and experience replay buffer.

[0025] Get the current state s of the network system at the current time step t , according to the current state s t Select and perform an action t , based on executing action a t The execution result gets the current reward value r t and the state s at the next time step t+1 , will (s t , a t , r t , s t+1 ) is stored in the experience replay buffer.

[0026] A sample data tuple is randomly sampled from the experience replay buffer and input into the reinforcement learning network to update the weight of the target Q network.

[0027] The training is completed when the preset number of training times is reached or the loss function converges.

[0028] According to a malicious code escape detection method provided by the present invention, different types of target detection data related to malicious code escape detection in a network system are obtained, including the following steps.

[0029] Obtain different types of source detection data related to malicious code evasion detection in network systems.

[0030] The different types of source detection data are preprocessed, and the preprocessing includes: normalization, format conversion, noise removal, redundant data removal, time synchronization and event alignment, and the preprocessed data is determined to be the target detection data.

[0031] The present invention also provides a malicious code escape detection device, which includes the following functional units.

[0032] The data acquisition unit is used to acquire different types of target detection data related to malicious code escape detection in the network system.

[0033] The feature extraction unit is used to extract feature vectors of the different types of target detection data respectively.

[0034] The feature fusion unit is used to fuse different feature vectors to obtain a fused multi-modal feature vector.

[0035] The model execution unit is used to input the multimodal feature vector into a malicious code escape detection model constructed based on a reinforcement learning network, obtain a detection result output by the malicious code escape detection model, and output a corresponding response action when the detection result is that a malicious code escape exists.

[0036] Among them, the malicious code escape detection model is trained based on the sample data tuple (state, action, reward, next_state), and the training is completed when the preset training times are reached or the loss function converges. The state state is represented by the multimodal feature vector, the action action is one of the actions in the action set, and the action set includes the corresponding response action of the detection result. The reward value reward is determined based on the accuracy of the detection result and the execution result of the response action. The next state next_state represents the multimodal feature vector corresponding to the network system after the action is executed.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the malicious code escape detection method described in any one of the above is implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the malicious code escape detection method as described in any one of the above is implemented.

[0039] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the malicious code escape detection method as described above is implemented.

[0040] The malicious code escape detection method and device provided by the present invention obtain different types of target detection data related to malicious code escape detection in a network system; extract the feature vectors of the different types of target detection data respectively; fuse the different feature vectors to obtain the fused multimodal feature vector; input the multimodal feature vector into a malicious code escape detection model constructed based on a reinforcement learning network to obtain the detection result output by the malicious code escape detection model, and output the corresponding response action when the detection result is that there is a malicious code escape. In the present invention, the features of target detection data of multiple different types of network systems are fused to provide more comprehensive network system information, which helps the reinforcement learning network to more accurately identify malicious code escape behavior, and by continuously performing actions in the environment, observing results and adjusting strategies according to reward values, the reinforcement learning network can gradually optimize the malicious code escape detection model, improve the accuracy and timeliness of detection, and provide corresponding response measures according to the detection results, thereby achieving accurate and efficient malicious code escape detection and response. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flow chart of the malicious code escape detection method provided by the present invention.

[0043] Figure 2 It is a structural schematic diagram of the malicious code escape detection device provided by the present invention.

[0044] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In the related technologies, there are four main detection strategies for malicious code escape: memory monitoring, behavior analysis, network traffic monitoring and static analysis.

[0047] Memory monitoring can detect and respond to malicious code escape behaviors through multiple methods such as memory forensic analysis, memory integrity check, real-time monitoring, and anomaly detection. For the memory monitoring strategy, it is necessary to continuously access and analyze the memory content, which consumes the system's memory resources and may cause system performance degradation. At the same time, memory monitoring may misjudge some normal application behaviors as malicious behaviors, resulting in a high false alarm rate, requiring additional manpower and time for verification and troubleshooting.

[0048] Behavioral analysis monitors and analyzes system behavior, identifies and responds to malicious code escaping behavior through sandbox technology, behavioral fingerprints, behavioral analysis tools, advanced behavioral analysis technology, multi-level behavioral analysis, and automated response and defense mechanisms. This strategy requires a certain amount of time to collect sufficient data and perform processing and analysis, which may lead to detection delays and inability to achieve efficient detection.

[0049] Network traffic monitoring is a method of identifying and preventing malicious behavior by monitoring and analyzing network traffic. Malicious code escapes usually involve communication with external control servers, data leakage, or other network activities. By monitoring these network traffic, these malicious activities can be effectively detected and prevented. This strategy depends on the specific network environment and configuration. Different network architectures may require different detection strategies and adjustments, which increases the complexity of implementation. In addition, more and more network communications use encryption technology (such as TLS / SSL), which makes it difficult to directly analyze traffic content. Detection can only rely on metadata (such as traffic patterns and behavioral characteristics), which may reduce the accuracy of detection.

[0050] Static analysis identifies potential malicious behavior and escapes by directly analyzing source code, binary files, or intermediate code without executing program code. However, it cannot capture the dynamic behavior of malicious code. Moreover, as the amount of code increases, the performance and scalability of static analysis tools will become a detection bottleneck, resulting in long analysis time and failure to provide detection results in a timely manner.

[0051] In summary, the above four methods are unable to accurately and efficiently detect malicious code escape behavior. Based on this, an embodiment of the present invention provides a malicious code escape detection method. The specific process is as follows: Figure 1 As shown, the process includes the following steps S110 to S140.

[0052] Step S110: Acquire different types of target detection data related to malicious code escape detection in the network system. Specifically, in the network system, different types of target detection data include: network flow data, system log data, memory snapshot data, and file system change data.

[0053] Network traffic data includes: IP address, port, protocol, packet size and other data. Specifically, network traffic monitoring tools can be used to capture network packets and obtain network traffic data from them. Network traffic data can be used to analyze network communication behavior and identify abnormal traffic patterns.

[0054] System log data includes: log information of various events and activities in the system (operating system logs, security logs and application logs). By collecting log information of various events and activities in the network system, system behavior can be analyzed and abnormal operations and potential attack behaviors can be detected.

[0055] Memory snapshot data includes: process, thread, memory usage and other status information. By regularly capturing system memory snapshots through memory forensics tools, the activity and escape behavior of malicious code in memory can be detected based on the current process, thread and memory usage status information in the memory.

[0056] File system change data includes: file creation, modification, deletion operations and changes in file content. File system change data is used to detect abnormal activities and potential threats in the file system.

[0057] Step S120: extracting feature vectors of the different types of target detection data respectively.

[0058] Step S130: Fusing different feature vectors to obtain a fused multi-modal feature vector.

[0059] Specifically, the above two steps extract and fuse features of different types of target detection data, providing high-quality feature representation for the subsequent malicious code escape detection model built based on the reinforcement learning network (Deep Q-Network, DQN).

[0060] Step S140: Input the multimodal feature vector into a malicious code escape detection model constructed based on a reinforcement learning network, obtain the detection result output by the malicious code escape detection model, and output the corresponding response action when the detection result is that there is a malicious code escape. The malicious code escape detection model is trained based on the sample data tuple (state, action, reward, next_state), and the training is completed when the preset number of training times is reached or the loss function converges. The state state is represented by the multimodal feature vector, the action action is one of the actions in the action set, and the action set includes the response action corresponding to the detection result. The reward value reward is determined based on the accuracy of the detection result and the execution result of the response action. The next state next_state represents the multimodal feature vector corresponding to the network system after the action is executed.

[0061] Specifically, the reinforcement learning network learns the optimal detection strategy by interacting with the system environment. The reinforcement learning network includes four parts: state, action, reward and environment. In this embodiment, the environment can be understood as the current system operation state of the network system. In different system operation states, the corresponding multimodal feature vector representation can be obtained according to steps S110 to S130, and the multimodal feature vector is used as the state. The action is the decision of the model in the current state, that is, whether there is a malicious code escape behavior and if there is a malicious code escape, the corresponding response action is output. In the case of the current state, an action in the action set will be selected and executed. The action set includes: triggering further scanning, isolating suspicious processes, recording and reporting suspicious activities, etc. After executing the action, the network system will enter the next operation state, that is, the next state next_state of the reinforcement learning network will be obtained. The reward mechanism is formulated based on the accuracy of the detection results and the execution results of the response actions for malicious code escape.

[0062] In the malicious code escape detection method of this embodiment, the features of target detection data of multiple different types of network systems are integrated to provide more comprehensive network system information, which helps the reinforcement learning network to more accurately identify malicious code escape behavior. Moreover, by continuously performing actions in the environment, observing results and adjusting strategies according to reward values, the reinforcement learning network can gradually optimize the malicious code escape detection model, improve the accuracy and timeliness of detection, and provide corresponding response measures based on the detection results, thereby achieving accurate and efficient malicious code escape detection and response.

[0063] In some embodiments, for different types of target detection data, step S120 specifically includes the following feature extraction methods.

[0064] For network traffic data, the traffic feature vector of the network traffic data is extracted based on a bidirectional long short-term memory (Bi-LSTM) network. Specifically, the key information in the network traffic data is encoded as the input of the Bi-LSTM neural network. For example, the network converts various attributes of the traffic data packet (IP address, port number, protocol type, etc.) into numerical features, divides the time series data into time windows of fixed length, and the features in each time window constitute an input sample. For IP addresses, convert IP addresses (IPv4) into 32-bit binary numbers, such as "192.168.1.1", which is converted to 11000000.10101000.00000001.00000001; for ports, directly use the port number after standardization (map the port to the range of [0, 1)) as the feature vector; for protocol types, use one-hot encoding to map the protocol type into digital form, for example: map "TCP" to [1, 0, 0], and map "UDP" to [0, 1, 0]; for network traffic time series data, divide the network traffic data into fixed-length windows according to the event window, and combine the features in each window into an input data. Common network traffic data usually contains timestamp, IP address, port, protocol type, and packet size. Assume that a fixed-length time window is selected, each window contains 3 time steps, and the feature vectors of each time step are combined into a matrix as the input of the Bi-LSTM neural network. In this embodiment, a five-layer Bi-LSTM neural network is used. By stacking five layers, the long-distance dependencies in the network time series data can be effectively captured, and the network's expression ability and feature extraction ability as well as the generalization ability of the model can be improved.

[0065] For system log data, the log feature vector of the system log data is extracted based on the attention model. Specifically, based on the attention mechanism, feature extraction is performed on system logs such as operating system logs, security logs, and application logs. Word2vec is used to convert the log text sequence of each type of log into a vector representation as a word embedding. At the same time, based on position encoding PE The formula constructs positional embedding, where the position encoding PE The formula is as follows:

[0066] .

[0067] .

[0068] in,i represents the index of each element in the feature vector to be output, d Represents the dimension of the output feature vector, and pos represents the position of each word in the text sequence corresponding to the system log data.

[0069] The word embedding and positional embedding are added as the final input to build an attention model to extract features from the sequence. The attention mechanism can assign weights to each log event, so that the model can pay more attention to important events and reasonably integrate the information of these events into the feature extraction process, so as to better capture the key features in the system log and the correlation between the data features. In this embodiment, the attention mechanism is used to automatically identify and extract important features, dynamically adjust the weights, focus on key log fragments, improve the detection effect, effectively capture long-distance dependencies in the log, and identify complex attack patterns.

[0070] A memory snapshot feature vector of the memory snapshot data is extracted based on a first graph convolutional neural network (GCN). Specifically, the relationship between data such as process lists, thread information, and memory usage is established based on the first graph convolutional neural network. By converting the memory snapshot data into a graph representation, and using the first graph convolutional neural network for information propagation and node feature updates, global and local features can be extracted. In this embodiment, the memory snapshot data is represented as a graph structure, in which nodes represent data units (such as processes and threads), edges represent relationships between nodes (such as communication and calls), and an initial feature vector is assigned to each node to represent the attributes of the current node (such as CPU usage and memory usage, etc.). Construct an adjacency matrix representing the node connection relationship , use the first graph convolutional neural network layer to aggregate and update the node features according to the update formula, the update formula is as follows:

[0071] .

[0072] in is the normalized adjacency matrix, For the The node feature matrix of the layer, For the The learnable weight matrix of the layer, is the activation function. By stacking multiple layers of graph convolutional layers to gradually aggregate information from more distant nodes, the final node features are pooled to generate a memory snapshot feature vector at the graph structure level.

[0073] Extract the file system change feature vector of the file system change data based on the second graph convolutional neural network. Specifically, take the file as a node, the change relationship between the files as an edge, construct a graph structure of the file change, assign an initial feature vector to each file node, and construct a second graph convolutional neural network (GCN) for feature propagation and update on the graph structure. For the entire graph structure, graph feature aggregation can be performed to merge or aggregate the features of all file nodes to obtain an overall graph-level feature representation. In this embodiment, in the process of modeling the file system change as a graph structure, the node represents the file or directory, the edge represents the relationship between them (such as reference and hierarchy), and the initial feature value is assigned to each node according to the relevant attributes of each node (such as file size, permission and timestamp). Construct an adjacency matrix to represent the node connection relationship, use a graph convolution layer to aggregate neighbor node features, and extract meaningful change patterns. Stack multiple graph convolution layers to gradually aggregate information from more distant nodes, and use a nonlinear activation function (ReLU) to introduce model nonlinearity. Add a global pooling layer after the convolution layer to generate a graph-level file system change feature vector.

[0074] It should be noted that before Bi-LSTM, attention model, first graph convolutional neural network and second graph convolutional neural network are respectively applied to extract network traffic data, system log data, memory snapshot data and file system change data, each model needs to be trained separately. During training, each model can be trained by using various types of data samples and their corresponding labels. For example, for Bi-LSTM, it is trained by using pre-designed network traffic sample data and whether there is malicious code escape behavior as a label (no label is 0, and label is 1). In the application of each model, the output of the last hidden layer in each model is taken as the corresponding feature vector.

[0075] In some embodiments, step S130 specifically includes: inputting the traffic feature vector, log feature vector, memory snapshot feature vector and file system change feature vector into a multilayer perceptron (MLP) model to obtain the multimodal feature vector fused and output by the multilayer perceptron model. Specifically, the traffic feature vector, log feature vector, memory snapshot feature vector and file system change feature vector are input into the MLP, and feature extraction and fusion are performed through multiple fully connected layers in the MLP, and the feature vector output by the last layer is used as the fused multimodal feature vector. In this embodiment, the traffic feature vector, log feature vector, memory snapshot feature vector and file system change feature vector are used as input, where the dimension of the traffic feature vector is , the dimension of the log feature vector is , the dimension of the memory snapshot feature vector is , the dimension of the file system change feature vector is , concatenate all feature vectors into a large input vector with dimension . Construct 3 fully connected layers, and use The activation function is transformed nonlinearly to increase the nonlinear expression ability of the model. The output of the last layer is used as the fused multimodal feature vector to perform the malicious code escape identification task, that is, to identify whether the current system network has the risk of malicious code escape. When training MLP, the binary cross entropy loss (Binary Cross Entropy Loss) is used. ,in is the label of sample i (for example, if there is malicious code escape behavior, the label is 1, and if there is no malicious code escape behavior, the label is 0). is the probability that sample i is predicted to be a positive class, and N is the number of samples.

[0076] In this embodiment, a multi-layer perceptron is used to perform feature fusion on multimodal data, capture complex nonlinear relationships, automatically learn interactions and associations between different modalities, generate richer feature representations, and use fully connected layers to efficiently integrate and process large-scale data.

[0077] In some embodiments, the reward value reward is determined based on the accuracy of the detection result and the result of executing the response action, including the following determination methods.

[0078] Correctly identify normal behavior and assign reward as the first positive reward value.

[0079] Correctly identify the malicious code escape behavior and assign the reward as the second positive reward value.

[0080] Correctly identify the malicious code escape behavior and output effective response actions, and assign reward as the third positive reward value.

[0081] Misjudging normal behavior as malicious code escape behavior, the reward is assigned the first negative reward value.

[0082] Misjudged malicious code escape behavior as normal behavior, and assigned a reward of the second negative reward value.

[0083] The third positive reward value is greater than the second positive reward value, the second positive reward value is greater than the first positive reward value, and the first negative reward value is greater than the second negative reward value.

[0084] In this embodiment, the reward mechanism is formulated based on the accuracy of the detection results and the execution results of the response strategy for malicious code escape, so that the reinforcement learning network adjusts the strategy of selecting actions according to the reward value during training, and is more inclined to select the correct action. It can be understood that: the accuracy of the state next_state detection result of the next time step and the execution result of the response strategy for malicious code escape can be used to determine whether it is correct. For example, in a certain state, after identifying the malicious code escape behavior and outputting an effective response action, it is determined through the process of obtaining next_state that the operation state of the network system has been improved, which means that the malicious code escape behavior is correctly identified and an effective response action is output. At this time, the highest third positive reward value is given to reward, so that in actual applications, when encountering a state similar to a certain state, the malicious code escape behavior will be identified first and the corresponding response action will be output.

[0085] In some embodiments, the malicious code escape detection model training process is as follows.

[0086] Construct the reinforcement learning network structure, loss function and experience replay buffer. Specifically, initialize the DQN network: determine the structure of the deep Q network, including the number and type of input layers, hidden layers, and output layers, as well as the connection method between the layers. In the scheme, the input layer receives the feature vector after feature fusion, and the output layer gives the Q value corresponding to different actions (such as "normal", "there is potential escape behavior and corresponding response strategy"), and the Q value represents the benefit or probability of selecting an action in the current state. At the same time, to train the DQN network, use Adam as the optimizer and the mean square error (MSE) as the loss function ,in is the reward corresponding to the current behavior, is a discount factor used to balance immediate rewards with future rewards, The action to be taken at the next moment, The state after executing the current action. is the current state, is the action taken at the current moment, w is the parameter of the deep Q network, y= is the target Q value, i.e. the label, represents a deep Q network, is the prediction value of the current deep Q network. The experience replay buffer is used to store the experience tuples (state, action, reward, next_state, done) of the interaction with the environment, where done indicates whether the training is completed (during training, the reinforcement learning network is trained with the training set. After a certain number of trainings, it is tested with the test set. If the test finds that the prediction accuracy of the reinforcement learning network reaches the predetermined value, such as 95%, it is not necessary to complete the preset number of trainings, and done is set to 1 to terminate the training). During the training process, experience tuples are randomly sampled from the experience replay buffer as sample data tuples for training.

[0087] Get the current state s of the network system at the current time step t , according to the current state s t Select and perform an action t , based on executing action a t The execution result gets the current reward value r t and the state s at the next time step t+1 , will (s t , a t , r t , s t+1 ) is stored in the experience replay buffer;

[0088] Randomly sample sample data tuples from the experience replay buffer and input them into the reinforcement learning network to update the weights of the target Q network. Specifically, randomly sample small batches (16 to 64 experience tuples, for example, 32) of experience tuples for training to reduce the correlation between samples and improve training efficiency and stability. Small batch samples help alleviate the catastrophic forgetting problem, that is, the network will not quickly forget old data while remembering new data. When using experience replay, small batch samples can be extracted from the experience stored at different time points in the buffer, which increases the diversity of samples. Small batch training can reduce the time of a single calculation, allowing the model to update parameters faster.

[0089] The training is completed when the preset number of training times is reached or the loss function converges.

[0090] In some embodiments, the step S110 specifically includes the following steps.

[0091] Step 1: Obtain different types of source detection data related to malicious code escape detection in the network system. Specifically, for different types of data, source detection data is data directly captured from different types of data.

[0092] Step 2: Preprocess the different types of source detection data, including normalization, format conversion, noise removal, redundant data removal, time synchronization and event alignment, and determine the preprocessed data as the target detection data. For example, network traffic data records TCP connection information within a certain period of time, including timestamp, source IP, destination IP, port number, number of bytes transmitted, etc.

[0093] Normalize: Normalize the byte count by dividing it by the maximum number of bytes observed during the time period.

[0094] Format conversion: If the raw data is in binary or software-specific log format, parse and convert it into formatted data such as CSV or database table format.

[0095] Noise removal: Use statistical methods to identify and remove outliers, such as removing anomalous data points that indicate a sudden spike in the number of bytes transferred.

[0096] Remove redundant data: Delete duplicate TCP connection records and keep only the first occurrence record.

[0097] Time synchronization: Convert the timestamps of all data points to milliseconds since 00:00 of the current day.

[0098] Event alignment: If there are events in the system log that are related to TCP connection establishment in the network traffic, use the timestamps in the system log to align the network traffic data to ensure that the two are arranged on the same timeline.

[0099] In this embodiment, target detection data is obtained by preprocessing source detection data, thereby ensuring the availability and time consistency of the target detection data, thereby making the final malicious code escape detection result more accurate.

[0100] The malicious code escape detection device provided by the present invention is described below. The malicious code escape detection device described below and the malicious code escape detection method described above can be referenced to each other.

[0101] The malicious code escape detection device of the embodiment of the present invention is as follows: Figure 2 As shown, it includes the following functional units.

[0102] The data acquisition unit 210 is used to acquire different types of target detection data related to malicious code escape detection in the network system.

[0103] The feature extraction unit 220 is used to extract feature vectors of the different types of target detection data respectively.

[0104] The feature fusion unit 230 is used to fuse different feature vectors to obtain a fused multi-modal feature vector.

[0105] The model execution unit 240 is used to input the multimodal feature vector into a malicious code escape detection model constructed based on a reinforcement learning network, obtain the detection result output by the malicious code escape detection model, and output a corresponding response action when the detection result is that a malicious code escape exists.

[0106] Among them, the malicious code escape detection model is trained based on the sample data tuple (state, action, reward, next_state), and the training is completed when the preset training times are reached or the loss function converges. The state state is represented by the multimodal feature vector, the action action is one of the actions in the action set, and the action set includes the corresponding response action of the detection result. The reward value reward is determined based on the accuracy of the detection result and the execution result of the response action. The next state next_state represents the multimodal feature vector corresponding to the network system after the action is executed.

[0107] In some embodiments, the different types of target detection data include: network traffic data, system log data, memory snapshot data, and file system change data.

[0108] The feature extraction unit 220 is specifically used to: extract the traffic feature vector of the network traffic data based on a bidirectional long short-term memory network; extract the log feature vector of the system log data based on an attention model; extract the memory snapshot feature vector of the memory snapshot data based on a first graph convolutional neural network; and extract the file system change feature vector of the file system change data based on a second graph convolutional neural network.

[0109] In some embodiments, the feature fusion unit 230 is specifically used to input the traffic feature vector, log feature vector, memory snapshot feature vector and file system change feature vector into a multi-layer perceptron model to obtain the multi-modal feature vector fused and output by the multi-layer perceptron model.

[0110] In some embodiments, the reward value reward is determined based on the accuracy of the detection result and the result of executing the response action, including: correctly identifying the normal behavior and assigning the reward as a first positive reward value.

[0111] Correctly identify the malicious code escape behavior and assign the reward as the second positive reward value.

[0112] Correctly identify the malicious code escape behavior and output effective response actions, and assign reward as the third positive reward value.

[0113] Misjudging normal behavior as malicious code escape behavior, the reward is assigned the first negative reward value.

[0114] Misjudged malicious code escape behavior as normal behavior, and assigned a reward of the second negative reward value.

[0115] The third positive reward value is greater than the second positive reward value, the second positive reward value is greater than the first positive reward value, and the first negative reward value is greater than the second negative reward value.

[0116] In some embodiments, the malicious code escape detection model training process is as follows.

[0117] Construct reinforcement learning network structure, loss function and experience replay buffer.

[0118] Get the current state s of the network system at the current time step t , according to the current state s t Select and perform an action t , based on executing action a t The execution result gets the current reward value r t and the state s at the next time step t+1 , will (s t , a t , r t , s t+1 ) is stored in the experience replay buffer.

[0119] A sample data tuple is randomly sampled from the experience replay buffer and input into the reinforcement learning network to update the weight of the target Q network.

[0120] The training is completed when the preset number of training times is reached or the loss function converges.

[0121] In some embodiments, the data acquisition unit 210 includes the following functional units.

[0122] The source detection data acquisition unit is used to acquire different types of source detection data related to malicious code escape detection in the network system.

[0123] A data preprocessing unit is used to preprocess the different types of source detection data, wherein the preprocessing includes normalization, format conversion, noise removal, redundant data removal, time synchronization and event alignment, and determines that the preprocessed data is the target detection data.

[0124] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the malicious code escape detection method, which includes the following steps.

[0125] Obtain different types of target detection data related to malicious code evasion detection in network systems.

[0126] The feature vectors of the different types of target detection data are extracted respectively.

[0127] Different feature vectors are fused to obtain a fused multimodal feature vector.

[0128] The multimodal feature vector is input into a malicious code escape detection model constructed based on a reinforcement learning network to obtain a detection result output by the malicious code escape detection model, and when the detection result is that a malicious code escape exists, a corresponding response action is output.

[0129] Among them, the malicious code escape detection model is trained based on the sample data tuple (state, action, reward, next_state), and the training is completed when the preset training times are reached or the loss function converges. The state state is represented by the multimodal feature vector, the action action is one of the actions in the action set, and the action set includes the corresponding response action of the detection result. The reward value reward is determined based on the accuracy of the detection result and the execution result of the response action. The next state next_state represents the multimodal feature vector corresponding to the network system after the action is executed.

[0130] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0131] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the malicious code escape detection method provided by the above methods. The method includes the following steps.

[0132] Obtain different types of target detection data related to malicious code evasion detection in network systems.

[0133] The feature vectors of the different types of target detection data are extracted respectively.

[0134] Different feature vectors are fused to obtain a fused multimodal feature vector.

[0135] The multimodal feature vector is input into a malicious code escape detection model constructed based on a reinforcement learning network to obtain a detection result output by the malicious code escape detection model, and when the detection result is that a malicious code escape exists, a corresponding response action is output.

[0136] Among them, the malicious code escape detection model is trained based on the sample data tuple (state, action, reward, next_state), and the training is completed when the preset training times are reached or the loss function converges. The state state is represented by the multimodal feature vector, the action action is one of the actions in the action set, and the action set includes the corresponding response action of the detection result. The reward value reward is determined based on the accuracy of the detection result and the execution result of the response action. The next state next_state represents the multimodal feature vector corresponding to the network system after the action is executed.

[0137] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for detecting malicious code escapes provided by the above methods is implemented, and the method includes the following steps.

[0138] Obtain different types of target detection data related to malicious code evasion detection in network systems.

[0139] The feature vectors of the different types of target detection data are extracted respectively.

[0140] Different feature vectors are fused to obtain a fused multimodal feature vector.

[0141] The multimodal feature vector is input into a malicious code escape detection model constructed based on a reinforcement learning network to obtain a detection result output by the malicious code escape detection model, and when the detection result is that a malicious code escape exists, a corresponding response action is output.

[0142] Among them, the malicious code escape detection model is trained based on the sample data tuple (state, action, reward, next_state), and the training is completed when the preset training times are reached or the loss function converges. The state state is represented by the multimodal feature vector, the action action is one of the actions in the action set, and the action set includes the corresponding response action of the detection result. The reward value reward is determined based on the accuracy of the detection result and the execution result of the response action. The next state next_state represents the multimodal feature vector corresponding to the network system after the action is executed.

[0143] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A malicious code escape detection method, characterized in that: include: Acquire different types of target detection data related to malicious code escape detection in the network system, wherein the different types of target detection data include: network traffic data, system log data, memory snapshot data, and file system change data; Extracting feature vectors of the different types of target detection data respectively; Fuse different feature vectors to obtain a fused multi-modal feature vector; Inputting the multimodal feature vector into a malicious code escape detection model constructed based on a reinforcement learning network, obtaining a detection result output by the malicious code escape detection model, and outputting a corresponding response action when the detection result indicates that a malicious code escape exists; Among them, the malicious code escape detection model is trained based on sample data tuples, and the training is completed when the preset number of training times is reached or the loss function converges. The sample data tuple includes: state state, action action, reward value reward and the next state next_state of the state state. The state state is represented by the multimodal feature vector. The action action is one of the actions in the action set. The action set includes the corresponding response actions of the detection results. The reward value reward is determined based on the accuracy of the detection result and the execution result of the response action. The next state next_state represents the multimodal feature vector corresponding to the network system after executing the action.

2. The malicious code escape detection method according to claim 1, characterized in that: The extracting the feature vectors of the different types of target detection data respectively includes: Extracting a traffic feature vector of the network traffic data based on a bidirectional long short-term memory network; Extracting a log feature vector of the system log data based on an attention model; Extracting a memory snapshot feature vector of the memory snapshot data based on a first graph convolutional neural network; A file system change feature vector of the file system change data is extracted based on a second graph convolutional neural network.

3. The malicious code escape detection method according to claim 2, characterized in that: Different feature vectors are fused to obtain a fused multimodal feature vector, including: The traffic feature vector, the log feature vector, the memory snapshot feature vector and the file system change feature vector are input into a multi-layer perceptron model to obtain the multi-modal feature vector fused and output by the multi-layer perceptron model.

4. The malicious code escape detection method according to claim 1, characterized in that: The reward value reward is determined based on the accuracy of the detection result and the result of the response action execution, including: Correctly identify normal behavior and assign the reward as the first positive reward value; Correctly identify the malicious code escape behavior and assign the reward as the second positive reward value; Correctly identify the malicious code escape behavior and output effective response actions, and assign reward as the third positive reward value; Misjudging normal behavior as malicious code escape behavior, the reward is assigned the first negative reward value; Misjudging the malicious code escape behavior as normal behavior, the reward is assigned the second negative reward value; The third positive reward value is greater than the second positive reward value, the second positive reward value is greater than the first positive reward value, and the first negative reward value is greater than the second negative reward value.

5. The malicious code escape detection method according to claim 1, characterized in that: The training process of the malicious code escape detection model is as follows: Construct reinforcement learning network structure, loss function and experience replay buffer; Get the current state s of the network system at the current time step t , according to the current state s t Select and perform an action t , based on executing action a t The execution result gets the current reward value r t and the state s at the next time step t+1 , will (s t , a t , r t , s t+1 ) is stored in the experience replay buffer; Randomly sampling sample data tuples from the experience replay buffer and inputting them into the reinforcement learning network to update the weights of the target Q network; The training is completed when the preset number of training times is reached or the loss function converges.

6. The malicious code escape detection method according to any one of claims 1 to 5, characterized in that: Obtain different types of target detection data related to malicious code evasion detection in network systems, including: Obtain different types of source detection data related to malicious code escape detection in network systems; The different types of source detection data are preprocessed, and the preprocessing includes: normalization, format conversion, noise removal, redundant data removal, time synchronization and event alignment, and the preprocessed data is determined to be the target detection data.

7. A malicious code escape detection device, characterized in that: include: A data acquisition unit, used to acquire different types of target detection data related to malicious code escape detection in a network system, wherein the different types of target detection data include: network flow data, system log data, memory snapshot data, and file system change data; A feature extraction unit, used to extract feature vectors of the different types of target detection data respectively; A feature fusion unit is used to fuse different feature vectors to obtain a fused multi-modal feature vector; A model execution unit, configured to input the multimodal feature vector into a malicious code escape detection model constructed based on a reinforcement learning network, obtain a detection result output by the malicious code escape detection model, and output a corresponding response action when the detection result indicates that a malicious code escape exists; Among them, the malicious code escape detection model is trained based on sample data tuples, and the training is completed when the preset number of training times is reached or the loss function converges. The sample data tuple includes: state state, action action, reward value reward and the next state next_state of the state state. The state state is represented by the multimodal feature vector. The action action is one of the actions in the action set. The action set includes the corresponding response actions of the detection results. The reward value reward is determined based on the accuracy of the detection result and the execution result of the response action. The next state next_state represents the multimodal feature vector corresponding to the network system after executing the action.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the malicious code escape detection method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the malicious code escape detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the malicious code escape detection method according to any one of claims 1 to 6 is implemented.

Images