A malware detection optimization method and system based on a multi-layer graph neural network

Abstract

This invention discloses an optimized method and system for malware detection based on multilayer graph neural networks. The method first performs static analysis on the software's binary file to extract three layers of static features. Next, the software is hierarchically represented as the extracted three layers of static features, and a program-level representation of these features is learned through a graph neural network. Finally, the program-level representation is input into a multilayer perceptron, and a single binary classification score is output to obtain the predicted probability, thus completing malware detection. This invention achieves unified modeling at the instruction, function, and program levels, significantly improving the accuracy of static malware detection.

Description

Technical Field

[0001] This invention belongs to the field of malware detection, specifically relating to an optimized method and system for static analysis and detection of Windows malware based on instruction sequence graph neural networks. Background Technology

[0002] With the rapid development of information technology and network applications, the number and complexity of malware continue to grow, posing a serious threat to the cybersecurity of individual users and enterprises. Traditional malware detection methods are mainly divided into two categories: signature-based detection methods and behavior analysis-based detection methods.

[0003] Signature-based detection identifies samples by comparing unique signatures or patterns of malware, offering advantages such as high detection speed and simple implementation. However, it often fails against polymorphism, variants, and obfuscation techniques. Behavioral analysis-based methods identify malice by running programs and monitoring their system calls, file operations, network connections, and other behavioral characteristics. While capable of discovering unknown malicious samples, these methods suffer from complex operating environments, high analysis costs, and susceptibility to anti-virtualization and anti-debugging techniques employed by malicious code. In recent years, with the development of machine learning and deep learning technologies, researchers have increasingly attempted to apply them to malware detection. For example, static feature extraction methods based on byte sequences or function call sequences can utilize classifiers to identify samples.

[0004] However, malware detection technologies based on machine learning and deep learning face two serious challenges: (1) the methods rely on surface features and cannot fully capture rich instruction semantics and program structure; (2) the feature representations used are often fragile and easily affected by adversarial perturbations and evasion attacks. Summary of the Invention

[0005] To address the aforementioned challenges, this invention proposes an optimized method and system for malware detection based on multi-layer graph neural networks. Specifically, it proposes a static analysis and detection method for Windows malware based on instruction sequence graph neural networks. This method, by introducing instruction sequence graphs and a hierarchical program representation mechanism, enables unified modeling of executable files at multiple abstraction levels, including instruction, function, and program levels. This improves model robustness and detection rate with low false positive rates.

[0006] In one aspect, this invention provides an optimized method for malware detection based on multi-layer graph neural networks, comprising the following steps:

[0007] S1. Perform static analysis on the software's binary file to extract three layers of static features, including the following process:

[0008] S1.1 Construct the function call graph structure of the software by generating call instructions from the software disassembly.

[0009] S1.2 Construct the control flow graph structure of the software's native functions by analyzing the jump instructions generated from the software disassembly.

[0010] S1.3 Construct the instruction sequence diagram of the basic blocks according to the instruction order of the basic blocks in the control flow diagram.

[0011] S2. Represent the software hierarchically as the three static features extracted in S1, and learn the program-level representation of the features through a graph neural network, including the following process:

[0012] S2.1 Encode the feature of instruction nodes in the instruction sequence graph, learn the representation through graph neural network, and concatenate the block size prior as the initial representation of the basic block in the next layer of control flow graph.

[0013] S2.2. The initialization representation is used as the node input of the control flow graph of the function. After being aggregated by a multi-layer graph neural network and pooled across the entire graph, the function-level representation is obtained and used as the initialization representation of the local function in the function call graph of the next layer.

[0014] S2.3 In the function call graph, nodes are initialized by embedding the names, properties, and parameters of local functions and Windows system APIs. After multi-layer graph neural network aggregation and full graph pooling, a program-level representation is obtained.

[0015] S3. By inputting the program-level representation into the multilayer perceptron and outputting a single binary classification score to obtain the predicted probability, malware detection is completed.

[0016] As a preferred approach, S1.1 uses the Radare2 disassembler to disassemble the malware and identify call instructions, performs reachability analysis starting from the program's main function, and thus constructs the program's function call graph.

[0017] As a preferred approach, S1.2 extracts each local function from the function call graph structure generated by S1.1, identifies the jump instructions, where each basic block is a node and the jump relationship is an edge, and constructs the corresponding control flow graph structure based on the jump instructions.

[0018] As a preferred approach, S1.3 extracts the instruction sequence of each basic block from the control flow graph structure generated in S1.2, establishes the data dependency and control dependency between instructions based on register write / read relationships and memory access constraints, and constructs the instruction sequence graph of the basic block to depict the data flow and execution order between instructions.

[0019] As a preferred approach, S2.1 first establishes an operator vocabulary and an operand vocabulary, maps a single instruction node to a learning vector of operator concatenation with operands, uses it as the original input of the graph neural network for learning, and concatenates the logarithmic scale of the basic block instruction size as the initial feature representation of the basic block node in the next layer of the control flow graph.

[0020] As a preferred approach, S2.2 uses the initialization representation of the basic blocks obtained in S2.1 to construct function control flow graph nodes. After aggregation by a multi-layer graph neural network and full graph pooling, a function-level representation is generated, which serves as the initial input for local functions in the function call graph.

[0021] As a preferred approach, S2.3 first establishes a Windows system API function vocabulary in the function call graph, mapping Windows system API functions to learnable vectors aligned with the initial input of local functions; simultaneously, it establishes a parameter vocabulary, mapping function parameters to learnable vectors. Then, it concatenates the API vectors (if the function is a system API) or the initial input vector of the local function (if the function is a local function), the parameter vectors, and the function's in-degree, out-degree, stack size, cyclomatic complexity, and recursion flag to form a node representation of the function call graph. This node representation is input to a multi-layer graph neural network, and after aggregation and full graph pooling, a program-level representation is obtained.

[0022] As a preferred approach, S3 inputs the program-level representation into a multilayer perceptron (MLP) and trains it using a binary cross-entropy loss function, ultimately outputting a single binary classification score to calculate the predicted probability, thereby enabling the detection of malware.

[0023] In another aspect, the present invention provides a malware detection optimization system based on a multi-layer graph neural network, comprising the following modules:

[0024] The static feature extraction module is used to perform static analysis on the software's binary files to extract three layers of static features.

[0025] The program-level representation module is used to represent the software hierarchically as extracted three-layer static features, and learns the program-level representation of the features through a graph neural network.

[0026] The detection result output module is used to obtain the predicted probability by inputting the program-level representation into the multilayer perceptron and outputting the binary classification score, thereby completing the malware detection.

[0027] The beneficial effects of this invention are:

[0028] This invention achieves unified modeling at the instruction, function, and program levels by introducing instruction sequence diagrams and hierarchical program representation mechanisms. This enables the model to capture rich instruction semantics and program structure, thereby significantly improving the accuracy of static malware detection. Simultaneously, the use of multi-hot encoded instruction representation and cross-layer semantic alignment mechanisms enhances the model's robustness against adversarial perturbations and variant malicious samples, effectively reducing false positive rates. Furthermore, this invention employs a static analysis method, extracting features without executing suspicious programs, reducing resource consumption and runtime environment dependence, making it suitable for large-scale malware detection and enterprise endpoint security protection. Attached Figure Description

[0029] Figure 1 This is an example of an instruction sequence diagram;

[0030] Figure 2 This is a framework diagram of a malware model. Detailed Implementation

[0031] The technical solutions of the present invention will now be clearly and completely described in conjunction with the accompanying drawings of the embodiments of the present invention. The described embodiments are merely a part of the embodiments of the present invention and cannot encompass all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. In one aspect, the present invention provides a malware detection optimization method based on a multi-layer graph neural network, the specific process of which is as follows:

[0032] S1 performs static analysis on the software's binary file to extract three layers of static features.

[0033] S1.1 Constructing the Software's Function Call Graph Structure: Static analysis of the software was performed using the Radare2 disassembler. First, the program entry point was identified. Then, starting from the entry point, the program was disassembled at the function level to extract the call instructions. Based on this, reachability analysis was performed by tracing the target of each function call, recursively identifying all called native functions and system API functions. Finally, these function nodes and their call relationships were organized into the program's function call graph, providing a foundation for subsequent feature extraction and graph neural network modeling.

[0034] S1.2 Constructing the Control Flow Graph Structure of Software Native Functions: Extract each native function node from the program function call graph generated in step S1.1, and perform static disassembly on each native function to identify jump instructions, including conditional jumps and unconditional jumps. Subsequently, based on the basic block control flow relationships indicated by these jump instructions, construct the corresponding control flow graph (CFG), where each basic block is a node and jump relationships are edges, accurately describing the execution path and branch structure inside the function, providing structured input for subsequent function-level representation learning based on control flow.

[0035] S1.3 Instruction sequence diagram for constructing basic blocks: The following is in conjunction with the appendix to the instruction manual. Figure 1 The method for constructing instruction sequence diagrams according to the present invention will be described in detail below. It should be understood that the accompanying drawings are merely illustrative examples intended to help understand the technical solution of the present invention, and are not intended to limit the scope of protection.

[0036] S1.3.1 Basic Block Node Initialization: For a basic block b, its instruction sequence is denoted as follows: To ensure graph connectivity and semantic integrity, two special nodes are first introduced into this basic block: the start node. With the termination node Then each instruction Corresponding to a node Thus the node set is

[0037] .

[0038] S1.3.2, Edge Type Definition and Register Alias ​​Equivalence Class: The edge set of the instruction sequence graph is divided into two main categories: data-dependent edges. , representing a basic block Data between instructions Dependency relationships and effect dependency edges Data dependencies are used to express precise value flows (e.g., read-after-write RAW for registers), while effect dependencies are used to express execution order, visibility, and approximate value flows (e.g., read-after-write WAR, write-after-write WAW, memory barriers, and control barriers). Register aliases use equivalence class functions. express, Give registers The alias equivalence class to which it belongs, for example, in the x86 architecture, register EAX is a 32-bit general-purpose register used to store integer values ​​or perform arithmetic and logical operations. AX is the lower 16 bits of EAX, i.e., EAX[15:0], used to access smaller operands. AX can be further divided into two 8-bit registers: AH represents the high 8 bits (AX[15:8]), and AL represents the low 8 bits (AX[7:0]). Therefore, For any instruction , This represents the set of registers to which it is written (defined). This indicates the set of registers it reads (uses).

[0039] S1.3.3, State Variable Initialization: Before constructing the instruction sequence diagram, state variables used to record recently accessed variables are set: Register-side states include three types of maps: Record equivalence class The index of the most recent "write" instruction; Record the index of the most recent "read" command; Record the index of the most recent "write" command (and) Consistent, but convenient for directly forming WAW). Memory-side states include two types of mappings: Record the "known address key" "The most recently written instruction index, Record the "known address key" "The index of the most recently read instruction. Two scalars are retained simultaneously." and Each index records the instruction index of the most recent write and read operation to the "unknown address," with an initial value of None. This state is used to find the "most recent relevant access" at any time during a linear scan, so that edges can be built immediately.

[0040] S1.3.4, Data Dependency Edge Construction For each instruction First, construct data dependency edges at the register level. Each register read If a subscript exists Then add a line The data dependency edge represents "the most recent data dependency edge". The precise value stream from the definition of "to" to the current read. Its formal expression is:

[0041]

[0042] That is, in and There is no redefinition of the same equivalence class between them, ensuring a RAW relationship between the "most recently defined" and the current read. Then, dependencies for memory reads are built within the same line range: if... If true, then calculate .when and When, add The data dependency edge represents the most recent write to the current read RAW at the "same precise address"; if If it cannot be found in the mapping, it degenerates into a conservative rule. Then add The effect depends on the edge, and the type is denoted as , is used to express conservative visibility constraints in the case of unknown addresses.

[0043] S1.3.5, Effect Dependency Edge Construction: Construct two types of effect dependency edges at the register level: WAR and WAW. These two types of edges do not represent exact value flow, but rather express coverage and order constraints. For each register written to by the current instruction... If it exists Then add If the WAR edge exists Then add The WAW edges. Their set form can be written as...

[0044] These two types of edges are merged , This indicates the order constraint between processing memory writes and previous memory accesses. If true, calculate first. .when This is the case of a precise address: if Then add of edge; if Then add of Side. When When the address is unknown: If Then add of If Then add of Edge. The above rules ensure that when static address differentiation is not possible while maintaining memory access order and visibility is necessary, conservative constraints are used to avoid accidentally deleting real dependencies. Barriers and call effects are established if... If true, then from all previous nodes ( )Towards Add to Effect edges of this type are used to express the barrier effect of a jump on the rearrangement of subsequent instructions within the same basic block. If If true, then first check each recorded exact address key. Add to of Side, at the same time if Then add of This indicates that the call is a memory barrier, requiring all previously pending writes to be seen. Subsequently, on... Each register in Check the most recent write Add if it exists. The WAW effect edge explicitly indicates the destructive (overwrite) effect of the call on these registers, thereby blocking the inference of erroneous value streams across calls.

[0045] S1.3.6, State variable update: Update each register currently being written to. ,set up and For each register currently read ,set up If the current operation is a memory read, then retrieve... ,when season and will , This represents an unknown value; if the current operation is a memory write, then it is taken as... ,when season and will With the above updates, the algorithm can obtain the position of the "most recently visited" at any time during a single linear traversal, thus enabling online graph construction.

[0046] S1.3.7 Post-processing of the graph and generation of the final instruction sequence graph: First, delete all self-loops; then perform transitive simplification for each edge type, i.e., when there are paths of the same type... Delete directly on the side To reduce redundancy; if the same ordered node pairs If both data dependency and effect dependency exist, retain the data dependency and remove the effect dependency to avoid duplicate expressions; to eliminate isolated instruction nodes, nodes with an in-degree of zero are... Add effect edges to nodes with an out-degree of zero. Add effect edges; remove known pseudo-dependencies, such as flag stack pseudo-dependencies caused by pushfd / pushfq or false dependencies caused by fixed stack reads based on EBP; finally, when more sparse visualization is needed, a global transitive simplification can be performed without distinguishing edge types, solely for graphical display. This yields the instruction sequence diagram described in this invention. .

[0047] S2 represents the software as three layers of static features extracted in the above steps, and learns the program-level representation of the features through a graph neural network.

[0048] This invention employs a bottom-up program graph hierarchy as an inductive bias, modeling the program at three structural scales: the Instruction-Order Graph (IOG) within basic blocks, the Control Flow Graph (CFG) within functions, and the Program-Level Function Call Graph (FCG). (See attached specification) Figure 2 This is an overall framework diagram of the malware detection model. This invention, through explicit cross-layer mapping alignment, enables the lossless transfer of fine-grained instruction semantics and local control structures to the global program representation, ultimately inputting the prediction header and outputting the malware probability. This section first presents a unified message passing and graph readout format, and then explains the three-layer encoding and prediction layers in turn.

[0049] To avoid redefining across different layers, this invention employs the same GraphSAGE-based message passing and max-pooling reads across all layers. Let the graph be... , For the image The set of nodes, Let be the set of edges. For the first After layer message passing, the node The vector represents , where For the first Layer channel dimension, Let be the initial feature vector of this node. For nodes The set of adjacent nodes. The unified form for updating nodes at a single level is:

[0050]

[0051] in It is an element-wise nonlinear activation function. and This is the learnable parameter matrix for this layer. The number of adjacent nodes; for For isolated nodes, their neighbor entries are treated as zero vectors. The entire graph is read out using channel-wise max pooling.

[0052]

[0053] in Encode the message passing layers of the stack for this graph. Take the maximum value of each element according to the channel dimension.

[0054] S2.1 Instruction Node Feature Encoding and Representation Initialization Method Based on Block Size Priority:

[0055] At the basic block level, for each basic block First, construct its IOG, denoted as .in This is the set of instruction nodes within the basic block. Indicates the number of instructions. This is a set of directed edges established according to instruction order and local dependencies. For each instruction... Initialize feature vectors This vector can be obtained by linear mapping of static attributes such as instruction mnemonics, operand types, addressing modes, immediate values, and memory access modes through embedding and concatenation. This is the initial dimension. Then in... Stacking Layered GraphSAGE message passing yields the final representation of each instruction. Reading from the nodes within the block yields the block-level semantic vector.

[0056]

[0057] in The dimension is defined after reading from IOG. To explicitly inject a scale prior and suppress numerical offsets caused by the long tail of instruction counts, the scale scalar of this basic block is calculated. The semantic vector and the scale scalar are concatenated to form the initial representation of the CFG node. Use cross-layer mapping block of Injected into the corresponding basic block node in the CFG of its own function as the initial feature of that CFG node.

[0058] S2.2 A method for generating function-level representations and initializing function call graph nodes based on control flow graph aggregation:

[0059] At the function level, given a certain local function Control flow graph .in It is the set of basic block nodes within the function. This is the set of directed edges for control flow between blocks. For each basic block node... Its initial features are set as ,That To map block-level concatenation features to the initial dimension of CFG Learnable linear transformations. Subsequently in Stacking The GraphSAGE layer yields the final representation of all basic block nodes. Then, perform max pooling to read out the function-level representation.

[0060]

[0061] in This serves as the function-level representation dimension, acting as a semantic anchor for cross-layer alignment to the program-level FCG. For system API function nodes, due to their unique function names and stable functionality, a learnable embedding vector is maintained for each API name. , in relation to local functions They are in the same semantic space and facilitate subsequent fusion.

[0062] S2.3 A program-level representation generation method based on function call graph aggregation:

[0063] At the program level, this method constructs a function call graph. ,in For a collection of local function nodes, For the system API node set, For the set of edges called; for local function nodes Its initial vector is directly set as the readout vector of the corresponding function layer. ,in For nodes The corresponding local functions This is the representation of the CFG of the function read out after passing it through a graph neural network; for system API nodes... According to its name Obtaining same-dimensional embeddings through a learnable vocabulary To ensure semantic comparability with local function representations, static properties are then appended to the initial vector of each FCG node to enhance the representation and improve robustness to renaming or minor structural perturbations. These properties include normalized cyclomatic complexity. recursion pointer Stack frame size System API Indicator In-degree and out-degree and parameter type embedding (where parameter type sequence) By type embedding table Map and aggregate in an average manner The parameter types of local functions are recovered from the calling convention and disassembly information such as register / stack parameter passing (to maintain type abstraction consistent with system API nodes), and the resulting concatenated vector is used as the input to the graph neural network on the FCG. ,exist Stacking Layer GraphSAGE uses mean aggregation for message passing and employs an update-based approach. ,in For nonlinear activation functions, For learnable weights, For nodes The set of neighbors and Using its cardinality, the final representation of all nodes. Perform channel-wise max pooling to obtain the program-level vector. .

[0064] The S3 program represents the input multilayer perceptron, outputs a binary classification score to obtain the predicted probability, and completes malware detection.

[0065] In the prediction layer, the program-level representation z_{G^e} is input into a shallow multilayer perceptron (MLP) to obtain a binary logit value, which is then converted into a malicious probability using the sigmoid function.

[0066]

[0067] in For program samples The predicted probability of malicious activity is calculated. During the training phase, the Binary Cross-Entropy (BCE) loss function is used to measure the difference between the predicted value and the true label, and its form is:

[0068]

[0069] in This represents the true label of the sample. This represents the predicted probability of the model. By minimizing this loss function, the model can accurately distinguish between malicious and benign samples.

[0070] In another aspect, the present invention provides a malware detection optimization system based on a multi-layer graph neural network, comprising the following modules:

[0071] The static feature extraction module is used to perform static analysis on the software's binary files to extract three layers of static features.

[0072] The program-level representation module is used to represent the software hierarchically as extracted three-layer static features, and learns the program-level representation of the features through a graph neural network.

[0073] The detection result output module is used to obtain the predicted probability by inputting the program-level representation into the multilayer perceptron and outputting the binary classification score, thereby completing the malware detection.

[0074] Method evaluation:

[0075] To verify the effectiveness of the malware detection method described in this invention, the detection performance and robustness of the detector were experimentally verified. For malware detectors, maintaining a high detection rate with a low false positive rate and exhibiting strong robustness against adversarial attacks are important indicators for evaluating the performance of the detection system. Therefore, this embodiment evaluates the invention from two aspects: detection rate and robustness.

[0076] For performance comparison, this embodiment selects two existing Windows malware detection models based on graph neural networks (GNNs) as references: the MalGraph detector and the MAGIC detector. Experimental results show that, under the same conditions, the detector of this invention outperforms the aforementioned existing models in both detection accuracy and robustness against adversarial examples.

[0077] Experimental environment:

[0078] The detector described in this invention is implemented in a Linux operating system environment, and a multi-level program graph is constructed based on instruction sequences for static analysis and detection. The experimental operating environment is shown in Table 1.

[0079] Table 1 Experimental Environment

[0080]

[0081] The core parameter settings of the detection model of this invention are shown in Table 2. The model as a whole adopts a hierarchical graph neural network structure, including three layers of representation: instruction directed graph (IOG), control flow graph (CFG), and function call graph (FCG), and combines semantic embedding of nodes and edges to improve detection accuracy.

[0082] Table 2 Model Parameter Settings

[0083]

[0084] Experimental evaluation

[0085] Detection rate assessment with low false alarm rate:

[0086] To verify the detection performance of the detector of this invention under low false alarm rate conditions, this embodiment conducted a comparative experiment on static malware detection based on graph neural networks (GNNs). This experiment selected two existing graph neural network detection models, MalGraph and MAGIC, as baseline models, and used the instruction sequence-based graph neural network Windows malware detection method proposed in this invention (hereinafter referred to as "I2P-Aligned-GNN") as the test object. All three detectors performed static analysis and classification based on program graph structures. The dataset used in the experiment came from PE file samples of the Windows platform, including both malware and benign software. The samples were divided into training, validation, and test sets in an 8:1:1 ratio, and the dataset information is shown in Table 3.

[0087] Table 3 Test Set Information

[0088]

[0089] To objectively evaluate detection performance, the experiment used metrics such as the area under the receiver operating characteristic curve (AUC), the true positive rate (TPR) at a fixed false alarm rate, and the balanced accuracy (bACC)**. Specifically, TPR and bACC were measured at false alarm rates of 1% and 0.1%, respectively, to assess the stability and reliability of the model under high-safety-requirement environments. The experimental results of the three detectors on the same test set are shown in Table 4. The I2P-Aligned-GNN detector proposed in this invention exhibits significantly superior detection capabilities under low false alarm rate conditions. When the false alarm rate is 1%, its true positive rate (TPR) reaches 91.93%, which is approximately 6.1% and 6.2% higher than MalGraph and MAGIC, respectively. Under the more stringent false alarm rate condition of 0.1%, the TPR remains at 81.64%, significantly higher than MalGraph (58.39%) and MAGIC (0.00%).

[0090] Meanwhile, this invention also achieves a significant improvement in balanced accuracy (bACC), indicating that the model effectively reduces the false detection probability of benign samples while maintaining a high detection rate. This result fully demonstrates that the instruction sequence-based graph neural network detection method described in this invention has higher detection reliability and stability under low false alarm constraints, and can achieve stronger defense effects in practical applications.

[0091] Table 4. Detection rate of the model at low false alarm rate

[0092]

[0093] Robustness assessment against adversarial attacks:

[0094] To further verify the robustness of the detector of this invention against adversarial attacks, this embodiment conducted an anti-adversarial perturbation experiment on the I2P-Aligned-GNN model and compared it with the MalGraph and MAGIC models. In this embodiment, the experiment used the MalGuise adversarial attack method to perturb the malware sample. This attack method inserts empty semantic nodes (i.e., invalid nodes that do not affect program logic and function) into the program graph, thereby perturbing the graph structure features to induce the detection model to make false judgments without changing the file behavior.

[0095] To evaluate the model's stability under stringent detection conditions, the experiment used a detection threshold with a false positive rate of 0.1%. If the model could not maintain normal classification performance at this threshold, a threshold with a false positive rate of 1% was used instead. The experimental sample consisted of 1000 software samples that were initially classified as highly malicious in the original predictions, i.e., samples with prediction scores higher than the set threshold. By applying a MalGuise attack to these samples and re-detecting them, the proportion of samples misclassified as benign after the attack was statistically analyzed (i.e., the escape rate).

[0096] Table 5. Robustness detection rate of the model

[0097]

[0098] The detection results of each model under adversarial attacks are shown in Table 5. As can be seen from Table 5, the existing MalGraph and MAGIC models exhibit high escape rates (87.39% and 97.81% respectively) when facing MalGuise attacks, indicating that their detection results are susceptible to structural perturbations. In contrast, the I2P-Aligned-GNN model described in this invention has an escape rate of only 1.44% under the same attack conditions, significantly lower than the comparative models. This result demonstrates that this invention, by introducing a feature alignment mechanism based on instruction sequences and multi-layer program graphs, effectively enhances the detector's ability to perceive structural perturbations and its adversarial stability. Even under complex graph-level adversarial attacks, this invention can still accurately identify malicious samples, demonstrating stronger robustness and generalization ability. In summary, the detection method of this invention not only maintains high detection accuracy with a low false positive rate but also exhibits significant anti-interference performance when facing adversarial samples, verifying its application value in practical malware detection and protection systems.

[0099] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A malware detection optimization method based on a multi-layer graph neural network, characterized in that, Includes the following steps: S1. Perform static analysis on the software's binary file to extract three layers of static features; S2. Represent the software hierarchically as three layers of extracted static features, and learn the program-level representation of the features through a graph neural network; S3. By inputting the program-level representation into the multilayer perceptron and outputting the binary classification score to obtain the predicted probability, malware detection is completed.

2. The malware detection optimization method based on multi-layer graph neural network according to claim 1, characterized in that, The specific implementation process of step S1 is as follows: S1.1 Construct the function call graph structure of the software by analyzing the calling instructions generated from the software disassembly. S1.2 Construct the control flow graph structure of the software's native functions by analyzing the jump instructions generated from the software disassembly. S1.3 Construct the instruction sequence diagram of the basic blocks according to the instruction order of the basic blocks in the control flow diagram.

3. The malware detection optimization method based on a multi-layer graph neural network according to claim 2, characterized in that, The specific implementation process of S1.3 is as follows: S1.3.1 For a basic block b, its instruction sequence is denoted as follows: Introduce a start node on this basic block. With the termination node ; Each instruction Corresponding to a node The set of nodes is ; S1.3.2, The edge set of the instruction sequence graph is divided into two main categories: data-dependent edges. Effect-dependent edge , Representing basic blocks Data between instructions Dependency Expressing execution order, visibility, and approximate value flow; register aliases use equivalence class functions. express, Give registers The alias equivalence class to which it belongs, for any instruction , This indicates the set of registers it writes to. This indicates the set of registers it reads; S1.3.3 Before constructing the instruction sequence diagram, set up a status variable to record recently accessed states: the register-side states include mappings: Record equivalence class The index of the most recently written instruction; Record the index of the most recent read instruction; the memory-side state includes two types of mappings: Record for known address keys The most recent command index I wrote Record for known address keys The index of the most recently read instruction; retaining both scalars simultaneously. and Record the instruction indices of the most recent write and read operations at the unknown address, with initial values ​​of None; these states are used to find the most recent related access and establish edges in real time during a linear scan. S1.3.4, For each register currently being written... ,set up and For each register currently read ,set up If the current operation is a memory read, then retrieve... ,when season and will , This represents an unknown value; if the current operation is a memory write, then it is taken as... ,when season and will ; S1.3.5, Delete all self-loops; then perform transitive simplification for each edge type, i.e., when a path of the same type exists... Delete directly on the side Reduce redundancy; if the same ordered node pair If both data dependency and effect dependency exist, retain the data dependency and remove the effect dependency to avoid duplicate expressions; From nodes with an in-degree of zero Add effect edges to nodes with an out-degree of zero. Add effect edges; remove known pseudo-dependencies; obtain the instruction sequence graph. .

4. The malware detection optimization method based on a multi-layer graph neural network according to claim 3, characterized in that, The data dependency edge construction For each instruction First, construct data dependency edges at the register level; then... Each register read If a subscript exists Then add a line The data dependency edge represents the most recent data dependency edge. The exact value stream between the definition and the current read; The effect depends on edge construction The effect of constructing register-level edges depends on two edge types: WAR and WAW. These two edge types do not represent precise value flow, but rather express coverage and order constraints; for each register written by the current instruction... If it exists Then add If the WAR edge exists Then add The WAW edge.

5. The malware detection optimization method based on a multi-layer graph neural network according to claim 4, characterized in that, In step S1.2, each local function is extracted from the function call graph structure generated in S1.1, and the jump instructions are identified. Each basic block is used as a node, and the jump relationship is used as an edge. The corresponding control flow graph structure is constructed based on the jump instructions.

6. The malware detection optimization method based on a multi-layer graph neural network according to claim 5, characterized in that, In step S1.3, the instruction sequence of each basic block is extracted from the control flow graph structure generated in S1.

2. Based on the register write / read relationship and memory access constraints, the data dependency and control dependency between instructions are established to construct the instruction sequence graph of the basic block, so as to depict the data flow and execution order between instructions.

7. The malware detection optimization method based on a multi-layer graph neural network according to claim 3, characterized in that, The specific implementation process of step S2 is as follows: S2.1 Encode the feature of instruction nodes in the instruction sequence graph, learn the representation through graph neural network, and concatenate the block size prior as the initial representation of the basic block in the next layer of control flow graph; S2.

2. The initialization representation is used as the node input of the control flow graph of the function. After being aggregated by a multi-layer graph neural network and pooled across the entire graph, the function-level representation is obtained and used as the initialization representation of the local function in the function call graph of the next layer. S2.3 In the function call graph, nodes are initialized by embedding the names, properties, and parameters of local functions and Windows system APIs. After multi-layer graph neural network aggregation and full graph pooling, a program-level representation is obtained.

8. The malware detection optimization method based on a multi-layer graph neural network according to claim 7, characterized in that, In step S2.1, an operator vocabulary and an operand vocabulary are established. A single instruction node is mapped to a learning vector of operator concatenation and operand concatenation, which is used as the original input of the graph neural network for learning. The logarithmic scale of the basic block instruction size is concatenated as the initial feature representation of the basic block node in the next layer control flow graph.

9. The malware detection optimization method based on a multi-layer graph neural network according to claim 8, characterized in that, In step S2.3 of the function call graph, firstly, a Windows system API function vocabulary is established, mapping Windows system API functions to learnable vectors aligned with the initial input of local functions; simultaneously, a parameter vocabulary is established, mapping function parameters to learnable vectors; then, the API vectors or the initial input vectors of local functions, the parameter vectors, and the in-degree, out-degree, stack size, cyclomatic complexity, and recursion flag of the functions are concatenated to form a node representation of the function call graph. This node represents the input to a multi-layer graph neural network, which, after aggregation and full graph pooling, yields a program-level representation.

10. A malware detection optimization system based on a multi-layer graph neural network, used to implement the malware detection optimization method according to any one of claims 1 to 9, characterized in that, Includes the following modules: The static feature extraction module is used to perform static analysis on the software's binary files to extract three layers of static features. The program-level representation module is used to represent the software hierarchically as extracted three-layer static features, and learns the program-level representation of the features through a graph neural network; The detection result output module is used to obtain the predicted probability by inputting the program-level representation into the multilayer perceptron and outputting the binary classification score, thereby completing the malware detection.

Images