Binary patch existence detection method and device based on multi-dimensional signature features

Abstract

The invention discloses a binary patch existence detection method and device based on a multi-dimensional signature feature, and the method comprises the steps: obtaining the vulnerability description of third-party software, compiling the source code of a corresponding vulnerability version and a repair version into a binary file for each vulnerability, and generating a reference retrieval library; the method comprises the following steps: acquiring public Internet of Things firmware, unpacking the firmware, and taking a binary file in the firmware as a to-be-detected file; extracting assembly instruction semantic information and control flow graph information of each function from all binary files to generate a function signature, and extracting a difference between a vulnerability function and a corresponding patch function from the binary files to generate a patch signature; training the model by using the function signature and the patch signature and generating embedding; finding out possible vulnerabilities and corresponding target detection files through an unsupervised clustering and similarity calculation method based on embedding of function signatures; and judging whether the patch exists in the target detection file or not through similarity calculation by utilizing the embedding of the patch signature.

Description

Technical Field

[0001] This invention relates to the field of IoT software supply chain security, specifically to a method and apparatus for detecting the existence of binary patches based on multi-dimensional signature features. Background Technology

[0002] With the continuous development of the open-source community, the number of newly discovered vulnerabilities is rapidly increasing. Software vulnerabilities are security issues that can lead to attacks or abuse of software systems by malicious users. Applying patches is one of the most effective ways to fix vulnerabilities and improve system security, and software developers continuously release patches to fix known vulnerabilities. Although developers often use patched versions for development in a timely manner, during vulnerability checks, because patch changes are usually small and subtle, and the differences between vulnerable functions and patched functions are small, it can lead to the misclassification of patched software as vulnerable software. With the development and widespread deployment of the Internet of Things (IoT), accurately identifying whether binary files in firmware are vulnerable versions or patched versions is becoming increasingly important.

[0003] In open-source libraries, recurring vulnerabilities are called 1-day vulnerabilities. Due to code reuse and sharing, they have become one of the most significant threats to cybersecurity. Large software systems contain tens of thousands of binary files, with different versions developed by various vendors. If binary code similarity vulnerability search methods are still used to detect software, it will lead to misclassification of patched and vulnerable versions, exposing millions of devices to serious software security threats that are not actually present. Developers will consume significant resources due to false positives. Furthermore, there are numerous IoT devices, but their firmware images only provide binary files, lacking source code available for security analysis. Their symbolic information is often stripped away, making many traditional analysis methods unsuitable for complex IoT scenarios.

[0004] Patch existence detection refers to the process of finding functions with high similarity to the vulnerable function within one or more binary files, given a vulnerable function and its corresponding patch function, using similarity detection techniques. This involves precise analysis to identify functions with high similarity to collected patch functions. Current research has proposed many methods for patch existence detection, which can be broadly categorized into two types: similarity-based methods and signature-based methods. However, existing methods become increasingly difficult in complex IoT environments, where software is compiled using different compilers and optimization options. Therefore, designing a more efficient binary patch existence detection method to find patches and filter out real vulnerabilities is of great significance for solving the problem of real-world n-day vulnerability detection in complex IoT environments. Summary of the Invention

[0005] To overcome the above-mentioned shortcomings of the existing technology, this invention proposes a binary patch existence detection method and device based on multi-dimensional signature features, with the goal of effectively detecting patches of third-party software in IoT firmware.

[0006] According to a first aspect of the embodiments of this application, a binary patch existence detection method based on multi-dimensional signature features is provided, comprising:

[0007] (1) Obtain binary files: Obtain the vulnerability description of third-party software. For each vulnerability, compile the source code of the corresponding vulnerability version and the patch version into binary files to generate a reference search library; obtain the publicly available IoT firmware and unpack it, and use the binary files in it as files to be detected.

[0008] (2) Extract signature features: Extract assembly instruction semantic information and control flow graph information of each function from the binary file in the reference retrieval library and the file to be detected to generate function signatures; extract the differences between the vulnerability function and the corresponding patch function from the binary file to generate patch signatures;

[0009] (3) Training the model: The model is trained using the function signatures and patch signatures extracted in step (2), and the embedded information is generated for each function signature and patch signature using the trained model.

[0010] (4) Search for vulnerability data: Based on the function signatures generated in step (3), the vulnerability functions in the reference retrieval library are divided into several classes by unsupervised clustering, and the functions to be detected in the file to be detected are classified into the corresponding classes. The similarity between the function to be detected and the vulnerability functions in the reference retrieval library is calculated in turn, and the search results are sorted to find possible vulnerabilities. The binary file corresponding to the vulnerability is used as the target detection file.

[0011] (5) Patch existence detection: By embedding the patch signature, the similarity calculation is used to determine whether a patch exists in the target detection file.

[0012] Further, step (1) includes:

[0013] (1-1) Analyze and statistically analyze the usage of third-party libraries in IoT application development, and collect commonly used third-party libraries by checking the vulnerability fixes;

[0014] (1-2) Based on the third-party library collected in (1-1), obtain the vulnerability information and repair information of the third-party library, including the software name, the version number of the vulnerability and the version number of the vulnerability repair, the vulnerability file name, the patch file name, the vulnerability function name, and the patch function name. Use the software name, vulnerability version number, vulnerability repair version number, vulnerability file name, patch file name, vulnerability function name, and patch function name as keywords and store the information in the reference retrieval library.

[0015] (1-3) Using the GCC compiler and MakeFile information, compile the files containing vulnerabilities and the files containing patches into binary files, and store them in the reference retrieval library as known vulnerability files and patch files;

[0016] (1-4) Collect the firmware to be tested from IoT firmware vendors, unpack the firmware, extract the binary file obtained from the unpacking as the file to be tested, and store it in the database to be tested.

[0017] Further, step (2) includes:

[0018] (2-1) Decompile each binary file in the reference retrieval library and the database to be detected, and extract the assembly instruction information and control flow graph information included in the binary file as function signatures. Based on the vulnerability function names obtained in step (1), filter out the vulnerability functions to be retrieved, update the corresponding function signatures in the reference retrieval library, and update the function signatures of the files to be detected in the database to be detected.

[0019] (2-2) For each function signature, use a tag. <str>Replace string literals in assembly instruction information; use tags <const>Replace constants; use mnemonics and operands, along with the name and label of the outer function call, as tokens; and replace the name of the inner function call with... <function>For each instruction jump pair, replace its source token with token JUMP_XXX, where XXX is the sequential number of the target token of the jump pair, and update the normalized function signature features back to the reference retrieval library and the database to be detected.

[0020] (2-3) The differences between the vulnerable functions and patch functions in the reference search library are obtained using Git tools. A basic block mapping algorithm is used to extract the basic blocks that have been modified, added, or deleted in the differences between the vulnerable functions and patch functions, based on basic block syntax and context information. This generates valid traces for the vulnerable functions and patch functions. Symbolic simulation technology is used to track changes in registers and memory relative to function parameters and the initial memory environment in the vulnerable and patch files, extracting register data writes, memory data storage, conditions, function calls, and return values. These valid traces and changes are then used as patch signatures to update the reference search library. For each function in the file to be detected, the valid traces and changes are generated as patch signatures and updated to the database to be detected.

[0021] (2-4) Sign each patch using a tag. <address>Replace the specific address; use tags <mem>Replace direct memory access; use <reg>Replace register reads; normalize immediate values, but do not normalize constants other than immediate values.

[0022] Furthermore, step (3) includes:

[0023] The model is pre-trained using a first training task based on a masked language model and a second training task based on a jump target prediction task. The pre-training objective is to minimize the total loss function.

[0024]

[0025] in For the total loss function, The loss function for the first training task is... Here, θ is the loss function for the second training task, and θ represents the model parameters.

[0026] By using a contrastive learning-based fine-tuning approach to maximize the similarity between similar pairs of binary functions and minimize the similarity between unrelated pairs of binary functions, the objective function is expressed as:

[0027]

[0028] in Let E represent the loss function for contrastive learning, ε be a parameter between 0 and 0.5, cos(·,·) represent the cosine similarity, f be the function to be detected, g be a vulnerability function in the reference retrieval database, let g+∈g be a function similar to f, and let g-∈g be an arbitrary function independent of f. Let the embedding of function f be denoted as E. f Let D be defined as all generated triples.<f,g+,g-> A set;

[0029] Using the fine-tuned model, function signature features from both the reference retrieval library and the database to be tested are embedded and updated into the corresponding databases.

[0030] Furthermore, the first training task using the mask language model specifically involves: for a function signature, randomly selecting a first predetermined proportion of the mask to be replaced by a special mask, the special mask indicating that the original mask needs to be re-predicted; a second predetermined proportion of the mask to be replaced by other random masks, while the remaining masks remain unchanged;

[0031] Based on this, the objective function for re-predicting the mask in the first training task is defined as:

[0032]

[0033] Where m x f is the index of the mask in the function signature. mlm =replace(f,m) x , <mask>) is a set of random position masks, replace represents the mask replacement function, f represents a function to be detected in the file to be detected, and x i This represents the i-th label of function f. <mask>This indicates a special mask to be replaced with.

[0034] Furthermore, the second training task based on the jump target prediction task specifically involves predicting the corresponding target token given a randomly selected jump source token.

[0035] Based on this, the objective function of the second training task is defined as:

[0036]

[0037] Among them l x For a set of positions of the jump symbol, f jtp =replace(f,l) x , <mask>) is a set of random jump tokens.

[0038] Further, step (4) includes:

[0039] (4-1) Using the K-MEANS clustering algorithm, based on the embedding values ​​corresponding to the vulnerability functions in the reference retrieval library, the vulnerability functions in the reference retrieval library are divided into K clusters and the K cluster center points are calculated;

[0040] (4-2) Calculate the distance between the function to be detected and the centers of the K clusters, find the point with the closest distance, and assign the function to be detected to that cluster.

[0041] (4-3) Calculate the similarity between the embedding of the function to be detected and all the vulnerability function embeddings of the same type in the reference retrieval library according to the classification results in (4-2);

[0042] (4-4) Select the vulnerable functions in the reference search library that have the highest similarity to the function to be detected as the functions most likely to cause vulnerabilities in the binary file to be detected, and mark the binary file as the target detection file.

[0043] Further, step (5) includes:

[0044] (5-1) Calculate the similarity between each pair of trajectories between the effective trajectory set of the target detection file and the effective trajectory set of the vulnerability function in the patch signature of the reference retrieval library, and the effective trajectory set of the patch function, to form the first patch similarity score S1 and the first vulnerability similarity score P1 between the effective trajectory sets;

[0045] The similarity formula between the effective trajectory sets T1 and t2 is as follows:

[0046]

[0047] t1 and t2 are the valid trajectories in the valid trajectory sets T1 and T2, respectively, len(·) is the trajectory length, and the similarity between the valid trajectories t1 and t2 is... edit(t1,t2) is the distance between trajectories t1 and t2;

[0048] (5-2) Calculate the matching score between the changes in the patch signatures of the target detection file and the reference retrieval library to form the second patch similarity score S2 and the second vulnerability similarity score P2;

[0049] The matching function between any two files s1 and s2 is:

[0050]

[0051] Among them, name represents the function call name, para represents the function call parameter, index represents the register addressing subscript, value represents the register addressing value, addr represents the memory address, exp represents the custom label name, Call represents the function call, RW represents the register write-back, MW represents the memory write-back, Condition represents the conditional expression, and Return represents the return;

[0052] (5-3) Add the two patch and vulnerability similarity scores to obtain S12 and P12 respectively. Check whether there is a patch in the target file by comparing the two similarity scores. If S12 > P12, it is determined that there is no patch in the target detection file; if S12 < P12, it is determined that there is a patch in the target detection file.

[0053] According to the second aspect of the embodiments of the present application, there is provided a binary patch existence detection device based on multi-dimensional signature features, including:

[0054] A binary file acquisition module, configured to obtain a vulnerability description of third-party software. For each vulnerability, compile the source codes of the corresponding vulnerable version and fixed version into binary files to generate a reference retrieval library; obtain the publicly available Internet of Things firmware and unpack it, and use the binary files therein as the files to be detected;

[0055] A signature feature extraction module, configured to extract the semantic information of the assembly instructions and the control flow graph information of each function from the binary files in the reference retrieval library and the files to be detected to generate function signatures, and extract the differences between the vulnerable functions and the corresponding patch functions from the binary files to generate patch signatures;

[0056] A model training module, configured to train the model with the function signatures and patch signatures extracted by the signature feature extraction module, and generate embeddings for each function signature and patch signature through the trained model;

[0057] A vulnerability data search module, configured to classify the vulnerable functions in the reference retrieval library into several categories by an unsupervised clustering method based on the embeddings of the function signatures generated by the model training module, classify the functions to be detected in the files to be detected into the corresponding categories, calculate the similarity between the functions to be detected and the vulnerable functions in the reference retrieval library in turn, sort the retrieval results, find the possible vulnerabilities, and use the binary files corresponding to the vulnerabilities as the target detection files;

[0058] A patch existence detection module, configured to use the embeddings of the patch signatures to determine whether there is a patch in the target detection file through similarity calculation.

[0059] According to the third aspect of the embodiments of the present application, there is provided an electronic device, including:

[0060] One or more processors;

[0061] Memory, used to store one or more programs;

[0062] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0063] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0064] As can be seen from the above embodiments, this application addresses the problems of high false positive rates in existing binary code similarity-based vulnerability searches and insufficient accuracy and low efficiency in existing patch existence detection. It proposes a detection method combining multi-dimensional signature features, which combines assembly instruction information and control flow information from binary files. A semantic model is pre-trained and fine-tuned to learn function semantics and structural information. Signature feature embedding identifies functions that may cause vulnerabilities. Finally, patch (vulnerability) similarity scores are calculated based on patch signatures, and the presence of patches in the target file is determined according to the scores. Combining these methods significantly reduces the false positive rate of vulnerability searches, accurately detects n-day vulnerabilities in complex IoT environments, achieves higher detection accuracy, and possesses better detection efficiency.

[0065] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0066] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0067] Figure 1 This is a flowchart illustrating a binary patch existence detection method based on multidimensional signature features according to an exemplary embodiment.

[0068] Figure 2 This is a block diagram illustrating a binary patch existence detection device based on multidimensional signature features according to an exemplary embodiment.

[0069] Figure 3 This is a schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0071] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms "a," "the," and "the" as used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0072] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0073] This application provides a binary patch existence detection method based on multi-dimensional signature features, including compiling relevant vulnerability and fix binary data, extracting IDA Pro function signature features and Git patch signature features, and outputting multi-dimensional signature features.

[0074] The function signature feature representation part of this invention designs two pre-training methods and one fine-tuning method for BERT to train the model and generate high-quality embeddings; the patch signature feature representation part uses changes such as effective trajectory sets and register memory to reduce false positives during vulnerability search and improve search accuracy and efficiency.

[0075] Reference Figure 1 A binary patch existence detection method based on multi-dimensional signature features includes the following steps:

[0076] (1) Obtain binary files: Obtain the vulnerability description of third-party software. For each vulnerability, compile the source code of the corresponding vulnerability version and the patch version into binary files to generate a reference search library; obtain the publicly available IoT firmware and unpack it, and use the binary files in it as files to be detected.

[0077] (2) Extract signature features: Extract assembly instruction semantic information and control flow graph information of each function from the binary file in the reference retrieval library and the file to be detected to generate function signatures; extract the differences between the vulnerability function and the corresponding patch function from the binary file to generate patch signatures;

[0078] (3) Training the model: The model is trained using the function signatures and patch signatures extracted in step (2), and the embedded information is generated for each function signature and patch signature using the trained model.

[0079] (4) Search for vulnerability data: Based on the function signatures generated in step (3), the vulnerability functions in the reference retrieval library are divided into several classes by unsupervised clustering, and the functions to be detected in the file to be detected are classified into the corresponding classes. The similarity between the function to be detected and the vulnerability functions in the reference retrieval library is calculated in turn, and the search results are sorted to find possible vulnerabilities. The binary file corresponding to the vulnerability is used as the target detection file.

[0080] (5) Patch existence detection: By embedding the patch signature, the similarity calculation is used to determine whether a patch exists in the target detection file.

[0081] Step (1) specifically involves collecting corresponding vulnerability descriptions of third-party software from the project's official website and the National Vulnerability Database (NVD). For each collected vulnerability, the source code of the corresponding vulnerability version and the patch version is compiled into a binary file using a compiler to generate a reference retrieval library. Publicly available IoT firmware is collected from the Internet, and the collected firmware is unpacked to obtain the binary file as the file to be detected. Further, step (1) may include the following process:

[0082] (1-1) Analyze and statistically analyze the usage of commonly used third-party libraries in IoT application development, and collect commonly used third-party libraries by checking the vulnerability fix situation. Specifically, if a third-party library has a vulnerability but the vulnerability has not been fixed, the third-party library will not be collected.

[0083] (1-2) Based on the third-party libraries collected in (1-1), search for vulnerability information and patch information of the corresponding third-party libraries on the project website and the National Vulnerability Data (NVD) website. This includes the software name, the version number of the vulnerability and the version number of the patch, the vulnerability file name, the patch file name, the vulnerability function name, and the patch function name. Use the software name, vulnerability version number, vulnerability patch version number, vulnerability file name, patch file name, vulnerability function name, and patch function name as keywords and store this information in the reference retrieval database.

[0084] (1-3) Using the GCC compiler and the information in the Makefile of the project, compile the files in the project that contain vulnerabilities and the files that contain patches into binary files, and store them in the reference retrieval library as known vulnerability files and patch files.

[0085] (1-4) Collect firmware to be tested (such as firmware of commonly used devices) from IoT firmware vendors, and unpack the firmware using BinWalk with the unblob tool, extract the unpacked binary file as the file to be tested, and store it in the database to be tested.

[0086] Step (2) specifically involves: using an IDA Pro automated analysis script to extract the assembly instruction semantics and control flow graph information of each function from the binary file to generate a function signature; and using a tokenization strategy to normalize the assembly code to reduce its vocabulary size; using Git to automatically obtain file differences, extracting the differences between the vulnerable function and the corresponding patch function from the binary file to generate a patch signature; and using binary instruction normalization to eliminate the influence of other differences. Further, step (2) may include the following process:

[0087] (2-1) Design an IDA Pro Python script, use the IDC module to decompile each binary file in the reference search library and the database to be detected, and extract the assembly instruction information and control flow graph information included in the binary file as function signatures. Based on the vulnerability function names collected in (1), filter out the vulnerability functions to be searched, update the function signatures in the reference search library, and update the function signatures of the files to be detected in the database to be detected.

[0088] (2-2) Design an assembly code instruction normalization strategy, using tags. <str>Replace string literals in assembly instruction information; use tags <const>Replace constants; use mnemonics and operands, along with the name and label of the outer function call, as tokens; and replace the name of the inner function call with... <function>For each instruction jump pair, its source token (i.e., the absolute or relative address of the jump target) is replaced with the token JUMP_XXX, where XXX is the sequential number of the target token for that jump pair. The normalized function signature features are then updated in the reference retrieval library and the database to be detected. By normalizing the assembly instruction information, OOM (Out Of Memory) problems that may be caused by the original assembly information are avoided, facilitating subsequent feature identification.

[0089] (2-3) The differences between the vulnerable functions and patch functions in the reference search library are obtained using Git tools. A basic block mapping algorithm is used to extract the basic blocks that have been modified, added, or deleted in the differences between the vulnerable functions and patch functions, employing basic block syntax and contextual information. Based on these difference basic blocks, valid traces are generated for the vulnerable functions and patch functions (Xu, Yifei, et al. "Patch based vulnerability matching for binary programs." ACM (2020)). Symbolic simulation technology is used to track changes in registers and memory relative to function parameters and initial memory environments in the vulnerable and patch files, i.e., register data writing, memory data storage, conditions, function calls, and return values. These valid traces and changes are used as patch signatures to update the reference search library. For the file to be tested, the same valid traces and changes need to be generated as patch signatures for each function, and these patch signatures are also updated in the database to be tested.

[0090] (2-4) Design a normalization strategy for patch signature binary instructions, using tags. <address>Replace the specific address; use tags <mem>Replace direct memory access; use <reg>Replace register reads; do not normalize constants, as some patches might only address the vulnerability by changing constants. Since there are no vulnerabilities designed for immediate values, they can be normalized.

[0091] In step (3), the model training process includes MLM pre-training and JTP pre-training, specifically as follows:

[0092] (3-1) Design a Masked Language Model (MLM) to perform the first pre-training task on the BERT model. For function signatures or patch signatures, randomly select 80% of the masks to be replaced by a special mask. This special mask indicates that the original mask needs to be re-predicted. In addition, 10% of the masks are replaced by other random masks, and 10% of the masks remain unchanged.

[0093] Based on these definitions, the objective function for re-predicting the masked token can be defined as:

[0094]

[0095] in Here, θ is the loss function for MLM pre-training, θ represents the BERT model parameters, and m... x f is the index of the mask in the function signature or patch signature. mlm =replace(f,m) x , <mask>) is a set of random position masks, replace represents the mask replacement function, f represents a function to be detected in the file to be detected, and x i This represents the i-th label of function f. <mask>This represents a special mask used for replacement. MLM pre-training enables the understanding of the semantic information of function signatures and patch signatures.

[0096] (3-2) Design a Jump Target Prediction (JTP) task to perform a second pre-training task on the BERT model. Given a randomly selected jump source token, predict its corresponding target token. The objective function of the jump task prediction can be defined as:

[0097]

[0098] in Here, θ is the loss function for JTP pre-training, and θ represents the BERT model parameters. x f represents a set of positions for a jump symbol (a symbol with jump logic corresponding to an assembly instruction). jtp =replace(f,l) x , <mask>) is a set of random jump tokens.

[0099] Based on the foregoing, the overall loss function of the model during the pre-training phase can be expressed as the sum of the MLM and JTP objective functions:

[0100]

[0101] After unsupervised pre-training based on the overall loss function, fine-tuning is performed to maximize the similarity between similar binary function pairs while minimizing the similarity between irrelevant binary function pairs. Training is conducted using a contrastive learning-based fine-tuning approach, and its objective function can be expressed as:

[0102]

[0103] in Let E represent the loss function for contrastive learning, θ represent the model parameters, ε represent parameters between 0 and 0.5, cos(·,·) represent the cosine similarity, f represent the function to be detected, g represent the vulnerability functions in the reference retrieval library, let g+∈g be functions similar to f (e.g., compiled from the same source code), and let g-∈g be an arbitrary function independent of f. Let E be the embedding of the function f. f Let D be defined as all generated triples.<f,g+,g-> A set of.

[0104] After fine-tuning the model, function signatures and patch signatures from the reference retrieval library and function signatures from the database to be tested are generated, embedded, and updated in the corresponding databases.

[0105] Step (4) specifically involves dividing the vulnerability functions in the reference retrieval library into several classes using unsupervised clustering, classifying the firmware function to be detected into the corresponding class, calculating the similarity between the firmware function to be detected and the vulnerability functions in the retrieval library, sorting the retrieval results, and identifying potential vulnerabilities (i.e., the vulnerability function and the binary file containing the vulnerability function). This step may specifically include the following processes:

[0106] (4-1) Using the K-MEANS clustering algorithm, the embedding values ​​corresponding to the vulnerability functions in the reference retrieval database are input, the vulnerability functions in the reference retrieval database are divided into K clusters and the K cluster centroids are calculated. The algorithm is as follows:

[0107]

[0108] Where J(C) is the loss function, K is the number of sample clusters, and C i Let x be the point set of the i-th cluster, and x belong to C. i Data points, u i It is the centroid of the i-th cluster.

[0109] (4-2) Calculate the distance between the embedding of the function to be detected and the K cluster centers, find the point with the closest Euclidean distance, and classify the function to be detected into that class;

[0110] The distance is calculated using Euclidean distance:

[0111]

[0112] Where x is the point to be detected, and y is the cluster center point.

[0113] (4-3) Calculate the similarity between the embedding of the function to be detected and all the embeddings of the same type of vulnerability function in the reference retrieval library according to the classification results in (4-2);

[0114] Similarity is calculated using cosine distance, and the formula is as follows:

[0115]

[0116] Where sim i This is the cosine similarity result, A i B is the embedding value of the function to be detected. i This is for referencing the embedded values ​​of vulnerable functions in the retrieval library.

[0117] (4-4) Select the vulnerable functions in the reference search library that have the highest similarity to the function to be detected as the functions most likely to cause vulnerabilities in the binary file to be detected, and mark the binary file as the target detection file.

[0118] Step (5) specifically involves using the embedding of patch signatures to sequentially calculate the similarity between the target detection file and the patch functions in the reference retrieval library, and analyzing the detection results to identify the functions with patches. This step may include the following processes:

[0119] (5-1) Patch existence is identified by calculating the similarity scores between the effective trajectory set of the target detection file and the effective trajectory set of the vulnerability function in the patch signature of the reference retrieval library, and the effective trajectory set of the patch function. To calculate the trajectory set similarity score, first calculate a pair of effective trajectory similarity scores. For the basic block in the trajectory, all jump instructions are deleted to form a sequential instruction trajectory. After obtaining the individual score of each pair of trajectories, the final similarity between the two trajectory sets is calculated to form the first patch (vulnerability) similarity score S1 (P1).

[0120] The formula for the similarity of a pair of valid trajectories is:

[0121]

[0122] After having a single score for each pair of traces, the similarity formula for two trace sets is as follows:

[0123]

[0124] where \(t_1\) and \(t_2\) are the valid traces in the valid trace sets \(T_1\) and \(T_2\) respectively, \(len(\cdot)\) is the trace length, and \(edit(t_1, t_2)\) is the distance between the traces \(t_1\) and \(t_2\).

[0125] (5 - 2) Patch existence recognition is performed by calculating the matching score between the patch signatures (register data writes, memory data stores, conditions, function calls, and return values) extracted from the target detection file and the reference retrieval library. For signatures of the same type, they match only when all components are equal. Since function calls and conditional expressions are more important than writing to memory or registers, higher weights are assigned to function calls and conditional expressions. The second patch (vulnerability) similarity score \(S_2(P_2)\) is formed by calculating the weighted matching scores between the vulnerability files and patch files in the target detection file and the reference retrieval library respectively.

[0126] The matching function between any two files \(s_1\) and \(s_2\) is as follows:

[0127]

[0128] where name represents the function call name, para represents the function call parameters, index represents the register addressing subscript, value represents the register addressing value, addr represents the memory address, exp represents the custom label name (similar to a variable name), Call represents the function call, RW represents the register write-back, MW represents the memory write-back, Condition represents the conditional expression, and Return represents the return.

[0129] (5 - 3) Add the two patch and vulnerability similarity scores to obtain \(S_{12}\) and \(P_{12}\) respectively. By comparing the two similarity scores, it is checked whether there is a patch in the target file. If \(S_{12}>P_{12}\), it means that the target function is more similar to the vulnerability function, that is, there is no patch in the target detection file and it is a vulnerable function. If \(S_{12}<P_{12}\), it means that the target function is more similar to the patch function, that is, there is a patch in the target detection file and it is a patch function.

[0130] Corresponding to the embodiment of the binary patch existence detection method based on multi-dimensional signature features described above, the present application also provides an embodiment of a binary patch existence detection device based on multi-dimensional signature features.

[0131] Figure 2 [[ID= Figure 2 The device may include:

[0132] The binary file acquisition module 21 is used to acquire vulnerability descriptions of third-party software. For each vulnerability, it compiles the source code of the corresponding vulnerability version and the patch version into a binary file to generate a reference retrieval library; it acquires publicly available IoT firmware and unpacks it, using the binary file within as the file to be detected.

[0133] The signature feature extraction module 22 is used to extract the assembly instruction semantic information and control flow graph information of each function from the binary file in the reference retrieval library and the file to be detected to generate a function signature, and to extract the difference between the vulnerable function and the corresponding patch function from the binary file to generate a patch signature;

[0134] Model training module 23 is used to train the model with the function signatures and patch signatures extracted by the signature feature extraction module, and to generate embeddings for each function signature and patch signature using the trained model;

[0135] The vulnerability data search module 24 is used for embedding the function signature generated by the model training module. It divides the vulnerability functions in the reference retrieval library into several classes through unsupervised clustering, and classifies the functions to be detected in the file to be detected into the corresponding classes. It calculates the similarity between the function to be detected and the vulnerability functions in the reference retrieval library in turn, sorts the search results, finds the possible vulnerabilities, and uses the binary file corresponding to the vulnerability as the target detection file.

[0136] The patch existence detection module 25 is used to determine whether a patch exists in the target detection file by using the embedding of the patch signature and similarity calculation.

[0137] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0138] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0139] Accordingly, this application also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the binary patch existence detection method based on multidimensional signature features as described above.

[0140] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the binary patch existence detection method based on multi-dimensional signature features as described above. Figure 3 The diagram shown is a hardware structure diagram of any data processing-capable device, including a binary patch existence detection device based on multi-dimensional signature features provided in an embodiment of the present invention. (Except for...) Figure 3 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0141] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the binary patch existence detection method based on multi-dimensional signature features as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0142] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.< / mask> < / mask> < / mask> < / reg> < / mem> < / address> < / function> < / const> < / str> < / mask> < / mask> < / mask> < / reg> < / mem> < / address> < / function> < / const> < / str>

Claims

1. A binary patch existence detection method based on multi-dimensional signature features, comprising the following steps: (1) obtaining binary files: obtaining vulnerability descriptions of third-party software, for each vulnerability, compiling the source code of the corresponding vulnerability version and the repair version into binary files to generate a reference search library; obtaining public Internet of Things firmware and unpacking, taking the binary files therein as detection files; (2) extracting signature features: extracting the assembly instruction semantic information and control flow graph information of each function from the binary files in the reference search library and the detection files to generate function signatures, and extracting the differences between the vulnerability functions and the corresponding patch functions from the binary files to generate patch signatures; (3) training a model: training the model using the function signatures and patch signatures extracted in step (2), and generating embeddings for each function signature and patch signature through the trained model; (4) searching for vulnerability data: based on the embeddings of the function signatures generated in step (3), the vulnerability functions in the reference search library are classified into several categories through an unsupervised clustering method, and the detection functions in the detection files are classified into the corresponding categories, the detection functions and the vulnerability functions in the reference search library are sequentially calculated for similarity, and the search results are sorted to find possible vulnerabilities, and the binary file corresponding to the vulnerability is taken as a target detection file; (5) detecting patch existence: using the embeddings of the patch signatures, and judging whether there is a patch in the target detection file through similarity calculation.

2. The method of claim 1, wherein, Step (1) includes: (1-1) analyzing and counting the use of third-party libraries in the development of Internet of Things applications, and collecting commonly used third-party libraries through vulnerability repair; (1-2) according to the third-party libraries collected in (1-1), obtaining the vulnerability information and repair information of the third-party libraries, including software name, version number of the vulnerability, version number of the vulnerability repair, vulnerability file name, patch file name, vulnerability function name, patch function name, and storing the software name, vulnerability version number, vulnerability repair version number, vulnerability file name, patch file name, vulnerability function name, and patch function name as keywords in the reference search library; (1-3) compiling the files with vulnerabilities and the files with patches into binary files through the GCC compiler combined with MakeFile information, and storing them as known vulnerability files and patch files in the reference search library; (1-4) collecting the firmware to be detected from Internet of Things firmware suppliers, unpacking the firmware, extracting the binary files obtained by unpacking as detection files, and storing them in a detection database.

3. The method of claim 1, wherein, Step (2) includes: (2-1) decompiling each binary file in the reference search library and the detection database, and extracting the assembly instruction information and control flow graph information included in the binary file as function signatures, and screening the vulnerability functions to be searched according to the vulnerability function names obtained in step (1), updating the corresponding function signatures to the reference search library, and updating the function signatures of the detection files to the detection database; (2-2) For each function signature, use the tag <str>Replace string literal in assembly instruction information with token <const>Replace constants; mnemonic and operand names and labels for external function calls as tokens; and the names of internal function calls <function>; for each instruction jump pair, replace its source token with a token JUMP_XXX, where XXX is the sequential number of the target token of the jump pair, and update the normalized function signature features to the reference retrieval library and the detection database;< / function> < / const> < / str> (2-3) using the Git tool to obtain the differences between the vulnerability functions and the patch functions in the reference retrieval library, and using the basic block mapping algorithm, the basic blocks that are changed, added or deleted in the differences between the vulnerability functions and the patch functions are extracted by using the basic block syntax and context information, so as to generate effective trajectories for the vulnerability functions and the patch functions; using symbolic simulation technology to track the changes of registers and memory relative to function parameters and initial memory environment in the vulnerability file and the patch file, and extracting register data writing, memory data storage, conditions, function calls and return values; update these effective trajectories and changes as patch signatures to the reference retrieval library; for each function in the detection file, generate the effective trajectories and changes as patch signatures, and update to the detection database; (2-4) For each patch signature, use the tag <address>Replace specific address; use tag <mem>replacement direct memory access; using <reg>Replace register read; normalize the immediate number, and do not normalize the constant other than the immediate number.< / reg> < / mem> < / address> 4. The method of claim 1, wherein, Step (3) comprises: The first training task of the mask language model and the second training task based on the jump target prediction task are used for model pre-training, and the pre-training target is to minimize the total loss function: wherein is a total loss function, is a first training task loss function, is a second training task loss function, and θ is a model parameter; The target function is expressed as: wherein represents the loss function of contrastive learning, ε is a parameter between 0-0.5, cos(·,·) represents the cosine similarity, f is the function to be detected, g is a vulnerability function in the reference search library, g+∈g is a function similar to f, and g-∈g is an arbitrary function irrelevant to f, and the embedding of the function f is denoted as E f D is defined as the set of all generated triplets <f,g+,g->; The functions in the reference retrieval library and the detection database are embedded by using the fine-tuned model, and the embedded functions are updated to the corresponding databases.

5. The method of claim 4, wherein, The first training task of the mask language model is specifically: for a function signature, randomly select a first predetermined proportion of masks to be replaced by a special mask, and the special mask indicates that the original mask needs to be predicted again; the second predetermined proportion of masks are replaced by other random masks, and the remaining masks remain unchanged; Based on this, the target function of the re-predicted mask in the first training task is defined as: where m x is the index of the mask in the function signature, f mlm = replace(f, m x , <mask>) is a set of random position masks, replace denotes a mask replacement function, f denotes a function to be detected in the file to be detected, x i denotes the i-th mark of function f, <mask>The special mask is replaced by.< / mask> < / mask> 6. The method of claim 4, wherein, The second training task based on the jump target prediction task is specifically: by giving a randomly selected jump source token, the corresponding target token is predicted; Based on this, the target function of the second training task is defined as: where l x is a set of positions of the skip symbol, f jtp = replace(f, l x , <mask>) is a set of random jump tokens.< / mask> 7. The method of claim 1, wherein, Step (4) comprises: (4-1) using K-MEANS clustering algorithm, based on the embedding value corresponding to the vulnerability function in the reference retrieval library, the vulnerability functions in the reference retrieval library are divided into K clusters and K cluster center points are calculated; (4-2) calculate the distance between the detection function and the K cluster centers, find the nearest point, and divide the detection function into the class; (4-3) calculate the similarity between the embedding of the detection function and the embedding of all vulnerability functions in the reference retrieval library in the same class according to the classification result in (4-2). (4-4)The vulnerability functions in the reference search library with the highest similarity to the function to be detected are taken as the functions most likely to cause vulnerabilities in the binary file to be detected, and the binary file is marked as a target detection file.

8. The method of claim 1, wherein, Step (5) includes: (5-1) Calculate the similarity of each pair of trajectories between the effective trajectory set of the target detection file and the effective trajectory set of the vulnerability function and the effective trajectory set of the patch function in the patch signature of the reference search library, to form the first patch similarity score S1 and the first vulnerability similarity score P1 between the effective trajectory sets; Wherein, the similarity formula between the effective trajectory sets T1, T2 is: t1, t2 are effective trajectories in the effective trajectory sets T1, T2, respectively, len(·) is the trajectory length, and the similarity between the effective trajectories t1, t2 edit(t1, t2) is the distance between the trajectories t1 and t2. (5-2) Calculate the matching degree score between the target detection file and the patch signature in the reference search library, to form the second patch similarity score S2 and the second vulnerability similarity score P2; Wherein, the matching function between any two files s1, s2 is: Wherein, name represents the function call name, para represents the function call parameter, index represents the register addressing index, value represents the register addressing value, addr represents the memory address, exp represents the custom label name, Call represents the function call, RW represents the register write back, MW represents the memory write back, Condition represents the condition expression, and Return represents the return; (5-3) Add the two patch and vulnerability similarity scores respectively to obtain S12 and P12, and check whether the patch exists in the target file by comparing the two similarity scores. If S12>P12, it is determined that the target detection file does not exist in the patch; if S12<P12, it is determined that the target detection file exists in the patch.

9. A binary patch existence detection device based on multi-dimensional signature features, comprising the following steps: A binary file acquisition module is configured to obtain vulnerability descriptions of third-party software, compile source codes of corresponding vulnerability versions and repair versions into binary files to generate a reference search library for each vulnerability, and obtain and unpack public Internet of Things firmware to take binary files therein as a detection file; A signature feature extraction module is configured to extract assembly instruction semantic information and control flow graph information of each function from the binary files in the reference search library and the detection file to generate function signatures, and extract differences between vulnerability functions and corresponding patch functions from the binary files to generate patch signatures; A model training module is configured to train a model using the function signatures and patch signatures extracted by the signature feature extraction module, and generate embeddings of each function signature and patch signature using the trained model; A vulnerability data search module is configured to classify vulnerability functions in the reference search library into several classes using an unsupervised clustering method based on the embeddings of the function signatures generated by the model training module, classify detection functions in the detection file into corresponding classes, calculate similarities between the detection functions and the vulnerability functions in the reference search library in sequence, sort the search results, find possible vulnerabilities, and take a binary file corresponding to the vulnerability as a target detection file. The patch existence detection module is configured to determine whether the patch exists in the target detection file by similarity calculation with the embedding of the patch signature.

10. An electronic device, comprising: The method comprises: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-8.

Images