File detection method and device, electronic equipment, storage medium and program product
By extracting and fusing the text word frequency and statistical features of the object code file, combined with machine learning models, the problem of low detection accuracy of WebShell in the existing technology is solved, efficient identification of unknown and variant WebShells is achieved, and APT attacks are enhanced defense ability.
Patent Information
- Application Number
- CN202410947293.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-01
AI Technical Summary
The prior art has low accuracy when detecting unknown or variant WebShells. Traditional signature matching methods based on signatures are insufficient in detection accuracy, especially the complexity and variability of advanced persistent threat (APT) attacks are difficult to deal with.
By extracting the text word frequency characteristics and statistical characteristics of the object code file, input the corresponding detection model to obtain the probability results of the WebShell file, and fuse the model output to improve detection accuracy. Multi-feature detection method combined with machine learning algorithms are used to target PHP and JSP The format WebShell performs static detection.
Improves the accuracy of WebShell file detection, can more effectively identify unknown and variant WebShell, and enhances defense capabilities against APT attacks.
Smart Images

Figure CN118890183A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network security technology, and in particular to a file detection method, device, electronic device, storage medium and program product. Background Art
[0002] With the popularization of the Internet, the number and influence of network attackers have also increased. Advanced Persistent Threat (APT) is a widely concerned threat in the field of network security. It is a network intrusion behavior carried out by skilled professionals and complex auxiliary tools. Among them, webshell, as one of the commonly used APT attack weapons, is a key research area for APT attack detection. Webshell usually exists in web page file formats such as ASP, PHP, JSP, etc., and is also called web backdoor / backdoor file. After invading the host, the attacker usually uploads the backdoor file to the web server and hides it in the normal script application folder, so as to achieve the purpose of long-term and continuous control of the web server.
[0003] At present, traditional antivirus software mainly uses signature matching methods based on feature codes to detect WebShell. By analyzing known WebShell code samples, extracting their text features as malicious signatures, and forming a malicious signature database, the malicious signatures in the malicious signature database are matched with the signatures of the web page files to be detected, and the webshells can be detected, and then the abnormal network data flows that match their rule sets can be blocked in real time.
[0004] However, this method has high detection accuracy on known webshells, but for unknown or variant WebShells, the detection accuracy of existing technologies is low. Summary of the invention
[0005] The present application provides a file detection method, device, electronic device, storage medium and program product to solve the technical problem of low detection accuracy of webshell in the prior art.
[0006] In a first aspect, the present application provides a file detection method, comprising:
[0007] Get the target code file to be tested;
[0008] Extract text word frequency features and statistical features of the target code file;
[0009] Extracting a first detection feature from the text word frequency feature and the statistical feature, and inputting the first detection feature into a first detection model to obtain a first output result; the first output result is used to characterize the probability that the target code file is a Webshell file in PHP format;
[0010] Extracting a second detection feature from the text word frequency feature and the statistical feature, and inputting the second detection feature into a second detection model to obtain a second output result; the second output result is used to characterize the probability that the target code file is a Webshell file in the JSP format;
[0011] The first output result and the second output result are integrated to obtain a detection result of the target code file; the detection result is used to characterize whether the target code file is a Webshell file.
[0012] Optionally, in the above method, the target code file includes a plurality of target words and sentences;
[0013] Extract the text word frequency features of the target code file, including:
[0014] Perform word frequency statistics on the target code file to obtain the probability of the target words and sentences appearing in the target code file;
[0015] The word frequency inverse document frequency method is used to process the probability of the target word and sentence appearing, obtain the weight vector of the target word and sentence in the target code file, and determine the weight vector as the text word frequency feature of the target code.
[0016] Optionally, as in the above method, the word frequency statistics are performed on the target code file to obtain the probability of the target words and sentences in the target code file appearing, including:
[0017] Convert the target code file into a string variable to extract a corresponding string list in the target code file; the string list includes multiple target words and sentences;
[0018] The bag-of-words model is used to process the string list to convert it into a vector representation of the text, and the probability of the target word or sentence appearing is obtained based on the frequency of the target word or sentence appearing recorded in the vector representation.
[0019] Optionally, in the above method, the statistical features include information entropy, coincidence index, longest word, and number of feature signatures;
[0020] Extract statistical features of target code files, including:
[0021] The information entropy of the target code file is calculated based on the probability of the characters in the target code file appearing in the target code file;
[0022] Calculate the coincidence index of the target code file according to the number of occurrences of different characters in the target code file;
[0023] Get the longest word in the target code file;
[0024] Extract the signature features in the target code and calculate the number of signature features in the target code.
[0025] Optionally, in the above method, the first output result and the second output result are merged to obtain the detection result of the target code file, including:
[0026] Selecting the maximum value between the first output result and the second output result as the detection result of the target code file;
[0027] When the detection result is greater than or equal to a preset classification threshold, it is determined that the detection result indicates that the target code file is a Webshell file;
[0028] When the detection result is less than a preset classification threshold, it is determined that the detection result indicates that the target code file is a normal script application.
[0029] Optionally, in the above method, the first detection model and the second detection model both include multiple binary classification models;
[0030] The training method of the first detection model includes:
[0031] Obtain a first training data set, where the first training data set includes a plurality of malicious PHP scripts and a plurality of normal PHP scripts;
[0032] Extract the first sample application word frequency feature and the first sample statistical feature of the first training data set to obtain the first sample detection feature;
[0033] Using the first sample detection feature to train each binary classification model in the first detection model to adjust the parameters of each binary classification model until a plurality of trained binary classification models are obtained;
[0034] Assigning initial weight values to multiple trained binary classification models, fusing the multiple trained binary classification models to obtain a first detection model, and retraining the first detection model using the first sample detection feature to adjust the weight values of the multiple binary classification models in the first detection model until a trained first detection model is obtained;
[0035] The training method of the second detection model includes:
[0036] Obtaining a second training data set, where the second training data set includes a plurality of malicious JSP scripts and a plurality of normal JSP scripts;
[0037] Extract the second sample application word frequency feature and the second sample statistical feature of the second training data set to obtain the second sample detection feature;
[0038] Using the second sample detection feature to train each binary classification model in the second detection model to adjust the parameters of each binary classification model until a plurality of trained binary classification models are obtained;
[0039] Assign initial weight values to multiple trained binary classification models, fuse the multiple trained binary classification models to obtain a second detection model, and retrain the second detection model using the second sample detection feature to adjust the weight values of the multiple binary classification models in the second detection model until a trained second detection model is obtained.
[0040] Optionally, in the above method, the first output result is calculated based on the output results of each binary classification model in the first detection model and the weight value of the corresponding binary classification model;
[0041] The second output result is calculated based on the output results of each binary classification model in the second detection model and the weight value of the corresponding binary classification model.
[0042] In a second aspect, the present application provides a file detection device, comprising:
[0043] An acquisition module is used to acquire the target code file to be detected;
[0044] A feature extraction module is used to extract text word frequency features and statistical features of the target code file;
[0045] A first detection module, used to extract a first detection feature from the text word frequency feature and the statistical feature, and input the first detection feature into a first detection model to obtain a first output result; the first output result is used to characterize the probability that the target code file is a Webshell file in PHP format;
[0046] A second detection module is used to extract a second detection feature from the text word frequency feature and the statistical feature, and input the second detection feature into a second detection model to obtain a second output result; the second output result is used to characterize the probability that the target code file is a Webshell file in JSP format;
[0047] The result output module is used to merge the first output result and the second output result to obtain the detection result of the target code file; the detection result is used to characterize whether the target code file is a Webshell file.
[0048] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;
[0049] The memory stores computer-executable instructions;
[0050] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.
[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.
[0052] The file detection method, device, electronic device, storage medium and program product provided by the present application obtain a target code file to be detected; extract text word frequency features and statistical features of the target code file; extract a first detection feature from the text word frequency features and statistical features, and input the first detection feature into a first detection model to obtain a first output result; the first output result is used to characterize the probability that the target code file is a Webshell file in PHP format; extract a second detection feature from the text word frequency features and statistical features, and input the second detection feature into a second detection model to obtain a second output result; the second output result is used to characterize the probability that the target code file is a Webshell file in JSP format; merge the first output result and the second output result to obtain the detection result of the target code file; the detection result is used as a means to characterize whether the target code file is a Webshell file, thereby achieving the effect of improving the accuracy of Webshell file detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0054] Figure 1 Schematic diagram of the process of the file detection method provided in this application Figure 1 ;
[0055] Figure 2 Schematic diagram of the process of the file detection method provided in this application Figure 2 ;
[0056] Figure 3 A schematic diagram of the structure of the file detection device provided by this application;
[0057] Figure 4 A schematic diagram of the structure of the electronic device provided in this application.
[0058] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0060] In the specific application scenarios of this application, Advanced Persistent Threat (APT) is a widely-watched threat in the field of network security. It is a network intrusion behavior carried out by skilled professionals and complex auxiliary tools, and its characteristics are becoming more and more secretive and complex. APT attacks achieve specific purposes by analyzing targets and exploiting their vulnerabilities, and have the characteristics of continuous attacks and threats to systems. The implementation process of APT attacks includes 5 stages: reconnaissance preparation, code input, intrusion access, extended actions, and attack benefits. Webshell, as one of the commonly used APT attack weapons, is a key research area for APT attack detection. Due to the activeness and strong iteration of APT attack methods, countermeasures need to be taken to detect new and existing malicious codes.
[0061] Webshells usually exist in web page file formats such as ASP, PHP, and JSP, and are also called web backdoors or backdoor files. After invading the host, attackers usually upload backdoor files to the web server and hide them in the normal script application folder, so as to achieve the purpose of long-term and continuous control of the web server. At present, traditional antivirus software mainly uses signature matching methods based on feature codes to detect WebShells. By analyzing known WebShell code samples, extracting their text features as malicious signatures, and forming a malicious signature database, it is possible to detect and block abnormal network data flows that match its rule set in real time. This method has high detection accuracy on known webshells, but its accuracy is greatly reduced for unknown or variant WebShells. Therefore, it is of great significance to study how to effectively detect various unknown types of Webshells.
[0062] At present, there are also many studies that use machine learning algorithms (MLAs) and deep learning algorithms (DLAs) to detect unknown Webshell types, use Word2Vec to vectorize the dynamic features and static text features of source file data, and then use logistic regression, multi-layer perceptron (MLP) or random forest as basic classifiers to achieve webshell detection. Among them, most methods focus on source code level analysis, for example: some studies are based on text statistical features, using information entropy, coincidence index, longest word, compression ratio and other features to detect WebShell in file content; some methods are based on text semantics, extracting the structure and content features of the script, using lexical analysis to extract features, and identifying and detecting WebShell from a grammatical perspective.
[0063] In addition to static detection methods, many researchers have also discussed detecting traffic behavior to discover malicious webshells. For example, based on network traffic detection methods, word2vec is used to map words in HTTP requests to a vector, thereby representing web requests as a feature matrix, which is input into a convolutional neural network (CNN) to classify normal traffic and traffic during webshell runtime; there are also studies that extract features from raw sequence data from web logs and then use LSTM-based models to detect webshells.
[0064] In the prior art, malicious webshell file detection has the following problems:
[0065] 1. As time goes by and attackers continue to improve, the content of webshells may change frequently. Traditional signature-based or rule-based network intrusion detection systems are not effective in detecting and blocking the new generation of webshells equipped with encryption and obfuscation technology. In addition, webshells are usually written in high-level languages, such as PHP, ASP, JSP, etc. The regular expression-based rule model cannot fully reflect the lexical and grammatical features of high-level languages, which greatly reduces the accuracy of identifying webshells.
[0066] 2. If dynamic methods are used for context detection, a large amount of log data needs to be stored, and the memory overhead required is a huge problem. In addition, most behavior detection strategies are based on prior knowledge and have a high false alarm rate, making them difficult to apply to actual scenarios.
[0067] 3. In the webshell detection AI (Artificial Intelligence) model, the accuracy depends largely on the training data set. For webshell attacks, the number of intrusion attack samples usually contained in the training data set is far less than that of benign samples. When the proportion of white samples in the training set is too high, the performance of the model will be affected and the accuracy will decrease. This requires the use of technical mechanisms when training AI models to minimize the impact of class imbalance on the model.
[0068] The data transmission method provided in this application is intended to solve the above technical problems in the prior art.
[0069] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0070] The present application embodiment provides a file detection method, which is a static detection method for extracting and learning features of webshell code files in php and jsp formats. Figure 1 As shown, the following steps are included:
[0071] S101, obtaining a target code file to be detected.
[0072] Specifically, the target code file can be a code file to be detected that is actively uploaded by the user, or it can be a code file that meets the target conditions that is automatically screened in a terminal such as a computer by setting a screening mechanism. By testing the target code file, it can be confirmed whether it is a webshell file, and further measures can be taken based on the detection results.
[0073] S102: Extract text word frequency features and statistical features of the target code file.
[0074] Among them, the text word frequency feature is a common feature representation method in natural language processing. Generally speaking, it can be a feature vector constructed by counting the frequency of occurrence of each word in the text. Exemplarily, the text word frequency feature can be obtained by processing the target code file through the bag of words model and the word frequency inverse document frequency (TFIDF) method, representing each text as a vector, the dimension of the vector is related to the number of different words in the text, and each dimension represents a word.
[0075] Optionally, the target code file includes a plurality of target words and sentences; extracting the text word frequency feature of the target code file may include the steps of:
[0076] S1021. Perform word frequency statistics on the target code file to obtain the probability of the target words and sentences in the target code file appearing.
[0077] Specifically, the bag-of-words model may be used to process the target code file to obtain the probability of the target words and sentences appearing in the target code file.
[0078] Optionally, the following steps are included:
[0079] S10211. Convert the target code file into a string variable to extract a corresponding string list in the target code file; the string list includes multiple target words and sentences.
[0080] The path of the target code file can be stored in a variable. By loading the target code file as a string variable and traversing all script applications, a string list can be obtained. The string list includes multiple words, and multiple words can form a target phrase or sentence, collectively referred to as a target phrase. Based on actual work requirements, the target phrase can be set to contain a fixed number of consecutive words, such as 2.
[0081] S10212. Use the bag-of-words model to process the string list to convert the string list into a vector representation of the text, and obtain the probability of the target word or sentence appearing based on the frequency of the target word or sentence appearing recorded in the vector representation.
[0082] Specifically, the bag-of-words model can be used to convert a string into a vector representation of the text. The bag-of-words model is a commonly used feature extraction model in the field of natural language processing (NLP), which is characterized by ignoring vocabulary, grammar, and word order relationships. In this model, a document is regarded as a bag of independent words, in which a word-based dictionary is established, and the dictionary is used to represent the vector of the document. The model assumes that there is a document collection containing normal script application documents and Webshell files, with a total of M documents. All words are extracted from the document to build a dictionary containing N operation code words. Next, each document is represented as an N-dimensional vector, where each dimension value of the vector represents the number of times the word appears in the document. In this way, each word or token is called a "gram", and the word pairs in the vocabulary constitute a "bi-gram". This method forms a key grammar list by calculating the frequency of occurrence of all grams, where each grammar in the list represents a feature vector dimension. The basic assumption of the model is that the occurrence of the Nth word is only related to the previous N-1 words and has nothing to do with other words. The probability of the occurrence of the entire sentence is the product of the probability of each word. These probabilities can be obtained by directly calculating the number of co-occurrences of N words in the corpus. Exemplarily, a 2-gram method may be used to generate a global vocabulary based on word segmentation, and a token of the 2-gram segmentation method is set to r'\b\w+\b'.
[0083] S1022. Use the word frequency inverse document frequency method to process the probability of the target word or sentence appearing, obtain the weight vector of the target word or sentence in the target code file, and determine the weight vector as the text word frequency feature of the target code.
[0084] Among them, TF-IDF (Term Frequency-Inverse Document Frequency) is a text analysis and feature extraction technology commonly used in natural language processing and information retrieval. TF stands for term frequency, that is, the number of times a word appears in an article:
[0085]
[0086] IDF (Inverse Document Frequency) is the inverse document frequency index. If there are fewer documents containing keyword w, it means that keyword w has good category distinction ability:
[0087]
[0088] TF-IDF combines TF and IDF to measure the importance of a word in a document. It can be expressed as:
[0089] TF-IDF=TF*IDF
[0090] The above formula is used to express that the importance of a word increases in direct proportion to the number of times it appears in the document, but at the same time it decreases in inverse proportion to the frequency of its occurrence in the corpus.
[0091] The statistical features of the target code file can be various statistics or features extracted from the file to describe the content, structure or other related attributes of the document. For target code files in PHP and JSP formats, the statistical features can include static features such as the number of words, the number of different words, the number of lines, the average number of words per line, the number of null characters and spaces, the maximum word length, the number of comments, etc., and dynamic features such as operation features, sensitive function running features, and sensitive string features.
[0092] Optionally, the embodiments of the present application mainly involve four static statistical features: information entropy, overlap index, longest word, matching feature function and feature signature number of sensitive codes (ie, feature signature number).
[0093] Based on the above statistical features, statistical features of the target code file are extracted, including:
[0094] 1. According to the probability of characters in the target code file appearing in the target code file, the information entropy of the target code file is calculated.
[0095] Information entropy is Information_Entrophy (IE). If the system is ordered, the information entropy is low; conversely, if the system is disordered, the information entropy is high. Information entropy can be defined as a measure of orderliness in a system. Specifically, let p(x i ) is the character x i The probability of appearing in the file content string can be calculated using the following formula to calculate the information entropy IE of the string:
[0096]
[0097] Where N is the number of characters in the string.
[0098] 2. Calculate the coincidence index IC of the target code file according to the number of occurrences of different characters in the target code file.
[0099] If the IC value is low, it means that there may be obfuscation or encryption in the file. The coincidence index IC can be calculated using the following formula:
[0100]
[0101] Where X is a string of length n, with a total of m different characters in the string, and the number of occurrences of each character in X is f1, f2, …, f m .
[0102] 3. Get the longest word in the target code file.
[0103] The length of word strings in normal files conforms to English standards, while webshells usually use base64 for encoding and obfuscation, and then call functions for decoding to avoid detection by WAF and other tools. If the string length exceeds a certain limit, it means that the code is likely to be encoded and obfuscated, and the file is likely to be a webshell.
[0104] 4. Extract the signature features in the target code and calculate the number of signature features in the target code. Since jsp files and php files have different code syntax and formats, the malicious feature functions and code keywords they use are also different. Different signature features need to be designed for the two different models.
[0105] The following is a detailed introduction to the signature number characteristics used in the jsp model in this application:
[0106] count_java_comdshell: contains the string 'Process' or 'Runtime' or 'ProcessBuilder' or 'start' or 'Scanner' or 'getInputStream' or 'exec'.
[0107] count_jythopn_cmdshell: the number of characters containing the string 'Jython' or 'DefaultConsoleImpl' or 'PythonInterpreter';
[0108] count_java_uploadshell: contains the number of strings 'getParameter' or 'getHeader' or 'FileOutputStream' or 'RandomAccessFile' or 'File' or 'PrintWriter' or 'write' or 'print' or 'println' or 'FileHandler';
[0109] count_java_logger: the number of strings containing the string 'Logger' or 'FileHandler' or 'getParameter' or 'getHeader' or 'info' or 'warn' or 'error' or 'debug' or 'trace';
[0110] count_java_cat: contains the number of strings 'shell' or 'downloadL' or 'exec' or 'cat' or 'auto' or 'download';
[0111] count_java_chopper: the number of strings containing 'Pwd' or 'AA' or 'BB' or 'CC' or 'EC' or 'GC';
[0112] count_java_reference: contains the string 'jspspy' or 'Hacker' or 'jshell' or 'devilzShell' or 'Manage-System' or 'JspWebshell' or 'jmmm' or 'JspDo' or 'JfileMan' or 'timeshell' or 'shack' or 'JspHelper' or 'JspTqz' or 'Silic' or 'mietian' or 'kukafei';
[0113] The following is a detailed introduction to the signature number features used by the PHP model in this application:
[0114] count_exec: contains the number of strings 'assert' or 'cmd_shell' or 'eval' or 'shell_exec' or 'system';
[0115] count_file: contains the number of strings 'file_get_contents' or 'fopen' or 'fwrite' or 'readdir' or 'scandir' or 'opendir' or 'curl';
[0116] count_zip: contains the number of strings 'base64_encode' or 'base64_decode';
[0117] count_code: the number of strings containing 'gzcompress' or 'gzuncompress' or 'gzinflate' or 'gzdecode';
[0118] count_chr: the number of characters containing the string 'chr' or 'ord';
[0119] count_re: contains the number of strings 'preg_replace' or 'substr' or 'str_replace';
[0120] count_other: contains the number of strings 'create_function' or 'pack'.
[0121] S103. Extract a first detection feature from the text word frequency feature and the statistical feature, and input the first detection feature into a first detection model to obtain a first output result; the first output result is used to characterize the probability that the target code file is a Webshell file in PHP format.
[0122] Specifically, the first detection feature can be a feature vector for detecting whether the target code file is a Webshell file in PHP format. Based on the aforementioned text word frequency features and statistical features, the number of feature signatures corresponding to the PHP format file, as well as information entropy, overlap index, longest word, and text word frequency features can be selected to extract 100 features to form the first detection feature. It should be noted that this application does not limit the number of features in the first detection feature. The first detection model can be a model for detecting whether the target code is a Webshell file in PHP format.
[0123] S104. Extract a second detection feature from the text word frequency feature and the statistical feature, and input the second detection feature into a second detection model to obtain a second output result; the second output result is used to characterize the probability that the target code file is a Webshell file in JSP format.
[0124] Specifically, the second detection feature can be a feature vector for detecting whether the target code file is a Webshell file in JSP format. Based on the aforementioned text word frequency features and statistical features, the number of feature signatures corresponding to the JSP format file, as well as information entropy, overlap index, longest word, and text word frequency features can be selected to extract 100 features to form the second detection feature. It should be noted that this application does not limit the number of features in the second detection feature. The second detection model can be a model for detecting whether the target code is a Webshell file in PHP format.
[0125] S105. The first output result and the second output result are integrated to obtain a detection result of the target code file; the detection result is used to indicate whether the target code file is a Webshell file.
[0126] Specifically, the first output result and the second output result may be subjected to maximum value fusion to obtain the detection result of the target code.
[0127] Exemplarily, the following steps may be included:
[0128] S1051. Select the maximum value between the first output result and the second output result as the detection result of the target code file.
[0129] S1052: When the detection result is greater than or equal to a preset classification threshold, determine that the detection result indicates that the target code file is a Webshell file.
[0130] S1053: When the detection result is less than a preset classification threshold, determine that the detection result indicates that the target code file is a normal script application.
[0131] In the fusion stage of the classification results of the PHP and JSP models, the maximum value of the two model results is selected as the final detection score, and the classification threshold is set. The score greater than or equal to the threshold is identified as a malicious webshell, while the score less than the threshold is identified as a normal script.
[0132] The file detection method provided in the embodiment of the present application detects whether the target code file is a webshell in PHP format and a webshell in JSP format respectively, and merges the two detection results to obtain the final classification result, thereby improving the accuracy of file detection.
[0133] In a possible implementation, the first detection model and the second detection model both include multiple binary classification models.
[0134] Specifically, since the file detection method provided in the embodiment of the present application belongs to a binary classification algorithm, a classification model can be used for detection, and the number of clusters of the classifier is set to 2, that is, a binary classification model is obtained. Exemplarily, RF, XGBoost, support vector machine (SVM) and lightGBM models can be used to fuse to obtain a first detection model and a second detection model.
[0135] For these four models, you can use grid search methods (such as the GridSearchCV class in the sklearn library) to adjust the parameters of the model to optimize its performance and generalization ability.
[0136] The GridSearchCV function finds the best hyperparameter combination by traversing all possible parameter combinations in the candidate value list of the specified parameters and evaluating the effect of each combination based on the cross-validation performance.
[0137] In a possible implementation, the training method of the first detection model may include the following steps:
[0138] S201. Obtain a first training data set, where the first training data set includes a plurality of malicious PHP scripts and a plurality of normal PHP scripts.
[0139] Specifically, the first detection model is used to detect whether the target code file is a PHP script. Therefore, a plurality of malicious PHP scripts and a plurality of normal PHP scripts are used to form a first training data set. Among them, normal samples mainly come from public Web sites that provide regular services, while webshells mainly come from related projects on github, as well as common webshell samples. In an exemplary embodiment, 1,308 malicious PHP scripts and 1,580 normal PHP scripts were collected to form the first training data set. Furthermore, in the experimental design, a random sampling strategy is adopted to allocate 80% of the overall data set as a training set for model learning and optimization, and the remaining 20% of the samples are retained as an independent test set for model performance evaluation and verification.
[0140] S202: Extract the first sample application word frequency feature and the first sample statistical feature of the first training data set to obtain the first sample detection feature.
[0141] Specifically, the steps of extracting word frequency features and statistical features of each sample for each script application (i.e., sample) in the data set are consistent with the above-mentioned processing of the target code file. It includes: preprocessing the sample and representing it using a 2-gram model. Then, the TFIDF method is used to extract sample features with different vocabulary sizes. On the other hand, four static statistical features are calculated and extracted for the entire file string in different ways. Finally, 100 features are extracted from these input feature vectors and used as input to the model. These 100 features together constitute the first sample detection feature. It should be noted that this application does not limit the number of features in the sample detection feature.
[0142] S203: Use the first sample detection feature to train each binary classification model in the first detection model to adjust the parameters of each binary classification model until a plurality of trained binary classification models are obtained.
[0143] In a specific example, the application selects the optimal parameters to initialize the model, and the model parameters are set as follows:
[0144] The RF model parameters are set as: n_estimators=200, max_depth=26, random_state=66, min_samples_split=2, criterion='entropy'.
[0145] XGBoost model parameters are set to: max_depth = 5, learning_rate = 0.01, n_estimators = 125, objective = 'binary:logistic'
[0146] The LightGBM model parameters are set as: boosting_type = 'gbdt', colsample_bytree = 0.8, learning_rate = 0.05, num_leaves = 63, subsample = 0.8, random_state = 2021.
[0147] The SVM model parameters are set as: kernel='RBF', C=10.
[0148] After training based on sample features, the parameters in each classification model are adjusted until multiple trained binary classification models are obtained.
[0149] S204, assigning initial weight values to multiple trained binary classification models, fusing the multiple trained binary classification models to obtain a first detection model, and retraining the first detection model using the first sample detection feature to adjust the weight values of the multiple binary classification models in the first detection model until a trained first detection model is obtained.
[0150] Specifically, within the algorithmic framework for building the fusion system, a classifier integration technology based on the majority voting mechanism is adopted. This technology involves preprocessing the prediction outputs produced by a series of basic classifier models, which is specifically manifested as a normalization operation. Subsequently, by assigning corresponding weight coefficients to each model, their processed results are combined to obtain an integrated classifier.
[0151] Optionally, the first output result is calculated based on the output results of each binary classification model in the first detection model and the weight value of the corresponding binary classification model;
[0152] Specifically, for any particular eigenvector x i , the result score s of the ensemble classifier is calculated as follows:
[0153]
[0154] Among them, s(x i ) represents the calculation of the binary classification result scores of N basic classifiers respectively, and the selection of the label with the highest score. For each classifier result label, its score is calculated as the classifier weight w j The sum of I(.). I(.) is an indicator function, if its argument is true, then I(.) = 1, otherwise it is zero (i.e., only the weights of the base classifiers whose output label is grand-truth).
[0155] In a possible implementation, the training method of the second detection model includes the following steps:
[0156] S301. Obtain a second training data set, where the second training data set includes a plurality of malicious JSP scripts and a plurality of normal JSP scripts.
[0157] Specifically, the second detection model is used to detect whether the target code file is a JSP script. Therefore, a plurality of malicious JSP scripts and a plurality of normal JSP scripts are used to form the second training data set. Among them, normal samples mainly come from public Web sites that provide regular services, while webshells mainly come from related projects on github, as well as common webshell samples. In an exemplary embodiment, 896 malicious jsp scripts and 1463 normal jsp scripts are collected to form the second training data set. Furthermore, in the experimental design, a random sampling strategy is adopted, and 80% of the overall data set is allocated as a training set for model learning and optimization, and the remaining 20% of the samples are retained as an independent test set for model performance evaluation and verification.
[0158] S302: extract the second sample application word frequency feature and the second sample statistical feature of the second training data set to obtain the second sample detection feature.
[0159] Specifically, the steps of extracting the word frequency features and statistical features of each sample for each script application (i.e., sample) in the data set are consistent with the above-mentioned processing of the target code file. It includes: preprocessing the sample and representing it using a 2-gram model. Then, the TFIDF method is used to extract sample features with different vocabulary sizes. On the other hand, four static statistical features are calculated and extracted for the entire file string in different ways. Finally, 100 features are extracted from these input feature vectors and used as input to the model. These 100 features together constitute the second sample detection feature. It should be noted that this application does not limit the number of features in the sample detection feature.
[0160] S303: Use the second sample detection feature to train each binary classification model in the second detection model to adjust the parameters of each binary classification model until a plurality of trained binary classification models are obtained.
[0161] In a specific example, the application selects the optimal parameters to initialize the model, and the model parameters are set as follows:
[0162] The RF model parameters are set as: n_estimators=200, max_depth=26, random_state=66, min_samples_split=2, criterion='entropy'.
[0163] XGBoost model parameters are set to: max_depth = 5, learning_rate = 0.01, n_estimators = 125, objective = 'binary:logistic'
[0164] The LightGBM model parameters are set as: boosting_type = 'gbdt', colsample_bytree = 0.8, learning_rate = 0.05, num_leaves = 63, subsample = 0.8, random_state = 2021.
[0165] The SVM model parameters are set as: kernel='RBF', C=10.
[0166] After training based on sample features, the parameters in each classification model are adjusted until multiple trained binary classification models are obtained.
[0167] S304, assigning initial weight values to multiple trained binary classification models, fusing the multiple trained binary classification models to obtain a second detection model, and retraining the second detection model using the second sample detection feature to adjust the weight values of the multiple binary classification models in the second detection model until a trained second detection model is obtained.
[0168] Specifically, within the algorithmic framework for building the fusion system, a classifier integration technology based on the majority voting mechanism is adopted. This technology involves preprocessing the prediction outputs produced by a series of basic classifier models, which is specifically manifested as a normalization operation. Subsequently, by assigning corresponding weight coefficients to each model, their processed results are combined to obtain an integrated classifier.
[0169] Optionally, the second output result is calculated based on the output results of each binary classification model in the second detection model and the weight value of the corresponding binary classification model;
[0170] Specifically, for any particular eigenvector x i , the result score s of the ensemble classifier is calculated as follows:
[0171]
[0172] Among them, s(x i ) represents the calculation of the binary classification result scores of N basic classifiers respectively, and the selection of the label with the highest score. For each classifier result label, its score is calculated as the classifier weight w jThe sum of I(.). I(.) is an indicator function, if its argument is true, then I(.) = 1, otherwise it is zero (i.e., only the weights of the base classifiers whose output label is grand-truth).
[0173] The file detection method provided in the embodiment of the present application proposes a multi-feature detection method, which performs feature analysis, feature extraction and feature selection on the script application text through machine learning clustering method and static statistical method. Fusion models are constructed for php file and jsp file formats respectively, giving play to the advantages of multiple models and further improving the accuracy of detection.
[0174] In order to further illustrate the solution of the embodiment of the present application, a specific example is given below. Figure 2 As shown, the file detection method provided by this application is essentially an APT attack weapon detection method based on anomaly detection, and its technical route mainly includes four steps: webshell data set establishment, feature extraction, model training and detection result output.
[0175] 1. Establishment of webshell data set.
[0176] In the study, the inventor used PHP and JSP scripts to detect malicious webshells. A large number of normal and malicious PHP and JSP scripts were collected from multiple websites as samples. Normal samples mainly come from public Web sites that provide regular services, while webshells mainly come from related projects on github, as well as common webshell samples. Finally, in the experiment of this application, a total of 1308 malicious PHP scripts and 1580 normal PHP scripts, 896 malicious jsp scripts and 1463 normal jsp scripts were collected, totaling 5247 for further experiments.
[0177] The training and testing process of the machine learning model used in this application is as follows Figure 1 As shown. First, the dataset is divided into training samples and test samples, including normal PHP scripts, jsp scripts and webshell scripts. Subsequently, these samples are preprocessed and represented by a 2-gram model. Next, the TFIDF method is used to extract sample features with different vocabulary sizes. On the other hand, four static statistical features are calculated and extracted from the entire file string in different ways. Finally, 100 features are extracted from these input feature vectors and used as the input of the model. In terms of experimental design, a random sampling strategy is adopted to allocate 80% of the overall dataset as a training set for model learning and optimization, and the remaining 20% of the samples are retained as an independent test set for model performance evaluation and verification.
[0178] 2. Feature extraction.
[0179] The meaning of "Web shell" is to open the Web service server and obtain operation permissions in some way. Taking the execution command as an example, as long as it has sufficient permissions, Webshell can use the functions inside the script application to execute the shell command of the operating system, such as the following sentence:
[0180] <? php echo system($_GET['cmd'];?>
[0181] The webshell script code template commonly used to execute client-side delivery is as follows:
[0182] <? php eval($_POST['code'];?>
[0183] If this common Webshell wants to perform specific functions, it needs special functions and variables. These special functions are distinguishing features for detection, such as system, eval, GET, POST, etc. These character features are usually eliminated and hidden by the Webshell through encryption, conversion, callback, hidden keywords, etc., making it difficult to determine whether the script is legal.
[0184] Therefore, this application mainly extracts webshell features from two aspects. First, for scripts with clear malicious webshell text features, the word bag model is used to segment the string into words, and different words are used as a new feature, and then the TFIDF model is used; second, for webshells with encrypted ascll text, experts select and calculate its multiple statistical features, and perform feature vectorization of the script application in digital dimensions. After obtaining the two features, they are spliced as the feature input of the subsequent classifier model.
[0185] 2.1 Text Feature Extraction
[0186] First, load each PHP and JSP script application file into a string variable, traverse all script application files, and get a string list.
[0187] Subsequently, the string is converted into a vector representation of the text using the bag-of-words model. The bag-of-words model is a commonly used feature extraction model in the field of natural language processing (NLP), which is characterized by ignoring vocabulary, grammar, and word order relationships. In this model, a document is regarded as a bag of independent words, in which a word-based dictionary is established and the dictionary is used to represent the vector of the document. The model assumes that there is a document collection containing normal script application files and Webshell files, with a total of M documents. All words are extracted from the documents to build a dictionary containing N opcode words. Next, each document is represented as an N-dimensional vector, where each dimension value of the vector represents the number of times the word appears in the document. In this way, each word or token is called a "gram", and the word pairs in the vocabulary constitute a "bi-gram". This method forms a list of key grammars by calculating the frequency of occurrence of all grams, where each grammar in the list represents a feature vector dimension. The basic assumption of the model is that the occurrence of the Nth word is only related to the previous N-1 words and has nothing to do with other words. The probability of the occurrence of the entire sentence is the product of the probability of each word. These probabilities can be obtained by directly calculating the number of co-occurrences of N words in the corpus.
[0188] For the black sample data set, this application uses the 2-gram method to generate a global vocabulary based on word cutting, and sets the token of the 2-gram cutting method to r'\b\w+\b'.
[0189] Then, TF-IDF is used for further processing. TF-IDF (Term Frequency-Inverse Document Frequency) is a text analysis and feature extraction technique commonly used in natural language processing and information retrieval. TF stands for term frequency, which is the number of times a word appears in an article:
[0190]
[0191] IDF (Inverse Document Frequency) is the inverse document frequency index. If there are fewer documents containing keyword w, it means that keyword w has good category distinction ability:
[0192]
[0193] TF-IDF combines TF and IDF to measure the importance of a word in a document. The importance of a word increases proportionally with the number of times it appears in the document, but decreases inversely with the frequency of its occurrence in the corpus:
[0194] TF-IDF=TF*IDF
[0195] This application uses the TF-IDF method to process and obtain text word frequency features.
[0196] 2.2 Statistical feature extraction
[0197] It mainly involves four static statistical features: information entropy, overlap index, longest word, matching feature function and feature signature number of sensitive codes.
[0198] Information_Entrophy: Information entropy. If the system is ordered, the information entropy is low; conversely, if the system is disordered, the information entropy is high. Information entropy can be defined as a measure of orderliness in a system. Let p(x i ) is the character x i The probability of appearing in the file content string, there are N different characters, the formula is defined as follows:
[0199]
[0200] IC: coincidence index. If the IC value is low, it means that there may be obfuscation or encryption in the file. Let X be a string of length n, with a total of m different characters in the string, and the number of occurrences in X is f1, f2, ..., f m , the calculation formula of coincidence index is as follows:
[0201]
[0202] Where X is a string of length n, with a total of m different characters in the string, and the number of occurrences of each character in X is f1, f2, …, f m .
[0203] max_length: The longest word in the file. The length of the word string in a normal file conforms to English standards, while webshells usually use base64 for encoding and obfuscation, and then call functions for decoding to avoid detection by WAF and other tools. If the string length exceeds a certain limit, it means that the code is likely to be encoded and obfuscated, and the file is likely to be a webshell.
[0204] Since jsp files and php files have different code syntax and formats, the malicious feature functions and code keywords they use are also different. Different signature features need to be designed for the two different models.
[0205] The following is a detailed introduction to the signature number characteristics used in the jsp model in this application:
[0206] count_java_comdshell: The number of strings containing 'Process' or 'Runtime' or 'ProcessBuilder' or 'start' or 'Scanner' or 'getInputStream' or 'exec'
[0207] count_jythopn_cmdshell: The number of columns containing the string 'Jython' or 'DefaultConsoleImpl' or 'PythonInterpreter'
[0208] count_java_uploadshell: contains the string 'getParameter' or 'getHeader' or 'FileOutputStream' or 'RandomAccessFile' or 'File' or 'PrintWriter' or 'write' or 'print' or 'println' or 'FileHandler'
[0209] count_java_logger: The number of strings containing the string 'Logger' or 'FileHandler' or 'getParameter' or 'getHeader' or 'info' or 'warn' or 'error' or 'debug' or 'trace'
[0210] count_java_cat: The number of strings containing 'shell' or 'downloadL' or 'exec' or 'cat' or 'auto' or 'download'
[0211] count_java_chopper: The number of choppers containing the string 'Pwd' or 'AA' or 'BB' or 'CC' or 'EC' or 'GC'
[0212] count_java_reference: The number of occurrences of the string 'jspspy' or 'Hacker' or 'jshell' or 'devilzShell' or 'Manage-System' or 'JspWebshell' or 'jmmm' or 'JspDo' or 'JfileMan' or 'timeshell' or 'shack' or 'JspHelper' or 'JspTqz' or 'Silic' or 'mietian' or 'kukafei'
[0213] The following is a detailed introduction to the signature number features used by the PHP model in this application:
[0214] count_exec: The number of strings containing 'assert' or 'cmd_shell' or 'eval' or 'shell_exec' or 'system'
[0215] count_file: contains the string 'file_get_contents' or 'fopen' or 'fwrite' or 'readdir' or 'scandir' or 'opendir' or 'curl'
[0216] count_zip: contains the number of strings 'base64_encode' or 'base64_decode'
[0217] count_code: The number of characters containing the string 'gzcompress' or 'gzuncompress' or 'gzinflate' or 'gzdecode'
[0218] count_chr: contains the number of strings 'chr' or 'ord'
[0219] count_re: contains the number of strings 'preg_replace' or 'substr' or 'str_replace'
[0220] count_other: contains the number of strings 'create_function' or 'pack'
[0221] 3. Model training.
[0222] During the training process, the application adjusted the parameters of RF, XGBoost, support vector machine (SVM) and lightGBM models to achieve better performance. Since the detection problem of this application belongs to a binary classification algorithm, the number of clusters of the classifier is set to 2. For these four models, this application uses the grid search method (GridSearchCV class in the sklearn library) to adjust the parameters of the model to optimize its performance and generalization ability.
[0223] The GridSearchCV function finds the best hyperparameter combination by traversing all possible parameter combinations in the candidate value list of the specified parameter and evaluating the effect of each combination based on the cross-validation performance. Finally, the optimal parameters are selected to initialize the model. In this application, the model parameters are set as follows:
[0224] The RF model parameters are set as: n_estimators=200, max_depth=26, random_state=66, min_samples_split=2, criterion='entropy'.
[0225] XGBoost model parameters are set to: max_depth = 5, learning_rate = 0.01, n_estimators = 125, objective = 'binary:logistic'
[0226] The LightGBM model parameters are set as: boosting_type = 'gbdt', colsample_bytree = 0.8, learning_rate = 0.05, num_leaves = 63, subsample = 0.8, random_state = 2021.
[0227] The SVM model parameters are set as: kernel='RBF', C=10.
[0228] 3.5 Fusion Algorithm
[0229] Multi-model integration: In the algorithm framework of building the fusion system, the classifier integration technology based on the majority voting mechanism is adopted. This technology involves preprocessing the prediction outputs generated by a series of basic classifier models, which is specifically manifested as a normalization operation. Then, by assigning corresponding weight coefficients to each model, their processed results are combined to obtain an integrated classifier. For any specific feature vector, the result score s of the integrated classifier is calculated as follows:
[0230] For each classifier’s result label, its score is calculated as the sum of the classifier’s weights wjI(.). I(.) is an indicator function, if its argument is true, then I(.) = 1, otherwise it is zero (i.e., only the weights of the base classifiers whose output label is grand-truth are included).
[0231] 4. Maximum value fusion: In the fusion stage of the classification results of PHP and JSP models, the maximum value of the two model results is selected as the final detection score, and the classification threshold is set. The value greater than or equal to the threshold is identified as a malicious webshell, and the value less than the threshold is identified as a normal script.
[0232] Figure 3 A schematic diagram of the structure of the model training device provided in this application, such as Figure 3 As shown, the file detection device provided in this embodiment includes:
[0233] The acquisition module 201 is used to acquire the target code file to be detected;
[0234] A feature extraction module 202 is used to extract text word frequency features and statistical features of the target code file;
[0235] The first detection module 203 is used to extract a first detection feature from the text word frequency feature and the statistical feature, and input the first detection feature into a first detection model to obtain a first output result; the first output result is used to characterize the probability that the target code file is a Webshell file in PHP format;
[0236] The second detection module 204 is used to extract a second detection feature from the text word frequency feature and the statistical feature, and input the second detection feature into a second detection model to obtain a second output result; the second output result is used to characterize the probability that the target code file is a Webshell file in the JSP format;
[0237] The result output module 205 is used to merge the first output result and the second output result to obtain the detection result of the target code file; the detection result is used to indicate whether the target code file is a Webshell file.
[0238] Optionally, the target code file includes a plurality of target words and sentences;
[0239] The feature extraction module is also used to:
[0240] Perform word frequency statistics on the target code file to obtain the probability of the target words and sentences appearing in the target code file;
[0241] The word frequency inverse document frequency method is used to process the probability of the target word and sentence appearing, obtain the weight vector of the target word and sentence in the target code file, and determine the weight vector as the text word frequency feature of the target code.
[0242] Optionally, the feature extraction module is further used to:
[0243] Convert the target code file into a string variable to extract a corresponding string list in the target code file; the string list includes multiple target words and sentences;
[0244] The bag-of-words model is used to process the string list to convert it into a vector representation of the text, and the probability of the target word or sentence appearing is obtained based on the frequency of the target word or sentence appearing recorded in the vector representation.
[0245] Optionally, the statistical features include information entropy, coincidence index, longest word, and number of feature signatures;
[0246] The feature extraction module is also used to:
[0247] The information entropy of the target code file is calculated based on the probability of the characters in the target code file appearing in the target code file;
[0248] Calculate the coincidence index of the target code file according to the number of occurrences of different characters in the target code file;
[0249] Get the longest word in the target code file;
[0250] Extract the signature features in the target code and calculate the number of signature features in the target code.
[0251] Optionally, the result output module is further used to:
[0252] Selecting the maximum value between the first output result and the second output result as the detection result of the target code file;
[0253] When the detection result is greater than or equal to a preset classification threshold, it is determined that the detection result indicates that the target code file is a Webshell file;
[0254] When the detection result is less than a preset classification threshold, it is determined that the detection result indicates that the target code file is a normal script application.
[0255] Optionally, the first detection model and the second detection model both include multiple binary classification models;
[0256] The training method of the first detection model includes:
[0257] Obtain a first training data set, where the first training data set includes a plurality of malicious PHP scripts and a plurality of normal PHP scripts;
[0258] Extract the first sample application word frequency feature and the first sample statistical feature of the first training data set to obtain the first sample detection feature;
[0259] Using the first sample detection feature to train each binary classification model in the first detection model to adjust the parameters of each binary classification model until a plurality of trained binary classification models are obtained;
[0260] Assigning initial weight values to multiple trained binary classification models, fusing the multiple trained binary classification models to obtain a first detection model, and retraining the first detection model using the first sample detection feature to adjust the weight values of the multiple binary classification models in the first detection model until a trained first detection model is obtained;
[0261] The training method of the second detection model includes:
[0262] Obtaining a second training data set, where the second training data set includes a plurality of malicious JSP scripts and a plurality of normal JSP scripts;
[0263] Extract the second sample application word frequency feature and the second sample statistical feature of the second training data set to obtain the second sample detection feature;
[0264] Using the second sample detection feature to train each binary classification model in the second detection model to adjust the parameters of each binary classification model until a plurality of trained binary classification models are obtained;
[0265] Assign initial weight values to multiple trained binary classification models, fuse the multiple trained binary classification models to obtain a second detection model, and retrain the second detection model using the second sample detection feature to adjust the weight values of the multiple binary classification models in the second detection model until a trained second detection model is obtained.
[0266] Optionally, the first output result is calculated based on the output results of each binary classification model in the first detection model and the weight value of the corresponding binary classification model;
[0267] The second output result is calculated based on the output results of each binary classification model in the second detection model and the weight value of the corresponding binary classification model.
[0268] The video generating device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.
[0269] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 4 As shown, the electronic device 30 provided in this embodiment includes: at least one processor 301 and a memory 302. Optionally, as in the above method, the device 30 also includes a communication component 303. The processor 301, the memory 302 and the communication component 303 are connected via a bus.
[0270] In a specific implementation process, at least one processor 301 executes the computer-executable instructions stored in the memory 302, so that at least one processor 301 executes the above method.
[0271] The specific implementation process of the processor 301 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0272] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0273] It should be further noted that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in the present application, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0274] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0275] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.
[0276] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.
[0277] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0278] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0279] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0280] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A file detection method, characterized in that: include: Get the target code file to be tested; Extracting text word frequency features and statistical features of the target code file; Extracting a first detection feature from the text word frequency feature and the statistical feature, and inputting the first detection feature into a first detection model to obtain a first output result; the first output result is used to characterize the probability that the target code file is a Webshell file in PHP format; Extracting a second detection feature from the text word frequency feature and the statistical feature, and inputting the second detection feature into a second detection model to obtain a second output result; the second output result is used to characterize the probability that the target code file is a Webshell file in JSP format; The first output result and the second output result are integrated to obtain a detection result of the target code file; the detection result is used to characterize whether the target code file is a Webshell file.
2. The method according to claim 1, characterized in that The target code file includes a plurality of target words and sentences; The step of extracting the text word frequency feature of the target code file comprises: Perform word frequency statistics on the target code file to obtain the probability of the target words and sentences appearing in the target code file; The word frequency inverse document frequency method is used to process the probability of occurrence of the target word or phrase, obtain a weight vector of the target word or phrase in the target code file, and determine the weight vector as the text word frequency feature of the target code.
3. The method according to claim 2, characterized in that The performing word frequency statistics on the target code file to obtain the probability of the target words and sentences appearing in the target code file includes: Converting the target code file into a string variable to extract a corresponding string list in the target code file; the string list includes a plurality of target words and phrases; The string list is processed using a bag-of-words model to convert the string list into a vector representation of text, and the probability of occurrence of the target word or sentence is obtained based on the frequency of occurrence of the target word or sentence recorded in the vector representation.
4. The method according to any one of claims 1 to 3, characterized in that: The statistical features include information entropy, coincidence index, longest word, and feature signature number; The extracting statistical features of the target code file comprises: Calculating the information entropy of the target code file according to the probability of characters in the target code file appearing in the target code file; Calculating the coincidence index of the target code file according to the number of occurrences of different characters in the target code file; Obtain the longest word in the target code file; The signature features in the target code are extracted, and the number of signature features in the target code is calculated.
5. The method according to claim 1, characterized in that The fusing the first output result and the second output result to obtain the detection result of the target code file includes: Selecting the maximum value between the first output result and the second output result as the detection result of the target code file; When the detection result is greater than or equal to a preset classification threshold, determining that the detection result indicates that the target code file is a Webshell file; When the detection result is less than a preset classification threshold, it is determined that the detection result indicates that the target code file is a normal script application.
6. The method according to claim 1, characterized in that The first detection model and the second detection model both include multiple binary classification models; The training method of the first detection model includes: Obtain a first training data set, wherein the first training data set includes a plurality of malicious PHP scripts and a plurality of normal PHP scripts; Extracting a first sample application word frequency feature and a first sample statistical feature of the first training data set to obtain a first sample detection feature; Using the first sample detection feature to train each binary classification model in the first detection model to adjust the parameters of each binary classification model until a plurality of trained binary classification models are obtained; Assigning initial weight values to the multiple trained binary classification models, fusing the multiple trained binary classification models to obtain the first detection model, and retraining the first detection model using the first sample detection feature to adjust the weight values of the multiple binary classification models in the first detection model until the trained first detection model is obtained; The training method of the second detection model includes: Acquire a second training data set, where the second training data set includes a plurality of malicious JSP scripts and a plurality of normal JSP scripts; Extract the second sample application word frequency feature and the second sample statistical feature of the second training data set to obtain the second sample detection feature; Using the second sample detection feature to train each binary classification model in the second detection model to adjust the parameters of each binary classification model until a plurality of trained binary classification models are obtained; Assign initial weight values to the multiple trained binary classification models, fuse the multiple trained binary classification models to obtain the second detection model, and re-train the second detection model using the second sample detection feature to adjust the weight values of the multiple binary classification models in the second detection model until the trained second detection model is obtained.
7. The method according to claim 6, characterized in that: The first output result is calculated based on the output results of each binary classification model in the first detection model and the corresponding weight value of the binary classification model; The second output result is calculated based on the output results of each binary classification model in the second detection model and the corresponding weight value of the binary classification model.
8. A file detection device, characterized in that: include: An acquisition module is used to acquire the target code file to be detected; A feature extraction module, used to extract text word frequency features and statistical features of the target code file; A first detection module, used for extracting a first detection feature from the text word frequency feature and the statistical feature, and inputting the first detection feature into a first detection model to obtain a first output result; the first output result is used to characterize the probability that the target code file is a Webshell file in PHP format; A second detection module is used to extract a second detection feature from the text word frequency feature and the statistical feature, and input the second detection feature into a second detection model to obtain a second output result; the second output result is used to characterize the probability that the target code file is a Webshell file in JSP format; A result output module is used to merge the first output result and the second output result to obtain a detection result of the target code file; the detection result is used to characterize whether the target code file is a Webshell file.
9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.
Citation Information
Patent Citations
WebShell detection method and device and storage medium
CN109743311A
Method and device for identifying script files
CN110427755A
Webshell file detection method and system
CN113761534A
WebShell detection method, electronic device and computer readable storage medium
CN114117430A
File classification detection method and device, electronic equipment and readable storage medium
CN117332089A
