[0031] The solution of the embodiment of the present invention is mainly: use a learning set composed of a predetermined malicious file and a normal file to generate a machine learning model; read files to be detected outside the learning set, and convert the files to be detected into vectors, through machine learning The model performs malicious file identification on the files to be detected that are converted into vectors, and uses the characteristics of timely response and fast processing speed of machine learning to improve the detection efficiency of malicious files.
[0032] The malicious file in the present invention may be a virus file or other malicious files. The following embodiment uses malicious files as examples. Among them, the technical terms involved include:
[0033] Black file: virus file
[0034] Black vector: the vector transformed into the virus file
[0035] White file: normal non-virus file
[0036] White vector: the vector converted from normal non-virus files
[0037] SVM: Support Vector Machine
[0038] PE file: an executable file format under windows system
[0039] Such as figure 1 As shown, a preferred embodiment of the present invention provides a method for identifying malicious files, including:
[0040] Step S101, generating a machine learning model using a learning set consisting of a predetermined malicious file and a normal file;
[0041] Taking the windows system as an example, in order to check the files under the windows system for viruses, this embodiment first uses known virus files and non-virus files (that is, the malicious files and normal files referred to in this embodiment) to generate a machine learning model, so that The machine learning model is used to identify viruses in the files under the windows system, and solve the problem of classification of virus files and normal files in the system.
[0042] The above known virus files and non-virus files can be pre-collected by virus analysts and form a learning set. After feature extraction, dimensionality merging and screening of each virus file and normal file in the learning set, the classifier will analyze each Virus files and normal files undergo vector learning, and finally a machine learning model is generated.
[0043] Specifically, first, the malicious files and the normal files in the learning set are respectively converted into vectors, that is, the malicious files and the normal files in the learning set are respectively extracted from the effective sample features.
[0044] For an executable file (PE file), features that are helpful for virus identification include: character strings, instruction sequences, function procedures, import and export functions, and attributes of each segment.
[0045] In this embodiment, these feature keys and the feature value value form a (key: value) pair. A file (including malicious files and normal files) becomes a (key: value) set. If each key is As a dimension, the set of (key:value) of a file can be regarded as a multidimensional vector with an unfixed dimension.
[0046] Through feature extraction, the file is transformed into a multidimensional vector with an unfixed dimension. However, the classifier that generates the machine learning model requires a vector with a fixed dimension, and the way to fix the dimension is to merge the dimensions (key) of all files, and if a single file does not have a certain dimension, set its value to 0 ; For massive files, there are massive dimensions, and there will be dimensional disasters. Therefore, these dimensions need to be merged and filtered; finally, the merged and filtered vectors are learned through the classifier to generate a machine learning model.
[0047] Step S102, reading files to be detected outside the learning set;
[0048] Step S103: Convert the file to be detected into a vector;
[0049] In step S104, the malicious file identification is performed on the file to be detected converted into the vector through the machine learning model.
[0050] In the above steps S102 to S104, when there is a file outside the learning set that needs to be detected, the file to be detected is read, the file to be detected is converted into a vector, and the machine learning model generated in step S101 is used for malicious File recognition.
[0051] As a preferred embodiment, taking a PC as an example, the machine learning model generated in step S101 can be applied to the virus-checking engine at the front end of the PC, and the virus-checking is performed on the user's PC. The specific implementation process is as follows:
[0052] 1. Read the file to be tested on the PC;
[0053] 2. Convert the file to be detected on the read PC into a vector.
[0054] As mentioned above, the classifier performs vector learning on each virus file and normal file in the learning set, thereby generating a machine learning model, that is, the file object processed by the machine learning model should be a vector. Therefore, in this embodiment, after reading When the file to be detected in the PC system, the file to be detected needs to be converted into a vector, that is, effective sample features are extracted from the file to be detected. The valid sample features include: character strings, instruction sequences, function procedures, import and export functions, and attributes of each segment.
[0055] Then, these feature keys and the feature value value are formed into a (key: value) pair. A file (including malicious files and normal files) becomes a (key: value) set. If each key is regarded as For one dimension, the set of (key: value) of a file can be regarded as a multidimensional vector with an unfixed dimension.
[0056] 3. Use the machine learning model to identify malicious files on the files to be detected on the PC converted into vectors.
[0057] The files to be detected on the PC converted into vectors are placed in the machine learning model for judgment, and virus files and normal files are identified from it. Specifically, the machine learning model is used to perform linear function calculation on the file to be detected after it is converted into a vector; the attributes of the malicious file and the normal file are judged according to the calculation result, and the malicious file and the normal file in the file to be detected are output.
[0058] Specifically, such as figure 2 As shown, the step S101 of generating a machine learning model using a learning set composed of a predetermined malicious file and a normal file includes:
[0059] Step S1011: Convert the malicious files and normal files in the learning set into vectors respectively;
[0060] The malicious files and normal files in the learning set are respectively converted into vectors, that is, the malicious files and normal files in the learning set are respectively extracted from the effective sample features.
[0061] For an executable file (PE file), features that are helpful for virus identification include: character strings, instruction sequences, function procedures, import and export functions, and attributes of each segment.
[0062] In this embodiment, these feature keys and the feature value value form a (key: value) pair. A file (including malicious files and normal files) becomes a (key: value) set. If each key is As a dimension, the set of (key:value) of a file can be regarded as a multidimensional vector with an unfixed dimension.
[0063] Step S1012, merging and filtering the vectors of malicious files and normal files in the learning set;
[0064] Through feature extraction, the file is transformed into a multidimensional vector with an unfixed dimension. However, the classifier that generates the machine learning model requires a vector with a fixed dimension, and the way to fix the dimension is to merge the dimensions (key) of all files, and if a single file does not have a certain dimension, set its value to 0 ; For a large number of files, there are a large number of dimensions, and there will be a dimensional disaster. Therefore, these dimensions need to be merged and filtered.
[0065] This embodiment specifically merges and filters dimensions to obtain K dimensions, where K dimension refers to the first K dimensions selected from multiple dimensions after merging and filtering according to certain rules. The follow-up will be combined image 3 Elaborate in detail.
[0066] In step S1013, the merged and filtered vectors are learned through the classifier to generate a machine learning model.
[0067] The classifier in this embodiment can be a linear classifier, the so-called linear SVM means that its kernel function is an inner product function. This embodiment specifically adopts a support vector machine (SVM). SVM is a trainable learning machine and belongs to a generalized linear classifier. The characteristics of this classifier are: it can minimize empirical errors and maximize set edges. Area. Applying SVM to virus identification is to solve the classification problem of virus files and normal files.
[0068] SVM learns the merged and filtered vectors, that is, generates a machine learning model.
[0069] Of course, in other embodiments, other machine learning methods can also be used for discrimination without using SVM.
[0070] More specifically, such as image 3 As shown, if the vectors of all malicious files in the learning set are set as black vector sets and the vectors of all normal files are white vector sets, the above step S1012 performs dimensional merging and filtering on the vectors of malicious files and normal files in the learning set The steps include:
[0071] Step S10, two black vectors are randomly selected from the black vector set, and the common dimension of the two black vectors is extracted as a black dimension set; two white vectors are randomly selected from the white vector set, and the common dimension of the two white vectors is extracted as white Dimension set
[0072] Step S11, removing all the dimensions appearing in the white dimension set in the black dimension set to form a new black dimension set, and assigning weight to each dimension in the white dimension set and the new black dimension set;
[0073] In the above step S10 and step S11, in order to merge and filter out the K dimensions, the following methods are used:
[0074] Combine the entire black vector set and white vector set and filter the dimension problem, and split it into sub-problems of two black vectors and two white vectors; then solve each sub-problem and extract the common dimension of the two white vectors (take the intersection) as In the white dimension set of the sub-problem, the two black vectors extract the common dimension as the black dimension set of the sub-problem, and remove all the dimensions that appear in the white dimension set in the black dimension set, and assign weight to each selected black and white dimension.
[0075] Step S12, combining the white dimension set and the new black dimension set separately according to the weights, and discarding the dimensions whose combined weight is lower than a predetermined weight threshold;
[0076] The solutions of all sub-problems are merged according to the dimensions, and a weight threshold w is set during the merging process. If the weight of the merged dimension (the weight values corresponding to the dimensions are added during the merge) is lower than w, the dimension is directly discarded to prevent the appearance of the dimension Set unlimited growth.
[0077] Step S13, respectively determine whether all the vectors in the black vector set and white vector set have been processed; if so; go to step S14; otherwise, return to step S10;
[0078] Step S14, filter the merged black dimension set with the merged white dimension set;
[0079] Step S15, sort the filtered black dimension set according to the weight, and take the black dimension of the top K with the highest ranking as the final dimension;
[0080] In the above step S13-step S15, when all the vectors in the black vector set and the white vector set have been learned, use the merged white dimension set to filter the black dimension set (ie black dimension set = black dimension set-white dimension set), The dimension set is ranked according to the weight, and the black dimension of the top K dimensions with the highest ranking is taken as the result.
[0081] Step S16: Convert all vectors in the black vector set and white vector set into K-dimensional vectors.
[0082] Convert all the vectors in the black-and-white file into the standard form of the selected K-dimensional vector, so that the SVM can learn the K-dimensional vector to generate a machine learning model.
[0083] The process of merging and screening the vectors of all virus files and normal files in the learning set is described in detail below with specific examples.
[0084] Such as Figure 4 As shown, FB and FW respectively represent the total set of black and white vectors, FBL and FWL respectively represent the common dimension set of black and white vectors, and B1 and B2 respectively represent the marks of two black vectors randomly selected from the black vector set. W1 and W2 respectively represent the marks of the two white vectors randomly selected from the white vector set. The process of merging and filtering the vectors of all virus files and normal files in the learning set is specifically as follows:
[0085] S1, initialize FB and FW, select the black and white vector set; if the black vector is selected, go to step S2, if the white vector is selected, go to step S3;
[0086] S2, judge whether all black vectors in the black vector set are marked; if yes, go to step S4; otherwise, go to step S21;
[0087] S21, randomly select two black vectors B1 and B2;
[0088] S22, extract the shared dimension set FBL and assign a weight to each dimension; enter S23;
[0089] S3, judge whether all the white vectors in the white vector set are marked; if yes, go to step S4; otherwise, go to step S31;
[0090] S31, randomly select two white vectors W1 and W2;
[0091] S32, extract the shared dimension set FWL and assign a weight to each dimension; enter S23;
[0092] S23, use FBL and FWL as the difference set as a new FBL;
[0093] S24, merge the new FBL and FWL into the total sets FB and FW respectively, and add the weight sets when merging;
[0094] S25: Eliminate dimensions in FB and FW whose weights are less than w-limit (the set weight threshold); return to steps S2 and S3 respectively.
[0095] S4, FB and FW make difference set as the new FB;
[0096] S5: Take out the first K dimensions of FB according to the weight order, and obtain the final result of FB.
[0097] In this embodiment, a machine learning model is generated from a learning set composed of a predetermined malicious file and a normal file, and the generated machine learning model is used to identify malicious files outside the learning set to be detected, that is, the machine automatically extracts malicious files such as viruses. Code features, without the participation of analysts, and the machine learning response is timely, can accurately and effectively extract virus features, and any malicious files found can be processed immediately, which greatly improves the efficiency of malicious file detection.
[0098] Such as Figure 5 As shown, a preferred embodiment of the present invention provides a malicious file identification device, including: a model generation module 501, a reading module 502, a vector conversion module 503, and an identification module 504, wherein:
[0099] The model generation module 501 is configured to generate a machine learning model using a learning set composed of predetermined malicious files and normal files;
[0100] The reading module 502 is used to read files to be detected outside the learning set;
[0101] The vector conversion module 503 is used to convert the file to be detected into a vector;
[0102] The recognition module 504 is configured to perform malicious file recognition on the files to be detected converted into vectors through the machine learning model.
[0103] Taking the windows system as an example, in order to check the files under the windows system for viruses, this embodiment first uses known virus files and non-virus files (that is, the malicious files and normal files referred to in this embodiment) to generate a machine learning model, so that The machine learning model is used to identify viruses in the files under the windows system, and solve the problem of classification of virus files and normal files in the system.
[0104] The above known virus files and non-virus files can be pre-collected by virus analysts and form a learning set. After the model generation module 501 performs feature extraction, dimensional merging and screening of each virus file and normal file in the learning set, the classifier Perform vector learning on each virus file and normal file in the learning set, and finally generate a machine learning model.
[0105] Specifically, first, the malicious files and the normal files in the learning set are respectively converted into vectors, that is, the malicious files and the normal files in the learning set are respectively extracted from the effective sample features.
[0106] For an executable file (PE file), features that are helpful for virus identification include: character strings, instruction sequences, function procedures, import and export functions, and attributes of each segment.
[0107] In this embodiment, these feature keys and the feature value value form a (key: value) pair. A file (including malicious files and normal files) becomes a (key: value) set. If each key is As a dimension, the set of (key:value) of a file can be regarded as a multidimensional vector with an unfixed dimension.
[0108] Through feature extraction, the file is transformed into a multidimensional vector with an unfixed dimension. However, the classifier that generates the machine learning model requires a vector with a fixed dimension, and the way to fix the dimension is to merge the dimensions (key) of all files, and if a single file does not have a certain dimension, set its value to 0 ; For massive files, there are massive dimensions, and there will be dimensional disasters. Therefore, these dimensions need to be merged and filtered; finally, the merged and filtered vectors are learned through the classifier to generate a machine learning model.
[0109] When there is a file that needs to be detected outside the learning set, the file to be detected is read, the file to be detected is converted into a vector, and the machine learning model generated in step S101 is used to identify malicious files on the file to be detected converted into the vector.
[0110] As a preferred embodiment, taking a PC as an example, the machine learning model generated by the model generation module 501 can be applied to the virus detection engine on the front end of the PC, and the virus detection is performed on the user's PC. The specific implementation process is as follows:
[0111] 1. Read the file to be tested on the PC;
[0112] 2. Convert the file to be detected on the read PC into a vector.
[0113] As mentioned above, the classifier performs vector learning on each virus file and normal file in the learning set, thereby generating a machine learning model, that is, the file object processed by the machine learning model should be a vector. Therefore, in this embodiment, in the reading module When 502 reads the file to be detected in the PC system, the vector conversion module 503 needs to convert the file to be detected into a vector, that is, extract valid sample features from the file to be detected. The valid sample features include: character strings, instruction sequences, function procedures, import and export functions, and attributes of each segment.
[0114] Then, these feature keys and the feature value value are formed into a (key: value) pair. A file (including malicious files and normal files) becomes a (key: value) set. If each key is regarded as For one dimension, the set of (key: value) of a file can be regarded as a multidimensional vector with an unfixed dimension.
[0115] 3. Use the machine learning model to identify malicious files on the files to be detected on the PC converted into vectors.
[0116] The recognition module 504 puts the files to be detected on the PC converted into vectors into the machine learning model for judgment, and identifies virus files and normal files from it. Specifically, the machine learning model is used to perform linear function calculation on the file to be detected after it is converted into a vector; the attributes of the malicious file and the normal file are judged according to the calculation result, and the malicious file and the normal file in the file to be detected are output.
[0117] Specifically, such as Image 6 As shown, the model generation module 501 includes a vector conversion unit 5011, a merging and screening unit 5012, and a generation unit 5013, wherein:
[0118] The vector conversion unit 5011 is configured to convert the malicious files and normal files in the learning set into vectors respectively;
[0119] The merging and screening unit 5012 is configured to perform dimensional merging and screening on the vectors of malicious files and normal files in the learning set;
[0120] The generating unit 5013 is configured to learn the merged and filtered vectors through the classifier to generate a machine learning model.
[0121] In this embodiment, the malicious file and the normal file in the learning set are respectively converted into vectors, that is, the malicious file and the normal file in the learning set are respectively extracted from the effective sample features.
[0122] For an executable file (PE file), features that are helpful for virus identification include: character strings, instruction sequences, function procedures, import and export functions, and attributes of each segment.
[0123] In this embodiment, these feature keys and the feature value value form a (key: value) pair. A file (including malicious files and normal files) becomes a (key: value) set. If each key is As a dimension, the set of (key:value) of a file can be regarded as a multidimensional vector with an unfixed dimension.
[0124] Through feature extraction, the file is transformed into a multidimensional vector with an unfixed dimension. However, the classifier that generates the machine learning model requires a vector with a fixed dimension, and the way to fix the dimension is to merge the dimensions (key) of all files, and if a single file does not have a certain dimension, set its value to 0 ; For a large number of files, there are a large number of dimensions, and there will be a dimensional disaster. Therefore, these dimensions need to be merged and filtered. This embodiment specifically merges and filters dimensions to obtain K dimensions, where K dimension refers to the first K dimensions selected from multiple dimensions after merging and filtering according to certain rules.
[0125] The classifier in this embodiment can be a linear classifier, the so-called linear SVM means that its kernel function is an inner product function. This embodiment specifically adopts a support vector machine (SVM). SVM is a trainable learning machine and belongs to a generalized linear classifier. The characteristics of this classifier are: it can minimize empirical errors and maximize set edges. Area. Applying SVM to virus identification is to solve the classification problem of virus files and normal files.
[0126] SVM learns the merged and filtered vectors, that is, generates a machine learning model.
[0127] Of course, in other embodiments, other machine learning methods can also be used for discrimination without using SVM.
[0128] More specifically, such as Figure 7 As shown, if it is assumed that the vectors of all malicious files in the learning set are black vector sets and the vectors of all normal files are white vector sets, the merging and filtering unit 5012 includes: a first extraction subunit 50121, a filtering subunit The unit 50122, the merging subunit 50123, the filtering subunit 50124, the second extraction subunit 50125, and the transformation subunit 50126, wherein:
[0129] The first extraction subunit 50121 is configured to randomly select two black vectors from the black vector set, and extract the common dimensions of the two black vectors as a black dimension set; randomly select two white vectors from the white vector set, and extract two white vectors The common dimensions of as the white dimension set;
[0130] The screening subunit 50122 is used to remove all the dimensions that appear in the white dimension set in the black dimension set to form a new black dimension set, and assign weights to each dimension in the white dimension set and the new black dimension set ;
[0131] The merging subunit 50123 is configured to merge the white dimension set and the new black dimension set separately according to the weight, and discard the dimensions whose weight is lower than a predetermined weight threshold after the merge;
[0132] The filtering subunit 50124 is configured to filter the merged black dimension set with the merged white dimension set after all the vectors in the black vector set and the white vector set are processed;
[0133] The second extraction subunit 50125 is used to sort the filtered black dimension set according to the weight size, and take the black dimension of the top K dimension with the highest ranking as the final dimension;
[0134] The transformation subunit 50126 is used to transform all the vectors in the black vector set and the white vector set into K-dimensional vectors.
[0135] In this embodiment, in order to merge and filter out K dimensions, the following methods are used:
[0136] Combine the entire black vector set and white vector set and filter the dimension problem, and split it into sub-problems of two black vectors and two white vectors; then solve each sub-problem and extract the common dimension of the two white vectors (take the intersection) as In the white dimension set of the sub-problem, the two black vectors extract the common dimension as the black dimension set of the sub-problem, and remove all the dimensions that appear in the white dimension set in the black dimension set, and assign weight to each selected black and white dimension.
[0137] The solutions of all sub-problems are merged according to the dimensions, and a weight threshold w is set during the merging process. If the weight of the merged dimension (the weight values corresponding to the dimensions are added during the merge) is lower than w, the dimension is directly discarded to prevent the appearance of the dimension Set unlimited growth.
[0138] When all vectors in the black vector set and white vector set have been learned, use the merged white dimension set to filter the black dimension set (ie, black dimension set = black dimension set-white dimension set), and rank the black dimension set according to the weight. Take the black dimension of the top K dimension with the highest ranking as the result.
[0139] Convert all the vectors in the black-and-white file into the standard form of the selected K-dimensional vector, so that the SVM can learn the K-dimensional vector to generate a machine learning model.
[0140] In addition, such as Figure 8 As shown, the aforementioned identification module 504 includes: a calculation unit 5041 and an output unit 5042, wherein:
[0141] The calculation unit 5041 is configured to obtain a calculation result through a machine learning model for the file to be detected after being converted into a vector;
[0142] The output unit 5042 is configured to output the malicious file and the normal file in the file to be detected according to the calculation result.
[0143] The malicious file identification method, device and storage medium in the embodiments of the present invention generate a machine learning model through a learning set composed of a predetermined malicious file and a normal file, and perform malicious actions on files to be detected outside the learning set through the generated machine learning model. File identification, that is, the machine automatically extracts the characteristics of malicious code such as viruses, eliminating the need for analysts to participate, and the machine learning response is timely, it can accurately and effectively extract virus characteristics, and any malicious files found can be processed immediately. Greatly improve the detection efficiency of malicious files.
[0144] In addition, the present invention also provides a computer-readable storage medium on which a program that enables the computer to run is stored. After the program is loaded into the computer's memory, a learning set composed of predetermined malicious files and normal files is used. Generate a machine learning model; read files to be detected outside the learning set; convert the files to be detected into vectors; perform malicious file identification on the files to be detected that are converted into vectors through the machine learning model.
[0145] It should be noted that the above-mentioned embodiments of the present invention are all illustrated by the windows operating system, but are not limited to the windows operating system. Other operating systems can also perform malicious file detection and identification with reference to the above scheme of the present invention, such as mac or linux systems. , Its specific principles will not be repeated here.
[0146] The above are only the preferred embodiments of the present invention, and do not limit the scope of the present invention. Any equivalent structure or process transformation made by using the content of the description and drawings of the present invention, or directly or indirectly applied to other related technical fields , The same reason is included in the scope of patent protection of the present invention.
