A code detection method, device and electronic equipment

Abstract

The application provides a code detection method and device and electronic equipment. In the application, data cleaning is performed on target data to obtain first data, each word in the first data is compared with words in a preset word library to determine a feature value corresponding to each word in the first data, the feature values corresponding to each word in the first data are combined according to a word arrangement order to obtain a feature matrix corresponding to the target data, a preset code detection model is called to process the feature matrix to obtain a vulnerability detection result of the target data; the preset code detection model is trained based on training samples; the training samples include the vulnerability types of vulnerability code samples and the feature matrix. In the application, the preset code detection model is trained based on a large number of training samples, so that the target data to be detected is analyzed by the preset code detection model obtained by training, the accuracy of vulnerability detection is improved, and the information security requirement is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and more particularly, to a code detection method and device and electronic equipment. BACKGROUND

[0002] In recent years, information security incidents caused by software have emerged in an endless stream. Software information security mainly refers to software vulnerabilities, and the main reason for software vulnerabilities is the high complexity of software. With the dramatic increase in the number of software, software vulnerabilities have become more and more frequent.

[0003] At present, when detecting software vulnerabilities, manual detection based on experience is generally used, which has low detection accuracy and cannot meet the information security requirements. SUMMARY

[0004] Therefore, the present application provides a code detection method, device and electronic equipment to solve the problem of low detection accuracy of manual detection of software vulnerabilities based on experience.

[0005] To solve the above technical problems, the present application adopts the following technical solutions:

[0006] A code detection method comprises:

[0007] Obtaining target data to be detected, and performing data cleaning on the target data to obtain first data;

[0008] Comparing each word in the first data with the words in the preset word library to determine the feature value corresponding to each word in the first data;

[0009] Combining the feature values corresponding to each word in the first data according to the word arrangement order to obtain a feature matrix corresponding to the target data;

[0010] Calling a preset code detection model to process the feature matrix to obtain a vulnerability detection result of the target data; the preset code detection model is trained based on training samples; the training samples include the vulnerability types of vulnerability code samples and feature matrices.

[0011] Optionally, the generation process of the preset code detection model comprises:

[0012] Obtaining code samples and code repair samples corresponding to the code samples through a preset data acquisition method;

[0013] Performing difference analysis on the code samples and the code repair samples to obtain vulnerability code samples corresponding to the code samples and existing in the code samples but not existing in the code repair samples;

[0014] determine a patch code sample corresponding to the vulnerability code sample from the code repair sample;

[0015] determine a feature matrix and a vulnerability type corresponding to the vulnerability code sample, and determine a feature matrix and a vulnerability type corresponding to the patch code sample;

[0016] train a preset code detection model using the feature matrix and the vulnerability type corresponding to the vulnerability code sample and the patch code sample respectively, and stop training when a training stop condition is met.

[0017] Optionally, determining the feature matrix and the vulnerability type corresponding to the vulnerability code sample comprises:

[0018] combine each word in the vulnerability code sample to obtain a preset word library;

[0019] perform position comparison between each word in the vulnerability code sample and a word in the preset word library to determine a feature value corresponding to each word in the vulnerability code sample;

[0020] combine the feature values corresponding to each word in the vulnerability code sample in a word arrangement order to obtain an initial feature matrix corresponding to the vulnerability code sample;

[0021] perform cluster analysis on the initial feature matrix corresponding to the vulnerability code sample to obtain a cluster result and a feature matrix corresponding to the vulnerability code sample;

[0022] obtain a vulnerability type corresponding to the cluster result, and use the vulnerability type corresponding to the cluster result as a vulnerability type corresponding to the vulnerability code sample in the cluster result.

[0023] Optionally, performing difference analysis on the code sample and the code repair sample to obtain a vulnerability code sample corresponding to the code sample and existing in the code sample but not existing in the code repair sample comprises:

[0024] perform difference analysis on the code sample and the code repair sample, and label the code sample based on a difference analysis result to obtain a labeling result;

[0025] filter out an identifier in the labeling result representing existence in the code sample but non-existence in the code repair sample, and use a code part corresponding to the identifier in the code sample as a vulnerability code sample.

[0026] Optionally, performing position comparison between each word in the first data and a word in the preset word library to determine a feature value corresponding to each word in the first data comprises:

[0027] determine a position of each word in the first data in a preset word library;

[0028] construct initial feature information corresponding to each word in the first data, and set an identifier corresponding to the position in the initial feature information as a first numerical value, and set an identifier not corresponding to the position as a second numerical value;

[0029] In a case where the number of words in the first data is less than a preset threshold, perform a data supplement operation on the initial feature information, so that the data amount of the initial feature information is the preset threshold;

[0030] use the initial feature information corresponding to each word in the first data after the data supplement operation as the feature value corresponding to the word.

[0031] A code detection device comprises:

[0032] A data processing module is configured to obtain target data to be subjected to code detection, and perform a data cleaning operation on the target data to obtain first data;

[0033] A feature determination module is configured to compare each word in the first data with a word in a preset word library in terms of position, so as to determine a feature value corresponding to each word in the first data;

[0034] A matrix determination module is configured to combine the feature value corresponding to each word in the first data according to a word arrangement order, so as to obtain a feature matrix corresponding to the target data;

[0035] A vulnerability detection module is configured to call a preset code detection model to process the feature matrix, so as to obtain a vulnerability detection result of the target data; the preset code detection model is obtained based on training samples; the training samples comprise a vulnerability type of a vulnerability code sample and a feature matrix.

[0036] Optionally, the device further comprises a model generation module, which comprises:

[0037] A sample acquisition submodule is configured to acquire a code sample and a code repair sample corresponding to the code sample by using a preset data acquisition manner;

[0038] A sample analysis submodule is configured to perform a difference analysis on the code sample and the code repair sample, so as to obtain a vulnerability code sample corresponding to the code sample and existing in the code sample but not existing in the code repair sample;

[0039] A sample determination submodule is configured to determine a repair code sample corresponding to the vulnerability code sample from the code repair sample;

[0040] a vulnerability determining submodule, configured to determine a feature matrix corresponding to the vulnerability code sample and a vulnerability type, and determine a feature matrix corresponding to the patch code sample and a vulnerability type;

[0041] a model training submodule, configured to train a preset code detection model using the feature matrix and the vulnerability type corresponding to the vulnerability code sample and the patch code sample respectively, and stop training when a training stop condition is met.

[0042] Optionally, the vulnerability determining submodule comprises:

[0043] a first combination unit, configured to combine each word in the vulnerability code sample to obtain a preset word library;

[0044] a comparison unit, configured to compare each word in the vulnerability code sample with a word in the preset word library in position to determine a feature value corresponding to each word in the vulnerability code sample;

[0045] a second combination unit, configured to combine the feature value corresponding to each word in the vulnerability code sample according to a word arrangement order to obtain an initial feature matrix corresponding to the vulnerability code sample;

[0046] a clustering unit, configured to perform clustering analysis on the initial feature matrix corresponding to the vulnerability code sample to obtain a clustering result and a feature matrix corresponding to the vulnerability code sample;

[0047] a type determining unit, configured to obtain a vulnerability type corresponding to the clustering result, and take the vulnerability type corresponding to the clustering result as a vulnerability type corresponding to a vulnerability code sample in the clustering result.

[0048] Optionally, the sample analyzing submodule is specifically configured to:

[0049] perform difference analysis on the code sample and the code repair sample, and label the code sample based on a difference analysis result to obtain a labeling result, filter out an identifier in the labeling result representing that the identifier exists in the code sample but does not exist in the code repair sample, and take a code part corresponding to the identifier in the code sample as a vulnerability code sample.

[0050] An electronic device comprises a memory and a processor.

[0051] The memory is configured to store a program.

[0052] The processor invokes the program and is configured to execute the code detection method.

[0053] Compared with the prior art, the present application has the following beneficial effects:

[0054] The application provides a code detection method, device and electronic equipment, in the application, target data to be subjected to code detection is acquired, and a data cleaning operation is performed on the target data to obtain first data, each word in the first data is subjected to position comparison with words in a preset word library to determine a feature value corresponding to each word in the first data, the feature values corresponding to each word in the first data are combined according to a word arrangement order to obtain a feature matrix corresponding to the target data, a preset code detection model is called to process the feature matrix to obtain a vulnerability detection result of the target data, the preset code detection model is obtained based on training samples, and the training samples include a vulnerability type and a feature matrix of a vulnerability code sample. In the application, the preset code detection model is obtained based on a large number of training samples, the target data to be subjected to code detection is analyzed by using the preset code detection model obtained through training, the accuracy of vulnerability detection is improved, and the information security requirement is met. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only are the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0056] Figure 1 A method flowchart of a code detection method provided for the embodiments of the present application;

[0057] Figure 2 A method flowchart of another code detection method provided for the embodiments of the present application;

[0058] Figure 3 A method flowchart of still another code detection method provided for the embodiments of the present application;

[0059] Figure 4 A method flowchart of yet another code detection method provided for the embodiments of the present application;

[0060] Figure 5 A structure schematic diagram of a preset code detection model provided for the embodiments of the present application;

[0061] Figure 6 A structure schematic diagram of a code detection device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0062] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0063] In recent years, software-induced information security incidents have emerged in an endless stream, which has also attracted people's attention to the protection of information infrastructure. The main reason for software vulnerabilities is the high complexity of software. Although some methods for detecting and analyzing software vulnerabilities have been proposed, software vulnerabilities have not been eliminated, and with the dramatic increase in the number of software, vulnerabilities are appearing more and more frequently.

[0064] Currently, the vulnerability code detection technology mainly has the following two problems:

[0065] 1. The code checking speed is limited, and it is difficult to realize automatic and batch detection, and it is difficult to keep up with the current surge in the number of codes in the Internet and software industry.

[0066] 2. The code checking coupling degree is high, and the whole process from the discovery, verification to the utilization of the vulnerability mainly depends on the experience of security engineers, and the coupling degree of the whole process is high, and the vulnerability mining cannot realize modularization, which is also one of the factors restricting the code checking speed.

[0067] In addition, the existing static detection method and some commercial tools also have room for improvement in the accuracy of vulnerability detection and false positive rate. Most of the mature tools in the engineering are based on fixed pattern matching, and cannot realize intelligent vulnerability detection.

[0068] In order to solve the problems of non-automatic detection, manual detection according to experience and low detection accuracy, the inventors have found that artificial intelligence can be used to study vulnerability detection. With the help of deep learning technology, intelligent, efficient and large-scale code vulnerability automatic detection can be realized, so as to solve the outstanding contradiction between the detection accuracy and detection efficiency of the vulnerability detection method and the actual demand, and also liberate manpower.

[0069] Specifically, the application provides a code detection method and device and electronic equipment, in the application, target data to be code detection is acquired, and data cleaning operation is performed on the target data to obtain first data, each word in the first data is compared with the word in the preset word library to determine the feature value corresponding to each word in the first data, the feature value corresponding to each word in the first data is combined according to the word arrangement order to obtain the feature matrix corresponding to the target data, the preset code detection model is called to process the feature matrix to obtain the vulnerability detection result of the target data; the preset code detection model is obtained based on the training sample; the training sample includes the vulnerability type and the feature matrix of the vulnerability code sample. In the application, since the preset code detection model is obtained based on a large number of training samples, the target data to be code detection is analyzed by the preset code detection model obtained by training, the accuracy of vulnerability detection can be improved, and the information security requirement is met.

[0070] On the basis of the above, another embodiment of the application provides a code detection method, referring to Figure 1 may include:

[0071] S11, target data to be code detection is acquired, and data cleaning operation is performed on the target data to obtain first data.

[0072] In the embodiment, the vulnerability detection method is a static detection method, that is, the static code is directly detected, in the embodiment, the data to be code detection is referred to as target data.

[0073] In order to avoid the influence of useless data in the target data on the vulnerability detection, such as reducing the processing efficiency, in the embodiment, data cleaning operation is performed on the target data, the data cleaning operation can be an operation of removing comments, and the first data is obtained after the data cleaning operation is completed.

[0074] S12, each word in the first data is compared with the word in the preset word library to determine the feature value corresponding to each word in the first data.

[0075] In the embodiment, the preset word library can be a word library used when the preset code detection model is trained, the word library stores the words added in advance, the words are extracted from the code vulnerability sample, and can be obtained based on the jieba word segmentation technology in natural language processing.

[0076] The words in the preset word library are arranged in a specified order, and the specified order can be set according to actual conditions. For example, the words are arranged in the order of the initial letter, or the words in the same code vulnerability sample are arranged in order, and the words in different code vulnerability samples are arranged in the arrangement order of the code vulnerability sample.

[0077] After the words in the preset word library are arranged in the specified order, sequence numbers are set for the words in the preset word library, for example, the sequence number of the word arranged in the first position is 1, the sequence number of the word arranged in the second position is 2, and so on.

[0078] In another implementation manner of the present application, referring to Figure 2 , step S12 can include:

[0079] S21, determining the position of each word in the first data in the preset word library.

[0080] Specifically, for each word in the first data, the operation of determining the feature value is the same.

[0081] Firstly, the position of the word in the preset word library can be determined based on the one-hot algorithm. Taking the word Injection as an example, the word is searched in the preset word library, and it is found that the word is located in the second position in the preset word library.

[0082] S22, constructing initial feature information corresponding to each word in the first data, setting the identifier corresponding to the position in the initial feature information as a first value, and setting the identifier not corresponding to the position as a second value.

[0083] In the embodiment, the TFIDF algorithm is used for encoding. Specifically, the initial feature information corresponding to the word is that the data quantity is the number of words in the first data, and each data is initially set as a value, for example, all zeros. In addition, it can also be a random value.

[0084] For example, if the number of words in the first data is 30, a matrix of [0, 0, 0, 0, 0, 0………, 0, 0, 0, 0, 0] is constructed, and the number of 0 is 30.

[0085] Then, the identifier corresponding to the position of the word in the preset word library in the initial feature information is set as a first value, and the first value can be 1. The identifier in the embodiment is the value corresponding to the position. Still taking the word located in the second position as an example, the second value is set as 1, and the remaining values except the second value are the second value, for example, 0.

[0086] Then, the initial feature information corresponding to the word is changed to [0, 1, 0, 0, 0, 0………, 0, 0, 0, 0, 0].

[0087] S23, in the case where the number of words in the first data is less than a preset threshold, performing a data supplement operation on the initial feature information, so that the data quantity of the initial feature information is the preset threshold.

[0088] In this embodiment, the preset threshold is the maximum number of words of the vulnerability code sample used when training the preset code detection model, such as 50.

[0089] To ensure the uniformity of data, in this embodiment, data with non-uniform word numbers are subjected to number uniformity processing operation, and data less than the preset threshold are subjected to data supplement operation, so that the data amount of the initial feature information is the preset threshold.

[0090] Still taking the initial feature information changed to [0, 1, 0, 0, 0, 0………, 0, 0, 0, 0, 0] as an example, the number of words is 30, which is less than 50, so 20 data need to be supplemented. At this time, the value of the supplemented data can be -1.

[0091] The initial feature information after data supplement is:

[0092] [0, 1, 0, 0, 0, 0………, 0, 0, 0, 0, 0, -1, -1……-1].

[0093] If yes, the number of words is greater than 50, then only the first 50 can be retained, and the values after 50 are deleted.

[0094] It should be noted that the data supplement operation can be performed first, and then the operation of "setting the identifier corresponding to the position in the initial feature information to the first value, and setting the identifier not corresponding to the position to the second value" is performed.

[0095] S24, the initial feature information after data supplement corresponding to each word in the first data is taken as the feature value corresponding to the word.

[0096] After determining the initial feature information after data supplement, the initial feature information after data supplement is the feature value corresponding to the word.

[0097] S13, the feature values corresponding to each word in the first data are combined according to the word arrangement order to obtain the feature matrix corresponding to the target data.

[0098] In this embodiment, after determining the feature value corresponding to each word in the first data, the feature value corresponding to each word is combined according to the arrangement order of the word in the first data, and the feature matrix corresponding to the target data is obtained. For example, the feature matrix can be:

[0099]

[0100] S14, calling a preset code detection model to process the feature matrix to obtain a vulnerability detection result of the target data.

[0101] The preset code detection model is trained based on training samples; and the training samples include the vulnerability types and feature matrices of vulnerability code samples.

[0102] After the preset code detection model is trained based on the training samples, the feature matrix is input into the preset code detection model, and the vulnerability detection result of the target data is obtained. The vulnerability detection result can be that there is a vulnerability or there is no vulnerability. When there is a vulnerability, the vulnerability detection result further includes the vulnerability type.

[0103] In this embodiment, the target data to be subjected to code detection is obtained, and data cleaning operation is performed on the target data to obtain first data. Each word in the first data is positionally compared with the words in the preset word library to determine the feature value corresponding to each word in the first data. The feature values corresponding to each word in the first data are combined according to the word arrangement order to obtain the feature matrix corresponding to the target data. The preset code detection model is called to process the feature matrix to obtain the vulnerability detection result of the target data. The preset code detection model is trained based on training samples. The training samples include the vulnerability types and feature matrices of vulnerability code samples. In the present application, since the preset code detection model is trained based on a large number of training samples, the target data to be subjected to code detection is analyzed by the preset code detection model trained, and the accuracy of vulnerability detection can be improved to meet the information security requirements.

[0104] The above embodiment refers to the preset code detection model. Now, the generation process of the preset code detection model will be introduced with reference to Figure 3 may include:

[0105] S31, a code sample and a code repair sample corresponding to the code sample are obtained through a preset data acquisition method.

[0106] In this embodiment, there are two sources when obtaining vulnerability data, which are crawled from the Cvedetails website (CVE security vulnerability database. Security vulnerabilities, exploits, references and more (cvedetails.com)) or directly downloaded from SARD (Software Assurance Reference Dataset Project). In the Cvedetails website, three different software applications, Openssl, Binutils and Linux, are selected, and vulnerability code C files corresponding to different CWE types are obtained, which are called code samples. The obtained vulnerability C files usually contain statements unrelated to known vulnerabilities, which seriously affect the accuracy of the model, thereby causing false positives and false negatives. Therefore, in addition to crawling vulnerability code C files, it is also necessary to obtain C files fixed after repairing vulnerability code C files, which are called code repair samples corresponding to the code samples in this embodiment.

[0107] S32, difference analysis is performed on the code sample and the code repair sample, and a vulnerability code sample corresponding to the code sample and existing in the code sample but not existing in the code repair sample is obtained.

[0108] In this embodiment, the difference between the code sample and the code repair sample is analyzed, which is the vulnerability code sample existing in the code sample but not existing in the code repair sample.

[0109] Specifically, step S32 includes:

[0110] 1) difference analysis is performed on the code sample and the code repair sample, and the code sample is labeled based on the difference analysis result to obtain a labeling result.

[0111] Specifically, the diff technique is used to analyze the code sample and the code repair sample, and a noise-removed difference patch is processed, which contains the "+" and "-" diff parts in the patch. The diff part is divided into two parts, which are the part of the code sample and the part of the code repair sample that exist in the difference. The two parts are marked with "+" and "-", wherein "+" represents that the statement is a vulnerability repair part, and "-" represents that the statement is a vulnerability part. The patch marked with "+" and "-" is the labeling result.

[0112] 2) screen out the mark in the annotation result, which represents the existence in the code sample and the non-existence in the code repair sample, and take the code part corresponding to the mark in the code sample as a vulnerability code sample.

[0113] In this embodiment, the code part marked with "-" is screened out by using xpath regular matching, that is, as a vulnerability code fragment. In order to weaken the interference of vulnerability irrelevant statements and improve the accuracy of the model, the vulnerability code fragment can be subjected to a comment removing and other data cleaning operations to obtain a vulnerability code sample. Finally, the vulnerability code samples obtained from the Cvedetails website and the SARD website are 407 samples and 3628 samples respectively.

[0114] S33, determining the patch code sample corresponding to the vulnerability code sample from the code repair sample.

[0115] In this step, the statement marked with "-" in the patch is the patch code sample corresponding to the vulnerability code sample.

[0116] S34, determining the feature matrix and vulnerability type corresponding to the vulnerability code sample, and determining the feature matrix and vulnerability type corresponding to the patch code sample.

[0117] In another implementation manner of the present application, referring to Figure 4 , step S33 can include:

[0118] S41, combining each word group in the vulnerability code sample to obtain a preset word library.

[0119] The generation process of the preset word library in this embodiment has been introduced in the above-mentioned corresponding part. Please refer to the corresponding description in the above-mentioned embodiment.

[0120] S42, positionally comparing each word in the vulnerability code sample with the words in the preset word library to determine the feature value corresponding to each word in the vulnerability code sample.

[0121] S43, combining the feature values corresponding to each word in the vulnerability code sample according to the word arrangement order to obtain the initial feature matrix corresponding to the vulnerability code sample.

[0122] The generation process of the initial feature matrix has been explained and described in the above-mentioned embodiment. Please refer to the corresponding description in the above-mentioned embodiment.

[0123] S44, performing cluster analysis on the initial feature matrix corresponding to the vulnerability code sample to obtain a clustering result and the feature matrix corresponding to the vulnerability code sample.

[0124] Before clustering analysis, the initial feature matrix corresponding to the vulnerability code sample can be subjected to a data cleaning operation of de-duplication operation to reduce duplicate data.

[0125] In feature selection, KMeans is used to determine the feature matrix. The core process of K-Means is to assume that the samples are to be divided into K classes, then K samples are randomly selected from the samples, and the initial feature matrix of the K samples is taken as the initial clustering center, and different clustering centers correspond to different categories. Then, the selected clustering centers are respectively calculated with each sample, each sample has K distances, the minimum distance is calculated and the corresponding clustering center is obtained, the sample is divided into the same category as the corresponding clustering center, and the mean of all samples in each category is calculated as the new clustering center. This process is repeated until the new clustering center is equal to the old clustering center, and the iteration ends.

[0126] After iteration, the clustering result is obtained, and the vulnerability code samples in each clustering result correspond to the same vulnerability type, and then the specific category of the vulnerability type such as memory leak is determined manually.

[0127] In addition, through continuous iteration clustering, the initial feature matrix corresponding to the vulnerability code sample is continuously changed, and finally the feature matrix corresponding to the vulnerability code sample is obtained, and the numerical value in the feature matrix is a numerical value between 0 and 1.

[0128] Based on the K-Means classification algorithm, it is verified whether the same CWE type has certain similarity, so that the accuracy evaluation of the prediction model is more convincing.

[0129] S45, obtaining the vulnerability type corresponding to the clustering result, and taking the vulnerability type corresponding to the clustering result as the vulnerability type corresponding to the vulnerability code sample in the clustering result.

[0130] The vulnerability type is the vulnerability type determined manually, the vulnerability type corresponding to the clustering result is the same as the vulnerability type corresponding to the vulnerability code sample in the clustering result. Therefore, the vulnerability type corresponding to the clustering result is taken as the vulnerability type corresponding to the vulnerability code sample in the clustering result, so that the vulnerability type of the vulnerability code sample is determined, and these vulnerability types are known vulnerability types.

[0131] The process of determining the feature matrix corresponding to the patch code sample is the same as the process of determining the feature matrix corresponding to the vulnerability code sample, but the vulnerability type corresponding to the patch code sample is uniformly set to no vulnerability.

[0132] S34, using the feature matrix and the vulnerability type corresponding to the vulnerability code sample and the patch code sample respectively to train a preset code detection model until the training stops when the training stop condition is met.

[0133] In this embodiment, in order not to rely on program-related prior knowledge, the application uses a deep learning method for code similarity matching, and the preset code detection model can be an LSTM model.

[0134] The LSTM model in the application adopts a Keras framework, and an eight-layer neural network model is designed. Figure 5 Now each layer of the neural network will be described.

[0135] The first layer is the Marsking layer. One of the advantages of LSTM is that it can process variable-length sequences. When using Keras to build a model, if the LSTM layer is directly used as the first layer of the network input, the size of the input needs to be specified. If a variable-length sequence is to be used, only a Masking layer or an embedding layer needs to be added in front of the LSTM layer. First, the sequence is converted into a fixed-length sequence, such as selecting a maximum sequence length, and filling -1 for sequences shorter than this length. Then the filtering characters are specified in the mask_value in the Masking layer.

[0136] It should be noted that the above data supplement operation can be realized by the Marsking layer. If the data supplement operation has been performed when the characteristic value is determined, the Marsking layer can be omitted. If the data supplement operation has not been performed when the characteristic value is determined, the Marsking layer is used to perform the data supplement operation. Figure 5 X1……Xn in the above formula refers to model input, such as the above-mentioned matrix.

[0137] The second to fifth layers are LSTM layers. The long short-term memory network LSTM has a unique gate mechanism, which can solve the problem of variable-length sequence and long-distance dependence that MLP and CNN cannot handle. The encoding of the current word depends on the intermediate state and output value generated after the previous word is encoded by LSTM. Multi-layer LSTM is to stack LSTM, which has the advantages of being able to express features more abstractly at a higher level, reducing the number of neurons, increasing the recognition accuracy and reducing the training time. In addition, when constructing the LSTM of each layer, part of the weight is also discarded to prevent overfitting.

[0138] The sixth layer is the Dropout layer. Dropout refers to randomly selecting certain neurons according to a certain probability during DL training, and temporarily not considering their role in the network, which can also improve the overfitting effect. After each Dropout, it is equivalent to finding a thinner network from the original network.

[0139] The seventh layer is a batch normalization layer. Learning data distribution is the essence of the neural network learning process, and different distributions of the train and test sets will reduce the generalization ability of the network. Different distributions of each batch of train data will cause the network to learn different distributions at each epoch, and the training speed will also be reduced. Therefore, the data needs to be normalized. To solve the problem of changes in the distribution of intermediate layer data during training, the BatchNormalization algorithm is used.

[0140] The eighth layer is a fully connected dense layer. After linear operation and Softmax activation function, the probability of the sample distribution in each category is output, and the category with the largest probability value is the category.

[0141] By designing the above eight-layer neural network model, the feature matrix corresponding to the vulnerability code sample and the patch code sample and the vulnerability type are used to train the preset code detection model until the training stopping condition is met.

[0142] During modeling, 80% of the vulnerability code samples in the Linux, Openssl, and Binutils applications are used as the training set, and the remaining samples are used as the validation set to verify the prediction accuracy.

[0143] In order to verify the accuracy of the preset code detection model, in this embodiment, the MLP model and the CNN model are used for comparison to prove the effectiveness of the LSTM model.

[0144] On the basis of the embodiment, in order to better show the process of the algorithm, a front-end page is selected in the form of a web page. In the framework of the web server, Flask based on Python is selected, which is a typical web micro framework. In the implementation of the visualization process, the specific process is as follows: first, the shell file calls the app.py file to find the Flask entry. Then create a Flask class instance, specify the static file location, and use register blueprint to register different businesses to Flask. It is quite cumbersome to generate HTML with Python, because HTML must be manually escaped to ensure the security of the application. Therefore, in the first blueprint views.py, the render_template() function is used to specify the template for rendering, which is displayed to the front end. The location of the js specified in the html is associated with the location of the static file specified in the Flask. After the browser loads the html web page, the function in the js is called, and the triggering function is set by the different id of the element to make different responses. The detailed information of cvedetails is crawled, the git_url vulnerability C file and the repaired C file are obtained, the data processing, the different data source model business switching of the spider data and the downloaded data, the CNN, the LSTM and the simple MLP are implemented. The parameters required for these functions are input by the html page, the js obtains the values input by the front end, and the values are passed to the second blueprint. The flask blueprint passes the parameters to the model and other function functions, and the js returns the results to the front page for feedback. Take the acquisition and result display of the vulnerability code C file and the repaired C file as an example for display. The application needs to crawl the vulnerability C file and the repaired C file. Click to obtain the crawler, and the right side displays the log output by the console. The result display calls the output result saving path, and the visualization data.

[0145] In the embodiment, LSTM is used for vulnerability detection to improve detection accuracy. FLASK is used to interact with the front end to realize interface visualization and facilitate user use. In addition, the Scrapy crawler completes the preliminary vulnerability library construction model, and not directly uses the existing data. 2) Realize the structured representation of the vulnerability code segment. For the collected vulnerability code, the black list statistics related to vulnerabilities are performed based on the word segmentation technology such as jieba word segmentation in natural language processing, and then the TFIDF is used to design a method for determining the feature value of the vulnerability code segment based on the black list.

[0146] Optionally, on the basis of the embodiment of the above code detection method, another implementation manner of the present application gives a code detection device, which is described with reference to Figure 6 may include:

[0147] The data processing module 11 is configured to obtain target data to be subjected to code detection, and perform data cleaning on the target data to obtain first data.

[0148] The feature determination module 12 is configured to compare each word in the first data with words in a preset word library to determine a feature value corresponding to each word in the first data.

[0149] The matrix determination module 13 is configured to combine the feature values corresponding to each word in the first data according to a word arrangement order to obtain a feature matrix corresponding to the target data.

[0150] The vulnerability detection module 14 is configured to call a preset code detection model to process the feature matrix to obtain a vulnerability detection result of the target data. The preset code detection model is trained based on training samples. The training samples include a vulnerability type and a feature matrix of a vulnerability code sample.

[0151] Further, the model generation module includes:

[0152] The sample acquisition sub-module is configured to acquire code samples and code repair samples corresponding to the code samples by a preset data acquisition manner.

[0153] The sample analysis sub-module is configured to perform difference analysis on the code samples and the code repair samples to obtain vulnerability code samples corresponding to the code samples and existing in the code samples but not existing in the code repair samples.

[0154] The sample determination sub-module is configured to determine repair code samples corresponding to the vulnerability code samples from the code repair samples.

[0155] The vulnerability determination sub-module is configured to determine a feature matrix and a vulnerability type corresponding to the vulnerability code samples, and determine a feature matrix and a vulnerability type corresponding to the repair code samples.

[0156] The model training sub-module is configured to train a preset code detection model using the feature matrices and the vulnerability types corresponding to the vulnerability code samples and the repair code samples, respectively, until a training stop condition is met.

[0157] Further, the vulnerability determination sub-module includes:

[0158] The first combination unit is configured to combine each word in the vulnerability code samples to obtain a preset word library.

[0159] The comparison unit is configured to positionally compare each word in the vulnerability code sample with a word in a preset word library to determine a feature value corresponding to each word in the vulnerability code sample.

[0160] The second combination unit is configured to combine the feature values corresponding to each word in the vulnerability code sample according to a word arrangement order to obtain an initial feature matrix corresponding to the vulnerability code sample.

[0161] The clustering unit is configured to perform clustering analysis on the initial feature matrix corresponding to the vulnerability code sample to obtain a clustering result and a feature matrix corresponding to the vulnerability code sample.

[0162] The type determination unit is configured to obtain a vulnerability type corresponding to the clustering result and take the vulnerability type corresponding to the clustering result as a vulnerability type corresponding to a vulnerability code sample in the clustering result.

[0163] Further, the sample analysis submodule is specifically configured to:

[0164] perform difference analysis on the code sample and the code repair sample, label the code sample based on a difference analysis result to obtain a labeling result, filter out an identifier in the labeling result that represents an existence in the code sample and an absence in the code repair sample, and take a code part corresponding to the identifier in the code sample as a vulnerability code sample.

[0165] Further, the feature determination module 12 includes:

[0166] The position determination submodule is configured to determine a position of each word in the first data in a preset word library.

[0167] The numerical value determination submodule is configured to construct initial feature information corresponding to each word in the first data, set an identifier corresponding to the position in the initial feature information as a first numerical value, and set an identifier not corresponding to the position as a second numerical value.

[0168] The data supplement submodule is configured to, in a case where a quantity of words in the first data is less than a preset threshold, perform a data supplement operation on the initial feature information, so that a data quantity of the initial feature information is the preset threshold.

[0169] The feature value determination submodule is configured to take the initial feature information corresponding to each word in the first data after the data supplement as a feature value corresponding to the word.

[0170] In the embodiment, target data to be subjected to code detection is acquired, data cleaning operation is performed on the target data to obtain first data, each word in the first data is positionally compared with a word in a preset word library to determine a feature value corresponding to each word in the first data, the feature value corresponding to each word in the first data is combined according to a word arrangement order to obtain a feature matrix corresponding to the target data, a preset code detection model is called to process the feature matrix to obtain a vulnerability detection result of the target data; the preset code detection model is trained based on training samples; the training samples include a vulnerability type and a feature matrix of a vulnerability code sample. In the application, since the preset code detection model is trained based on a large number of training samples, the target data to be subjected to code detection is analyzed by the preset code detection model trained, the accuracy of vulnerability detection is improved, and the information security requirement is met.

[0171] It should be noted that the working processes of the various modules, sub-modules and units in the embodiments are described above, and will not be described here.

[0172] Optionally, based on the embodiments of the code detection method and device, another implementation manner of the application provides an electronic device, comprising a memory and a processor.

[0173] The memory is configured to store a program.

[0174] The processor calls the program and is configured to execute the code detection method.

[0175] In the embodiment, target data to be subjected to code detection is acquired, data cleaning operation is performed on the target data to obtain first data, each word in the first data is positionally compared with a word in a preset word library to determine a feature value corresponding to each word in the first data, the feature value corresponding to each word in the first data is combined according to a word arrangement order to obtain a feature matrix corresponding to the target data, a preset code detection model is called to process the feature matrix to obtain a vulnerability detection result of the target data; the preset code detection model is trained based on training samples; the training samples include a vulnerability type and a feature matrix of a vulnerability code sample. In the application, since the preset code detection model is trained based on a large number of training samples, the target data to be subjected to code detection is analyzed by the preset code detection model trained, the accuracy of vulnerability detection is improved, and the information security requirement is met.

[0176] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A code detection method, characterized in that, include: Obtain the target data to be detected by code, and perform data cleaning on the target data to obtain the first data; The target data is static code; Each word in the first data is compared with the word in the preset dictionary to determine the feature value corresponding to each word in the first data; In the preset vocabulary, words from the same code vulnerability sample are placed sequentially, and words from different code vulnerability samples are arranged according to the order of the code vulnerability samples. The feature values ​​corresponding to each word in the first data are combined according to the word arrangement order to obtain the feature matrix corresponding to the target data; The feature matrix is ​​processed by a preset code detection model to obtain the vulnerability detection result of the target data. If the vulnerability detection result indicates the existence of a vulnerability, the vulnerability detection result also includes the vulnerability type. The preset code detection model is trained based on training samples. The training samples include vulnerability types and feature matrices of vulnerability code samples. In the preset code detection model, the first layer is the Masking layer, the second to fifth layers are LSTM layers, the sixth layer is the Dropout layer, the seventh layer is the Batch Normalization layer, and the eighth layer is the Dense layer. The generation process of the preset code detection model includes: By using a preset data acquisition method, code samples and corresponding code repair samples are acquired, wherein the code repair samples are the repaired code samples. The code sample and the code repair sample are analyzed for differences using the diff technique, and the code sample is labeled based on the results of the difference analysis to obtain the labeling results; The XPath regular expression is used to filter out the markers in the annotation results that represent the code sample but not the code repair sample, and the code part corresponding to the marker in the code sample is taken as the vulnerability code sample; From the code repair samples, the patch code sample corresponding to the vulnerability code sample is determined, and the patch code sample is the patched vulnerability code sample; Cluster analysis is performed on the vulnerable code samples based on clustering algorithms to determine the feature matrix and vulnerability type corresponding to the vulnerable code samples, and to determine the feature matrix and vulnerability type corresponding to the patch code samples. The vulnerability type corresponding to the patch code samples is no vulnerability. The preset code detection model is trained using the feature matrices and vulnerability types corresponding to the vulnerability code samples and the patched code samples, respectively, until the training stops when the training stop condition is met.

2. The code detection method according to claim 1, characterized in that, Cluster analysis is performed on the vulnerable code samples based on clustering algorithms to determine the feature matrix and vulnerability type corresponding to the vulnerable code samples, including: Each word in the vulnerability code sample is combined to obtain a preset word library; Each word in the vulnerability code sample is compared with the word in the preset dictionary to determine the feature value corresponding to each word in the vulnerability code sample. The feature values ​​corresponding to each word in the vulnerability code sample are combined according to the word order to obtain the initial feature matrix corresponding to the vulnerability code sample; Cluster analysis is performed on the initial feature matrix corresponding to the vulnerability code sample to obtain the clustering results and the feature matrix corresponding to the vulnerability code sample. Obtain the vulnerability type corresponding to the clustering result, and use the vulnerability type corresponding to the clustering result as the vulnerability type corresponding to the vulnerability code sample in the clustering result.

3. The code detection method according to claim 1, characterized in that, Each word in the first data is compared with the position of words in a preset dictionary to determine the feature value corresponding to each word in the first data, including: Determine the location of each word in the first data within a preset vocabulary; Construct initial feature information for each word in the first data, and set the identifier corresponding to the position in the initial feature information as a first value, and set the identifiers not corresponding to the position as a second value; If the number of words in the first data is less than a preset threshold, data supplementation is performed on the initial feature information so that the amount of data in the initial feature information is the preset threshold. The initial feature information corresponding to each word in the first data after data supplementation is used as the feature value corresponding to the word.

4. A code detection device, characterized in that, include: The data processing module is used to acquire target data to be detected by code and to perform data cleaning operations on the target data to obtain first data. The target data is static code; The feature determination module is used to compare the position of each word in the first data with the words in the preset word library in order to determine the feature value corresponding to each word in the first data; In the preset vocabulary, words from the same code vulnerability sample are placed sequentially, and words from different code vulnerability samples are arranged according to the order of the code vulnerability samples. The matrix determination module is used to combine the feature values ​​corresponding to each word in the first data according to the word arrangement order to obtain the feature matrix corresponding to the target data; The vulnerability detection module is used to call a preset code detection model to process the feature matrix and obtain the vulnerability detection result of the target data. If the vulnerability detection result indicates that a vulnerability exists, the vulnerability detection result also includes the vulnerability type. The preset code detection model is trained based on training samples. The training samples include vulnerability types and feature matrices of vulnerability code samples. In the preset code detection model, the first layer is the Masking layer, the second to fifth layers are LSTM layers, the sixth layer is the Dropout layer, the seventh layer is the Batch Normalization layer, and the eighth layer is the Dense layer. The sample acquisition submodule is used to acquire code samples and corresponding code repair samples through a preset data acquisition method. The code repair samples are the repaired code samples. The sample analysis submodule is used to perform difference analysis between the code sample and the code repair sample using diff technology, and to annotate the code sample based on the difference analysis results to obtain the annotation results; The XPath regular expression is used to filter out the markers in the annotation results that represent the code sample but not the code repair sample, and the code part corresponding to the marker in the code sample is taken as the vulnerability code sample; The sample determination submodule is used to determine the patch code sample corresponding to the vulnerability code sample from the code repair samples, wherein the patch code sample is the patched vulnerability code sample. The vulnerability determination submodule is used to perform cluster analysis on the vulnerability code sample based on a clustering algorithm to determine the feature matrix and vulnerability type corresponding to the vulnerability code sample, and to determine the feature matrix and vulnerability type corresponding to the patch code sample, wherein the vulnerability type corresponding to the patch code sample is no vulnerability; The model training submodule is used to train a preset code detection model using the feature matrices and vulnerability types corresponding to the vulnerability code samples and the patched code samples, respectively, until the training stops when the training stop condition is met.

5. The code detection device according to claim 4, characterized in that, The vulnerability determination submodule includes: The first combination unit is used to combine each word in the vulnerability code sample to obtain a preset word library; The comparison unit is used to compare the position of each word in the vulnerability code sample with the words in the preset word library to determine the feature value corresponding to each word in the vulnerability code sample. The second combination unit is used to combine the feature values ​​corresponding to each word in the vulnerability code sample according to the word arrangement order to obtain the initial feature matrix corresponding to the vulnerability code sample. A clustering unit is used to perform clustering analysis on the initial feature matrix corresponding to the vulnerability code sample to obtain the clustering result and the feature matrix corresponding to the vulnerability code sample. The type determination unit is used to obtain the vulnerability type corresponding to the clustering result and use the vulnerability type corresponding to the clustering result as the vulnerability type corresponding to the vulnerability code sample in the clustering result.

6. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor calls the program and is used to execute the code detection method as described in any one of claims 1-3.

Images