Open source code provenance detection method based on software reuse feature learning

Abstract

The application discloses an open source code tracing detection method based on software reuse feature learning, which mainly queries corresponding reused open source software feature sets and reuse probabilities from an association rule table of a built reuse feature association model according to software features for tracing detection, searches software code with the features from an open source software library in a descending order of reuse probabilities, and compares the software code with the detected code one by one, so that the method has the advantage of being capable of quickly finding open source software sets with code reuse possibility, reducing the open source software range for tracing detection comparison, and greatly improving the efficiency and precision of open source code tracing detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software code trace detection, in particular to an open source code trace detection method based on software reuse feature learning. BACKGROUND

[0002] With the vigorous development of Internet technology and open source software, more and more developers participate in open source software development, and the open source community has accumulated a large amount of excellent open source software resources and open source knowledge resources. The free, open, customizable, and community-supervised characteristics of open source software make it possible to provide technical rapid application, increase innovation, reduce cost and development time, etc. in software development. In the background of the need to improve software development efficiency and quality, open source has become a powerful force in the IT industry. The benefits of open source are also the source of risk. According to the "2018 Open Source Code Security and Risk Analysis Black Duck" report released by Synopsys, the number of open source vulnerabilities in each code library increased by 134% compared with last year. In addition, open source software needs users to track the vulnerabilities, fixes and updates of the open source software they use. If an organization does not understand all the open source components it uses, it cannot resist common attacks against known vulnerabilities in these components, and exposes itself to license compliance risks and intellectual property legal risks, causing varying degrees of security threats and economic or reputational losses.

[0003] However, due to the diverse use of open source code by software projects, the scope of open source software used by the code is difficult to predict in advance, and the same software may introduce code from different open source software, so the number of open source software participating in code comparison in trace detection will be very large. If the software to be detected is compared with all open source software, it will be very time-consuming and laborious, and it is not realistic to implement. Therefore, how to accurately determine the scope of open source software participating in trace detection comparison, and select the open source software most likely to use code from the vast amount of open source software library to participate in trace detection comparison, has a great influence on improving the accuracy and efficiency of code trace detection.

[0004] Therefore, the present application aims to learn the features of open source software with reuse relationship through machine learning, establish a reuse feature correlation model, analyze the software reuse relationship, and quickly determine the scope of open source software participating in trace comparison through the features of the software to be detected, thereby significantly improving the efficiency and accuracy of trace detection. SUMMARY

[0005] The purpose of the present application is to provide an open source code trace detection method based on software reuse feature learning.

[0006] To this end, the technical solution of the present application is as follows:

[0007] A software reuse feature learning-based open source code provenance detection method, comprising the following steps:

[0008] 1) Build a reuse feature correlation model; the building method of the model is as follows:

[0009] 1-1) Select and collect the feature data of the reused software and the reused software in the software project with reuse relationship, wherein the reuse relationship has been explicitly defined in the open source software community:

[0010] 1-2) Mine the reuse feature correlation rules by using the Apriori algorithm: thereby obtaining the association rule table, i.e. the reuse feature correlation model;

[0011] 2) According to the software features for provenance detection, query the corresponding reused open source software feature set and its reuse probability from the association rule table, and search for software code with these features from the open source software library in order from high to low according to the reuse probability, and compare them one by one with the detected code for provenance comparison.

[0012] Further, the software feature data selected and collected in step 1-1) includes programming language, open source license, software type, software label and collection amount.

[0013] Further, the selection and collection method of the software feature data in step 1-1) is:

[0014] a) Obtain the software project name and owner information according to the need to obtain the software feature, and obtain the project git address;

[0015] b) Use the data framework and API provided by the open source community to access the interface of the open source community;

[0016] c) Use Java web crawler technology to operate the remote library of the open source project hosting platform of the open source community website by using open source Java dependent library, thereby grabbing the required software feature data;

[0017] d) Store the collected target data into the database, and complete the collection of software feature data.

[0018] Further, the reuse feature correlation rule mining by using the Apriori algorithm comprises the following steps:

[0019] 1-21) Mine the frequent item set in the feature data;

[0020] 1-22) Mine the association rules based on the frequent item set;

[0021] 1-23) Form the association rule table obtained from the association rules and store it in the database.

[0022] Further, the mining method of the frequent item set is as follows: setting a minimum support value, calculating the support value of all item sets, deleting the item sets with the support value less than the minimum value and all super item sets corresponding to the item sets in the database, and the remaining item sets are the frequent item sets.

[0023] Further, the mining method of the frequent item set is as follows: setting a minimum support value, calculating the support value of all item sets, deleting the item sets with the support value less than the minimum value and all super item sets corresponding to the item sets in the database, and the remaining item sets are the frequent item sets.

[0024] Further, the mining method of the frequent item set is as follows: setting a minimum support value, calculating the support value of all item sets, deleting the item sets with the support value less than the minimum value and all super item sets corresponding to the item sets in the database, and the remaining item sets are the frequent item sets.

[0025] Further, the mining method of the frequent item set is as follows: setting a minimum support value, calculating the support value of all item sets, deleting the item sets with the support value less than the minimum value and all super item sets corresponding to the item sets in the database, and the remaining item sets are the frequent item sets.

[0026] Compared with the prior art, the open source code tracing detection method based on software reuse feature learning quickly finds a set of open source software with code reuse possibility according to some features of the software to be traced and detected, thereby reducing the range of open source software for comparison in the tracing detection, and greatly improving the efficiency and accuracy of the open source code tracing detection. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The method framework diagram provided by the application is shown.

[0028] Figure 2 The flowchart of software feature data collection is shown.

[0029] Figure 3 The flowchart of the reuse software feature association model establishment is shown.

[0030] Figure 4 The schematic diagram of the support pruning principle based on the Apriori algorithm is shown.

[0031] Figure 5 The schematic diagram of the confidence pruning principle based on the Apriori algorithm is shown. DETAILED DESCRIPTION

[0032] The application will be further described below in combination with the drawings and specific embodiments, but the following embodiments are by no means any limitation on the application.

[0033] The application discloses an open source code tracing detection method based on software reuse feature learning, which mainly comprises the following steps: according to the software features for tracing detection, the corresponding reused open source software feature set and reuse probability are queried from an association rule table of a built reuse feature association model, and the software code with the features is searched from an open source software library in a descending order of the reuse probability, and the software code is traced and compared with the detected code one by one.

[0034] The reuse feature association model building method framework diagram is shown in the figure. Figure 1

[0035] Firstly, the feature data of the reused software and the reused software in the software project with the reuse relationship is selected and collected, wherein the reuse relationship is clear in the open source software community, and the selected and collected software feature data includes a programming language, an open source license, a software type, a software label and a collection amount.

[0036] The programming language is a language used for software development, and several different languages may be involved in an open source software project, but there is a main programming language. The code level reuse is researched in the patent, and therefore the programming language is listed as the software reuse feature.

[0037] The open source license regulates the use of the open source software through some clauses, and therefore the open source license is one of the software reuse features required by the patent.

[0038] Most open source projects are provided with labels, and the labels mainly describe the type, function, language and other related information of the project. The type, function and language information of the open source project are important factors for selecting software during code reuse, and therefore the software label should be listed as the software reuse feature.

[0039] The collection function of the open source project is provided in the open source community websites such as GitHub and Gitee, and the collection amount is recorded. The collection amount of an open source project reflects the popularity and popularity of the project, and can also reflect the quality and good degree of the software to a certain extent, and therefore the software collection amount should be listed as the software reuse feature.

[0040] The software feature data selection and collection method comprises the following steps.

[0041] a) According to the software project name and owner information required to obtain the software features, the project git address is obtained.

[0042] b) The data framework and API provided by the open source community are used to access the interface of the open source community.​

[0043] c) using Java web crawler technology, using open source Java dependent library to operate the remote library of open source project hosting platform of open source community website, so as to capture the required software feature data;

[0044] d) connecting Java and MySQL through Hibernate framework, storing the collected target data into the database, completing the collection of software feature data, and it is necessary to note that the software features collected here include the reused open source software features and the reused open source software features.

[0045] The flow chart of software feature collection in GitHub open source community is shown in Figure 2 .

[0046] After the software feature collection work is completed, the Apriori algorithm is used to mine the reused feature association rules, so as to obtain the association rule table, that is, the reused feature association model, and the flow chart is shown in Figure 3 .

[0047] In the mining of reused feature association rules, it is necessary to mine the frequent item set in the feature data first; the frequent item set refers to the set of items that frequently appear together. The "frequent" here is measured by support. For an item set, the definition of its support is the proportion of records containing the item set in the data set. Support is a concept related to item set, and a minimum support value is usually set, and the item set with support not less than the minimum value is considered as a frequent item set. In the mining of software reuse feature association rules, the frequent item set that can be obtained is the set of software features such as programming language, open source license, software type or software label, for example, the frequent item set {Android, java} indicates that "Android" and "java" often appear together.

[0048] The Apriori algorithm provides a pruning principle for the screening of frequent item sets, that is, if an item set is non-frequent, then all its supersets are also non-frequent, and the pruning method is shown in Figure 4 , if the item set {a, b} is non-frequent, then all its supersets circled by the dashed line and crossed out are also non-frequent. Therefore, the frequent item set is set to have a minimum support value when mining, the support of all item sets is calculated, the item sets and all super item sets corresponding to the item sets with support less than the minimum value are deleted, and the remaining item sets are the frequent item sets.

[0049] Secondly, based on the frequent item set, the association rules are mined; the association rules refer to the strong association that may exist between two different items. The "strong" is measured by the confidence. The confidence is a concept about the association rules, which is a conditional probability, i.e. the probability of the occurrence of event B at the premise of the occurrence of event A. A frequent item set can obtain several association rules, and the confidence of the association rules can be calculated according to the support of the item set. The higher the confidence value of an association rule is, the higher the probability of the rule is. The minimum value of the confidence is set, and the association rules less than the minimum value are excluded, and the association rules meeting the minimum value requirement are the connections between the features that we are interested in.

[0050] When the data volume is large, there are also many problems of the association rules generated from the frequent item set, and the number of the association rules can be reduced by using the Apriori principle. The following principles are followed here:

[0051] If the rule X→Y-X does not meet the minimum confidence, then for the subset X' of X, the rule X'→Y-X' also does not meet the minimum confidence. m n

[0052] Taking a frequent item set {a, b, c, d} as an example, the association rule mining is performed. First, all possible association rules of the item set are generated, as shown in the following table: Figure 5 According to the above theorem, if the confidence value of the rule b, c, d→a is lower than the set minimum confidence, then all rules composed of the subset of {b, c, d} are lower than the minimum confidence.

[0053] The method for mining the association rules of the frequent item set comprises the following steps:

[0054] The minimum value of the confidence is set, and the association rules of each frequent item set are screened, and the association rules corresponding to the frequent item set and the association rules corresponding to the subset of the frequent item set with the confidence less than the minimum value are excluded;

[0055] The support of each frequent item set composed of the reused software features and the non-reused software features is calculated, and the support of the frequent item set composed of the reused software features corresponding to the support is calculated, and the ratio of the two is the confidence of the association rule.

[0056] In the Gitee open source community, 103 groups of open source software with reuse relationship are selected as the experimental objects of the software reuse relationship analysis experiment, and 20 groups of them are selected and shown in Table 1.

[0057] Table 1 Reuse cases of open source software

[0058] ​​

[0059]

[0060] The data collected and processed are input into the Apriori algorithm to learn the association rules between the characteristics of the open source software reuse, output the experimental results, and analyze the output test results to the association rule analysis results shown in the table:

[0061] Table 2 Association rule analysis results

[0062] Serial number If the software being reused has features Then the software being reused can have features Credibility 1 Mobile development Android 0.71 2 Mobile development Android, java 0.86 3 Management monitoring MIT 0.85 4 Mobile development Apache 2.0 0.73 5 Android Android 0.86 6 Mobile development, Android Apache 2.0, java 0.99 7 Mobile development, Android Android, java 0.75

[0063] In Table 2, for each row representing an association rule, the open source software project has the characteristics in the second column, and the software reused by it may have the characteristics in the third column, and the credibility value of this association rule is the value in the fourth column. Here we set the minimum credibility value to 0.7, the higher the credibility value, the greater the possibility of the rule occurring, and the more valuable it is for reference and use. Take the association rule of the second row with sequence number 1 in Table 2 as an example, the open source software project is a mobile development project (mobile development), and it has a 71% possibility of reusing the open source software project containing the label "Android". Take the association rule of the seventh row with sequence number 6 as an example, the open source software project is a mobile development project (mobile development) and the main programming language is Android, and it has a 99% possibility of reusing the open source software project with Apache 2.0 open source license and the main programming language is java.

Claims

1. An open source code provenance detection method based on software reuse feature learning, characterized in that, The method comprises the following steps: A reuse feature association model is built, and the building method of the model is as follows: For the reused software and the reused software in the software project with the reuse relationship, the software project name and the owner information of the software features are obtained according to the needs, and the project git address is obtained; wherein the reuse relationship has been clearly defined in the open source software community: Using the data framework and API provided by the open source community, the interface of the open source community is accessed; Using the Java web crawler technology, the open source project hosting platform of the open source community website is operated by using the open source Java dependency library, so that the required software feature data is captured; the software feature data includes programming language, open source license, software type, software label and collection amount; The collected target data is stored in the database, and the software feature data collection work is completed; The Apriori algorithm is used to mine the reuse feature association rules, and the specific steps are as follows: Set a minimum support value, calculate the support of all item sets, delete the item sets in the database whose support is less than the minimum value and all super item sets corresponding to the item sets, and the remaining item sets are the frequent item sets; Based on the frequent item sets, set a minimum value of the confidence, and filter the association rules of each frequent item set, and exclude the association rules corresponding to the frequent item sets and the association rules corresponding to the subsets of the frequent item sets whose confidence is less than the minimum value; The support of each frequent item set composed of the reuse software features and the non-reuse software features is calculated, and the support of the frequent item set composed of the reuse software features corresponding thereto is calculated, and the ratio of the two is the confidence of the association rule; The obtained association rules are formed into an association rule table and stored in the database, so that the association rule table, that is, the reuse feature association model, is obtained; According to the software features for traceability detection, the corresponding reused open source software feature set and the reuse probability are queried from the association rule table, the software code with the features is searched from the compared open source software library in the order from high to low according to the reuse probability, and the traceability comparison is performed with the detected code one by one. ​

Images