Open-source License Quick Identification Method, System and Medium for Large-scale Software

Abstract

The present invention discloses a method, system, and medium for rapidly identifying open source licenses for large-scale software. The method comprises determining a set of open source licenses to be identified; constructing an AC automaton for identifying the open source licenses using the identifiers of each open source license in the set to be identified; and obtaining, for a user code file to be identified, an identification result for each open source license in the set to be identified in the user code file using the AC automaton. The method employs the AC automaton to identify licenses in text through a single scan, reducing the time complexity of the identification process. When parsing the input user code file, annotation information containing the license declaration text is extracted for identification, reducing the amount of matching content during the identification process. The method is particularly suitable for rapidly identifying open source licenses for large-scale software.

Description

Technical Field

[0001] The present invention relates to technologies such as component analysis and open source license management in the field of computer software, and in particular to a method, system and medium for quickly identifying open source licenses for large-scale software. Background Art

[0002] Currently, the use of open source components has become one of the important ways to achieve rapid software development and technological innovation.

[0003] Software development is becoming increasingly globalized, and a large number of third-party open source components (OSS) are used to improve software development efficiency after being modified or expanded with required functionality. According to a report released by Forrester in 2021, the proportion of open source code in the code bases of 17 industries audited, such as the Internet of Things and network security, has almost doubled in five years. With the development of the information technology industry, the increasing difficulty of software development and the trend of open source software supply chain, the large-scale introduction of open source software has become the mainstream solution for software development. However, many developers and software buyers have a significant misunderstanding of the term "open source", confusing open source software with free software or shared software that does not require specific licenses, and believing that open source software can also be modified, copied and distributed as needed.

[0004] Open source software is accompanied by open source licenses, which specify the scope and conditions under which users can use the released open source software. However, some engineers who misunderstand the concept of open source often overlook the importance of open source licenses. They modify open source software or open source code and then republish or commercially exploit it, violating the terms of the open source license and causing legal disputes. According to statistics from the Black Duck audit service team, in 2021, 73% of audited codebases contained open source code with license conflicts, and 30% of audited codebases had no license or used open source code with custom licenses.

[0005] During software development, introducing conflicting open source licenses can lead to compliance risks. For example, the GNU General Public License (GPL) and the Mozilla Public License (MPL) require that any use of source code containing the license must be accompanied by the release of the entire source code in accordance with the GPL. The MPL, on the other hand, requires that if the MPL license appears in a separate code file, the additional files can be exempted from open source disclosure. Therefore, when enterprises use open source software licensed under both the GPL and MPL, they may face compliance risks due to conflicting terms in either open source license.

[0006] To avoid intellectual property infringement and legal disputes, and to ensure the proper and compliant use of open source software, open source license identification has become an essential task for software developers. Despite the numerous open source license identification and management tools currently available, most research on open source licenses focuses on tools from the open source community or applications developed by commercial companies, such as Go License Detector, FindLicense, and FOSSSology. While current open source license management tools utilize different algorithms for identifying open source licenses, they all perform matching and identification by processing specific clauses within the license text. For example, Go License Detector uses the Minhash method to pre-process the content of open source licenses, while FOSSology employs the bSAM algorithm to segment the text within the license clauses. Licenses contain relatively complex clauses, and matching based on the clauses requires a high number of text matches. However, with the rapid growth in the amount of open source software and the increasing scale of software development, efficient open source license identification is crucial. Summary of the Invention

[0007] Technical problem to be solved by the present invention: In response to the above-mentioned problems in the prior art, a method, system and medium for rapid identification of open source licenses for large-scale software are provided. The present invention adopts an AC automaton to identify the license in the text through a one-pass scanning method, thereby reducing the time complexity of the identification process. Then, the input user code file is parsed to extract the annotation information containing the license declaration text for identification, thereby reducing the matching content in the identification process. The present invention is particularly suitable for rapid identification of open source licenses for large-scale software.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] A method for rapid identification of open source licenses for large-scale software, including:

[0010] S101, determining the set of open source licenses that need to be identified;

[0011] S102, constructing an AC automaton for identifying open source licenses using the identifiers of each open source license in the set of open source licenses to be identified;

[0012] S103 , for the user code file to be identified, using the AC automaton to obtain an identification result of each open source license in the set of open source licenses to be identified in the user code file.

[0013] Optionally, determining the set of open source licenses to be identified in step S101 refers to using web crawling technology to crawl online information of the open source licenses to be identified from a web page containing the open source licenses to be identified to obtain the set of open source licenses to be identified.

[0014] Optionally, step S101 includes:

[0015] S201, obtaining the URL of a webpage of an open source license to be identified;

[0016] S202, reading table data for recording open source license information from the input URL of the webpage of the open source license to be identified, and converting the table data into a data frame format;

[0017] S203, extracting the full name and identifier of each open source license from the data frame format, and obtaining the license text content of each open source license according to the license text address pointed to by the hyperlink therein;

[0018] S204: Taking the full name, identifier, and license text of each open source license as an attribute of each open source license in the set of open source licenses to be identified, the set of open source licenses to be identified is constructed.

[0019] Optionally, the URL of the webpage of the open source license to be identified input in step S201 refers to the URL of the Software Package Data Exchange license list on the website of the Software Package Data Exchange SPDX: https: / / spdx.org / licenses / .

[0020] Optionally, when reading the table data for recording the open source license information from the URL of the web page of the open source license to be identified in step S202, the table data for recording the open source license information read includes at least one of table data for recording the open source licenses in normal use and table data for recording the open source licenses that have been abandoned.

[0021] Optionally, step S102 includes: for the open source license set to be identified, dividing the identifiers of the open source licenses in the open source license set into independent character strings that do not contain the separator “-”, using the independent character strings as nodes in the word search tree Trie of the AC automaton M, and constructing the word search tree Trie of the AC automaton M with the identifiers of the open source licenses in the open source license set to obtain an AC automaton for identifying open source licenses.

[0022] Optionally, step S103 includes:

[0023] S301: For the user code file to be identified, traverse the user software directory, obtain the directory tree of the user software, extract the annotation information containing the license statement in the user code file, establish a mapping relationship between the annotation information and the code file, mark the extracted annotation information with an index value in the reading order, and use the index value to mark the corresponding code file;

[0024] S302: According to the index value of the tag, the extracted annotation information is input as a string into the AC automaton, and the output result of the AC automaton is used to determine the open source license contained in the corresponding code file. If the open source license identifier is output, it means that the code file contains the open source license corresponding to the identifier, thereby obtaining the identification results of each open source license in the set of open source licenses that need to be identified in all user code files.

[0025] Optionally, the AC automaton constructed in step S102 is implemented in Python. Before inputting the extracted annotation information as a string into the AC automaton in step S302, the step further includes constructing an AC automaton service cluster using the Python distributed computing framework Ray. After receiving the input string, the AC automaton service cluster allocates one or more child nodes to execute the AC automaton program to recognize the string using a preset scheduling strategy by the master node of the AC automaton service cluster, and returns all recognition results through the master node. Constructing the AC automaton service cluster using the Python distributed computing framework Ray includes:

[0026] S401, pre-selecting a node from multiple physical or virtual machines as the master node of the AC automaton service cluster, and activating a Python runtime environment containing the distributed computing framework Ray on the master node and the remaining nodes;

[0027] S402, using the distributed computing framework Ray on each of the remaining nodes to specify the IP address of the master node and set the output of activating the AC automaton service cluster by the master node to add the remaining nodes as child nodes to the AC automaton service cluster;

[0028] S403: Use a decorator to annotate the program of the AC automaton, and serialize the program code of the AC automaton to the redis database of the distributed computing framework Ray and store it as an object for each child node to call and execute data exchange.

[0029] In addition, the present invention also provides a system for rapidly identifying open source licenses for large-scale software, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the method for rapidly identifying open source licenses for large-scale software.

[0030] In addition, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to be programmed or configured by a microprocessor to execute the method for rapid identification of open source licenses for large-scale software.

[0031] Compared with the prior art, the present invention has the following advantages: the present invention's method for rapidly identifying open source licenses for large-scale software includes determining a set of open source licenses to be identified; constructing an AC automaton for identifying open source licenses using the identifiers of each open source license in the set of open source licenses to be identified; and, for a user code file to be identified, utilizing the AC automaton to obtain identification results for each open source license in the set of open source licenses to be identified in the user code file. The present invention employs an AC automaton to identify licenses in text through a single scan, thereby reducing the time complexity of the identification process. When parsing the input user code file, annotation information containing the license declaration text is extracted for identification, thereby reducing the amount of matching content in the identification process. The method is particularly suitable for rapidly identifying open source licenses for large-scale software. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0033] Figure 2 The figure is a flowchart of the steps of extracting open source license information in an embodiment of the present invention.

[0034] Figure 3 This is a flow chart of the AC automatic machine structure in an embodiment of the present invention.

[0035] Figure 4 This is a flow chart for determining the output mode of an AC automaton in an embodiment of the present invention.

[0036] Figure 5 Flowchart of the open source license identification steps in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0038] like Figure 1 As shown, the method for quickly identifying open source licenses for large-scale software in this embodiment includes:

[0039] S101, determining the set of open source licenses that need to be identified;

[0040] S102, constructing an AC automaton for identifying open source licenses using the identifier of each open source license in the set of open source licenses to be identified;

[0041] S103 , for the user code file to be identified, using the AC automaton to obtain an identification result of each open source license in the set of open source licenses to be identified in the user code file.

[0042] It should be noted that the set of open source licenses to be identified in step S101 can be determined in a feasible implementation manner as needed. For example, as a preferred implementation manner, the set of open source licenses to be identified in step S101 of this embodiment refers to adopting web crawling technology to crawl the information of the open source licenses to be identified online from the web page containing the open source licenses to be identified, so as to obtain the set of open source licenses to be identified. The advantage of this method is that through online extraction, it can avoid missed detection and false detection of open source licenses due to untimely updates of the locally constructed open source license database.

[0043] like Figure 2 As shown, step S101 of this embodiment includes:

[0044] S201, obtaining the URL of a webpage of an open source license to be identified;

[0045] See also Figure 2 As an optional implementation, the URL of the webpage containing the open source license to be identified input in step S201 of this embodiment refers to the Software Package Data Exchange (SPDX) website, which lists licenses at: https: / / spdx.org / licenses / (hereinafter referred to as the official SPDX website). The URL of the webpage containing the open source license to be identified may also be the URL of another webpage containing open source licenses to be identified. SPDX is an open standard launched by the Linux Foundation for exchanging software bill of materials information. SPDX has currently standardized the names, identifiers, and other information of over 400 open source licenses and is continuously updating them. Therefore, SPDX can be used to construct a standard open source license database for identifying open source licenses used in user software, providing a basis for identifying open source licenses.

[0046] S202, reading table data for recording open source license information from the input URL of the webpage of the open source license to be identified, and converting the table data into a data frame format;

[0047] Reading tabular data recording open source license information from the URL of the webpage inputting the open source license to be identified can be done using any desired method, such as character analysis or a component or control that includes character analysis. For example, in this embodiment, the Python language pandas component is used to read tabular data recording open source license information from the URL of the webpage inputting the open source license to be identified. License-related information is categorized and extracted according to the header information in the form; the license text content is obtained through hyperlinks to the license terms, and the extracted license-related information is organized and outputted for storage. Specifically, in this embodiment, the read_html() method of the pandas component is used to read tabular data from the SPDX License List webpage (tabular data is divided into two categories: one for open source licenses that are still usable and the other for deprecated open source licenses). The HTML table is then converted into the pandas DataFrame data format. The license-related information is then categorized and extracted according to the header information in the form, including: "Full name," "Identifier," "FSF Free / Libre?", and "OSI Approved?" Each open source license's "Full Name" can be hyperlinked to a webpage displaying the license text, which details the terms and conditions of the license and specifies the rights and obligations of users. Using this hyperlink, use the Python requests method get() to retrieve the webpage content and return it as binary data. After decoding the binary data into text, extract the license text using the "license-text" keyword.

[0048] S203, extracting the full name and identifier of each open source license from the data frame format, and obtaining the license text content of each open source license according to the license text address pointed to by the hyperlink therein;

[0049] See also Figure 2 As an optional implementation, in step S202 of this embodiment, when reading the table data for recording the open source license information from the input URL of the webpage of the open source license to be identified, the table data for recording the open source license information read includes at least one of table data for recording the open source licenses in normal use and table data for recording the deprecated open source licenses;

[0050] S204: The full name, identifier, and license text of each open source license are used as attributes of each open source license in the set of open source licenses to be identified, thereby constructing a set of open source licenses to be identified (which may also be referred to as an open source license database). Specifically, in this embodiment, the full name, identifier, and license text of each open source license are output as attributes of each open source license in the set of open source licenses to be identified and saved in a CSV format file, thereby constructing a set of open source licenses to be identified (which may also be referred to as an open source license database).

[0051] This embodiment extracts open source license information from the official SPDX website through online extraction, eliminating the need to manually update the open source license database due to updates to the SPDX License List. Running the open source license information extraction program unit in this invention updates the constructed standard license database. Furthermore, the SPDX License List also includes information on deprecated open source licenses. These licenses are marked when constructing the open source license database. If such licenses are detected in a user's code files, the user is prompted to update promptly to avoid license conflicts and compatibility risks.

[0052] In this embodiment, step S102 includes: for a set of open source licenses to be identified (hereinafter referred to as the identifier set), segmenting the identifiers of the open source licenses in the open source license set into independent strings without the separator "-", using the independent strings as nodes in a trie of AC automaton M, and constructing a trie of AC automaton M using the identifiers of the open source licenses in the open source license set to obtain an AC automaton for identifying open source licenses. Research and analysis revealed that the URL for the Software Package Data Exchange (SPDX) website lists licenses with a large number of common substrings between the names and identifiers of different licenses developed by the same organization. These common substrings are generally separated by "-". Therefore, in this embodiment, the identifiers of the open source licenses in the open source license set are divided into independent character strings that do not contain the separator "-". The independent character strings are used as nodes in the word search tree Trie of the AC automaton M, so that common substrings can be used as nodes. Compared with the traditional method of constructing a word search tree Trie using single letters as nodes, the number of nodes is greatly reduced, which saves memory overhead on the one hand and improves the efficiency of traversing the Trie tree for scanning on the other hand. Moreover, after using independent character strings as nodes in the word search tree Trie of the AC automaton M, the identifiers of the open source licenses in the open source license set are used to construct the word search tree Trie of the AC automaton M to obtain an AC automaton for identifying open source licenses, the word search tree Trie is traversed to obtain the invalidation function value of each node through recursive rules, so that multi-mode matching can be completed in one traversal, reducing the time complexity of matching, and a judgment is made when the final node is output to determine whether the longest substring match is achieved, thereby improving the accuracy of recognition.

[0053] In this embodiment, the constructed AC automaton is denoted as AC automaton M, which is composed of (Q, Σ, g, f, q0, F), where Q represents a finite state set, Σ represents a finite input character set, g represents a state transition function, f is a failure function, q0 represents an initial state, and F represents a final state set.

[0054] like Figure 3As shown, the identifiers of the open source licenses in the open source license set are split into independent strings that do not contain the separator “-”, and the independent strings are used as nodes in the word search tree Trie of the AC automaton M. After the identifiers of the open source licenses in the open source license set are used to construct the word search tree Trie of the AC automaton M, the word search tree Trie includes the finite state set Q, the input character set Σ, the state transition function g, the initial state q0 and the final state set F in the AC automaton. Starting from the root node of the word search tree Trie, a recursive function is used to adopt a depth-first traversal method to construct the failure value (mismatch value) of each node to obtain the failure function of the AC automaton, that is, the next state to jump to when the match is unsuccessful. The root node of the word search tree Trie is layer 0, and the failure value of the first-layer node is the root node. According to the recursive function f(i)=g(f(i.pre),x), the corresponding failure function value can be obtained, where f(i) represents the failure function value of the current node, f(i.pre) represents the failure function value of the predecessor node of the current node, x represents the input character from the predecessor node to the current node, and g represents the state transition function. When the AC automaton reaches the final state, it is determined whether it has a successor node. If not, the current successful match identifier and its corresponding open source license are output. If so, continue to input characters for matching. If the match fails, the identifier recognized by the final state node and its corresponding open source license are output. If the match is successful and reaches the next final state, the above judgment steps are repeated. The constructed output pattern is as follows: Figure 4 As shown in the figure, the identification structure of each open source license in the set of open source licenses to be recognized is used as input to construct an AC automaton. Compared with using single letters as independent characters, the units separated by "-" are used to reduce the number of nodes in the word search tree Trie and save memory overhead. The invalidation function is constructed. When the current state fails to match, the prefix of the successfully matched state is used to find the next matching state to continue matching, without returning to the initial state, thus reducing the time complexity of matching. The final state is judged during output to ensure the longest identification match, avoid similar identification recognition errors (such as BSD-3-Clause and BSD-3-Clause-Attribution), and improve recognition accuracy.

[0055] like Figure 5 As shown, step S103 in this embodiment includes:

[0056] S301: For the user code file to be identified, traverse the user software directory, obtain the directory tree of the user software, extract the annotation information containing the license statement in the user code file, establish a mapping relationship between the annotation information and the code file, mark the extracted annotation information with an index value in the reading order, and use the index value to mark the corresponding code file;

[0057] S302: According to the index value of the tag, the extracted annotation information is input as a string into the AC automaton, and the output result of the AC automaton is used to determine the open source license contained in the corresponding code file. If the open source license identifier is output, it means that the code file contains the open source license corresponding to the identifier, thereby obtaining the identification results of each open source license in the set of open source licenses that need to be identified in all user code files.

[0058] In steps S301-S302, the user code file is parsed to extract the comment text containing the license declaration information, narrowing the matching range. Whether the code file contains the open source license is determined by the AC automaton's identification of the open source license. This step uses the extracted comment information containing the license declaration from the user code file as input for the AC automaton to identify the license. The license declaration in the code file is located at the beginning of the file (typically on the first line). Therefore, the first five lines of comment information in the user code file are read as the textual basis for identifying the open source license used. Using a distributed framework to execute the AC automaton's recognition process improves recognition efficiency, enabling rapid license identification even for large-scale software.

[0059] In this embodiment, the AC automaton constructed in step S102 is implemented in Python. Before the extracted annotation information is input as a string into the AC automaton in step S302, it also includes using Python's distributed computing framework Ray to build an AC automaton service cluster; after the AC automaton service cluster receives the input string, the main node of the AC automaton service cluster uses a preset scheduling strategy to allocate one or more child nodes to execute the AC automaton program to recognize the string and return all recognition results through the main node. By using Python's high-performance distributed computing framework Ray to build a distributed computing cluster, the AC automaton program is distributedly optimized to further improve the recognition efficiency.

[0060] In this embodiment, using the Python distributed computing framework Ray to build an AC automaton service cluster includes:

[0061] S401, pre-select one node from multiple physical or virtual machines as the master node of the AC automaton service cluster, and activate the Python runtime environment containing the distributed computing framework Ray on the master node and the remaining nodes; for example, in this embodiment, specifically, a computer is selected as the master node, and the command to activate the Python environment containing the distributed computing framework Ray on the master node and the remaining nodes is: "ray start --head";

[0062] S402, using the distributed computing framework Ray on each of the remaining nodes to specify the IP address of the master node and set the output of activating the AC automaton service cluster by the master node to add the remaining nodes as child nodes to the AC automaton service cluster;

[0063] After the child node activates the Python environment containing Ray, run "ray start --address='master node IP' --redis-password='system output when activated by the master node'" to join the AC automaton service cluster;

[0064] In S403, decorators are used to annotate the AC automaton program. The program code is serialized and stored as an object in the Redis database of the distributed computing framework Ray. This object is then used by child nodes to call and exchange data. Specifically, in this example, the decorator @ray.remote is used to annotate the AC automaton recognition program and serialize the code into the Redis database as an object, enabling asynchronous execution and data exchange. Tasks are prioritized for completion on the local node; if they are unsuccessful, they are dispatched to other nodes by the global scheduler of the AC automaton service cluster.

[0065] In summary, open source license management, as a key technology in software component analysis, is of great significance for avoiding open source license conflicts and compliance risks in software. Open source license identification is a prerequisite for implementing open source license management. As the scale of software expands, the identification efficiency of current open source license management tools needs to be improved. In response to the above problems, this embodiment provides a method for rapid identification of open source licenses for large-scale software. The method constructs an open source license set based on the identifier of the license in the SPDX open source license list, and uses an improved AC automaton for license identification based on the identifier, reducing the time complexity of the matching process. At the same time, Python's high-performance distributed computing framework Ray is used to further improve performance, so that the method can efficiently handle the open source license identification task of large-scale software.

[0066] In addition, this embodiment also provides a system for quickly identifying open source licenses for large-scale software, including

[0067] An open source license information extractor unit, used to determine a set of open source licenses that need to be identified;

[0068] An AC automaton construction program unit is used to construct an AC automaton for identifying open source licenses using the identifiers of the respective open source licenses in the set of open source licenses to be identified;

[0069] The open source license identification program unit is used for obtaining, for a user code file to be identified, an identification result of each open source license in the open source license set to be identified in the user code file by using an AC automaton.

[0070] In addition, this embodiment also provides a system for rapid identification of open source licenses for large-scale software, including a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute a method for rapid identification of open source licenses for large-scale software.

[0071] In addition, this embodiment further provides a computer-readable storage medium, in which a computer program is stored. The computer program is used to be programmed or configured by a microprocessor to execute a method for quickly identifying open source licenses for large-scale software.

[0072] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0073] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for rapid identification of open source licenses for large-scale software, characterized by: include: S101, determining the set of open source licenses that need to be identified; S102, constructing an AC automaton for identifying open source licenses using the identifiers of each open source license in the set of open source licenses to be identified; S103, for the user code file to be identified, using the AC automaton to obtain an identification result of each open source license in the set of open source licenses to be identified in the user code file; Step S102 includes: for the open source license set to be identified, splitting the identifiers of the open source licenses in the open source license set into independent character strings that do not contain the separator "-", using the independent character strings as nodes in a word search tree Trie of an AC automaton M, and constructing a word search tree Trie of the AC automaton M using the identifiers of the open source licenses in the open source license set to obtain an AC automaton for identifying open source licenses; Step S103 includes: S301: For the user code file to be identified, traverse the user software directory, obtain the directory tree of the user software, extract the annotation information containing the license statement in the user code file, establish a mapping relationship between the annotation information and the code file, mark the extracted annotation information with an index value in the reading order, and use the index value to mark the corresponding code file; S302: According to the index value of the tag, the extracted annotation information is input as a string into the AC automaton, and the output result of the AC automaton is used to determine the open source license contained in the corresponding code file. If the open source license identifier is output, it means that the code file contains the open source license corresponding to the identifier, thereby obtaining the identification results of each open source license in the set of open source licenses that need to be identified in all user code files.

2. The method for rapid identification of open source licenses for large-scale software according to claim 1, characterized in that: Determining the set of open source licenses to be identified in step S101 refers to using web crawling technology to crawl online the information of the open source licenses to be identified from a web page containing the open source licenses to be identified, thereby obtaining the set of open source licenses to be identified.

3. The method for rapid identification of open source licenses for large-scale software according to claim 1, characterized in that: Step S101 includes: S201, obtaining the URL of a webpage of an open source license to be identified; S202, reading table data for recording open source license information from the input URL of the webpage of the open source license to be identified, and converting the table data into a data frame format; S203, extracting the full name and identifier of each open source license from the data frame format, and obtaining the license text content of each open source license according to the license text address pointed to by the hyperlink therein; S204: Taking the full name, identifier, and license text of each open source license as an attribute of each open source license in the set of open source licenses to be identified, the set of open source licenses to be identified is constructed.

4. The method for rapid identification of open source licenses for large-scale software according to claim 3, characterized in that: The URL of the webpage of the open source license to be identified input in step S201 refers to the URL of the Software Package Data Exchange license list on the website of the Software Package Data Exchange SPDX: https: / / spdx.org / licenses / .

5. The method for rapid identification of open source licenses for large-scale software according to claim 3, characterized in that: When reading the table data for recording the open source license information from the URL of the web page of the open source license to be identified in step S202, the table data for recording the open source license information read includes at least one of table data for recording the open source licenses in normal use and table data for recording the abandoned open source licenses.

6. The method for rapid identification of open source licenses for large-scale software according to claim 1, characterized in that: The AC automaton constructed in step S102 is implemented in Python. Before the extracted annotation information is input as a string into the AC automaton in step S302, the AC automaton service cluster is constructed using the Python distributed computing framework Ray. After receiving the input string, the AC automaton service cluster uses a preset scheduling strategy to allocate one or more child nodes to execute the AC automaton program to recognize the string, and all recognition results are returned through the master node. The construction of the AC automaton service cluster using the Python distributed computing framework Ray includes: S401, pre-selecting a node from multiple physical or virtual machines as the master node of the AC automaton service cluster, and activating a Python runtime environment containing the distributed computing framework Ray on the master node and the remaining nodes; S402, using the distributed computing framework Ray on each of the remaining nodes to specify the IP address of the master node and set the output of activating the AC automaton service cluster by the master node to add the remaining nodes as child nodes to the AC automaton service cluster; S403: Use a decorator to annotate the program of the AC automaton, and serialize the program code of the AC automaton to the redis database of the distributed computing framework Ray and store it as an object for each child node to call and execute data exchange.

7. A system for rapid identification of open source licenses for large-scale software, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the open source license rapid identification method for large-scale software as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, wherein: The computer program is used to be programmed or configured by a microprocessor to execute the open source license rapid identification method for large-scale software as claimed in any one of claims 1 to 6.

Images