Open source software license identification method, device, readable storage medium and equipment

Abstract

This invention discloses an open-source software license identification method, apparatus, readable storage medium, and device, belonging to the field of computer software technology. The method includes: receiving an abstract syntax tree (AST) generated from parsing a source code file; traversing each comment node of the AST, collecting all comment blocks and performing preliminary screening; performing multi-dimensional comprehensive scoring based on the position, comment type, license keywords, and full-text semantic aggregation of the preliminarily screened candidate comment blocks to obtain a credibility score; identifying candidate comment blocks with credibility scores higher than a set credibility threshold as license comment blocks; normalizing the license comment blocks to obtain normalized license text and calculating a hash fingerprint; and performing fuzzy matching comparison between the hash fingerprint and a pre-built standard license hash fingerprint library to obtain the license identification result. This invention achieves highly fault-tolerant comparison of license text, significantly improving the robustness and accuracy of license identification.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and in particular to an open-source software license identification method, apparatus, computer-readable storage medium, and device. Background Technology

[0002] With the rise of the "software assembly" development model, open-source components have permeated almost all software projects. Developers often incorporate numerous third-party libraries into their projects; statistics show that up to 97% of modern codebases contain open-source code. Different open-source components come with different open-source licenses, which grant users freedom but also impose various usage restrictions and obligations. For example, the GPL requires derivative software to be open-source, while licenses like MIT are relatively more lenient. Due to the sheer number of open-source components and the wide variety of licenses (the SPDX standard library already includes hundreds of licenses), ensuring open-source software license compliance has become a critical technical issue. Enterprises must ensure that all open-source component licenses used are compatible and meet requirements when delivering products; otherwise, they will face risks.

[0003] To manage the aforementioned risks, academia and industry have introduced a variety of open-source license scanning tools and SCA (Software Composition Analysis) platforms, such as FOSSology's Nomos / Monk module, BlackDuck, and FOSSA.

[0004] Despite some progress in license detection tools, the following significant shortcomings remain:

[0005] Non-standard license information is difficult to identify: Many codebases contain non-standard license statements. As one study pointed out, customized licenses are constantly emerging, such as copyright notices embedded in source code comments, modified LICENSE text snippets, or licenses referred to using non-standard expressions (such as "MIT-style license"). Traditional scanning tools mainly match standard LICENSE files or known license text, and cannot accurately find license traces scattered in the source code. These scattered and variant forms of license information are often missed or misdetected.

[0006] The parsing of fuzzy descriptions lacks intelligence, making it difficult to categorize ambiguous license terminology: license names and terms are ambiguous, and developers may use abbreviations, aliases, or unofficial descriptions to refer to licenses (e.g., mixing informal terms like "LGPL / MIT-style"). Existing tools rely on fixed rules or keyword matching, which often fails to correctly categorize these fuzzy descriptions into standard license identities, requiring manual intervention for judgment. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides an open-source software license identification method, apparatus, computer-readable storage medium, and device, which achieves highly fault-tolerant comparison of license texts and significantly improves the robustness and accuracy of license identification.

[0008] The technical solution provided by this invention is as follows:

[0009] A method for identifying open-source software licenses, the method comprising:

[0010] Receive an abstract syntax tree generated from parsing a source code file, the abstract syntax tree including structural nodes and comment nodes of the source code;

[0011] Traverse each comment node, collect all comment blocks, and perform preliminary screening on the collected comment blocks to obtain candidate comment blocks;

[0012] A credibility score is obtained by comprehensively evaluating the candidate annotation blocks based on their location, annotation type, license keywords, and full-text semantic aggregation.

[0013] Candidate annotation blocks whose credibility scores are higher than the set credibility threshold are identified as license annotation blocks;

[0014] The license annotation block is text normalized to obtain the license normalized text;

[0015] Calculate the hash fingerprint of the license normalized text;

[0016] The calculated hash fingerprint is compared with a pre-built standard license hash fingerprint library using fuzzy matching to obtain the license identification result corresponding to the license annotation block.

[0017] Furthermore, the credibility score is obtained by performing a multi-dimensional comprehensive evaluation based on the position, annotation type, license keywords, and full-text semantic aggregation of the candidate annotation blocks, including:

[0018] The candidate comment blocks are assigned corresponding position scores based on their positions in the source code file.

[0019] The candidate annotation blocks are assigned corresponding type scores based on their form and scope.

[0020] Assign corresponding keyword scores based on the typical license keywords contained in the candidate annotation blocks;

[0021] All lines of the candidate annotation block are aggregated into a complete full-text text, and the full-text text is subjected to semantic and format analysis, and a corresponding text aggregation score is assigned.

[0022] The credibility score is obtained by weighting the location score, type score, keyword score, and text aggregation score.

[0023] Furthermore, the text normalization of the license annotation block to obtain normalized license text includes:

[0024] Remove comment markers and meaningless characters from the license comment block, retain the pure license text content, and perform formatting adjustments including merging and line breaks. While maintaining the original semantics of the license comment block text content, unify the format to obtain the standardized license text.

[0025] Furthermore, the hash fingerprint of the license normalization text includes a MinHash signature and a SimHash fingerprint value, and the standard license hash fingerprint library includes the identification information of known standard license templates, the MinHash signature and SimHash fingerprint value of the text corresponding to the standard license template.

[0026] Furthermore, the standard license hash fingerprint library is pre-built in the following manner:

[0027] Collect known standard license templates and perform normalization preprocessing on the text of the standard license templates to remove irrelevant format differences;

[0028] The normalized preprocessed text is divided into several element sets, and multiple sets of hash functions are calculated for each element set to form a set of MinHash signatures; the normalized preprocessed text is then vectorized into feature vectors and SimHash values ​​are calculated.

[0029] The MinHash signature and SimHash value are mapped to the identification information of the standard license template and stored in the standard license hash fingerprint database.

[0030] Furthermore, the step of performing a fuzzy matching comparison between the calculated hash fingerprint and a pre-built standard license hash fingerprint library to obtain the license identification result corresponding to the license annotation block includes:

[0031] Calculate the Jaccard similarity based on the MinHash signature of the license canonical text and the MinHash signature of the standard license template;

[0032] The standard license templates with Jaccard similarity scores higher than the set similarity threshold are identified as candidate templates;

[0033] The Hamming distance is calculated based on the SimHash fingerprint value of the license normalized text and the SimHash fingerprint value of the standard license template.

[0034] The matching standard license template is determined based on the Jaccard similarity and Hamming distance corresponding to each candidate template, and the license recognition result is obtained.

[0035] Furthermore, the step of determining the matching standard license template based on the Jaccard similarity and Hamming distance corresponding to each candidate template to obtain the license recognition result includes:

[0036] A comprehensive similarity score is calculated based on the Jaccard similarity and Hamming distance corresponding to each candidate template.

[0037]

[0038] Where Score represents the overall similarity score, and J represents the Jaccard similarity score. Indicates Hamming distance, and The weighting coefficients are set;

[0039] Candidate templates whose comprehensive similarity score is higher than the set comprehensive score threshold are identified as matching standard license templates, and the identification information of the matching standard license templates is used as the license identification result.

[0040] An open-source software license identification device, the device comprising:

[0041] Abstract syntax tree acquisition module, used to receive an abstract syntax tree generated from parsing source code files, the abstract syntax tree including structural nodes and comment nodes of source code;

[0042] The candidate annotation block filtering module is used to traverse each annotation node, collect all annotation blocks, and perform preliminary filtering on the collected annotation blocks to obtain candidate annotation blocks.

[0043] The multi-dimensional comprehensive scoring module is used to perform multi-dimensional comprehensive scoring based on the position, annotation type, license keywords and full-text semantic aggregation of the candidate annotation block to obtain a credibility score.

[0044] The license annotation block determination module is used to determine the candidate annotation blocks whose credibility scores are higher than the set credibility threshold as license annotation blocks;

[0045] The text normalization module is used to normalize the license annotation block to obtain the license normalized text.

[0046] A hash fingerprint calculation module is used to calculate the hash fingerprint of the license normalization text;

[0047] The fuzzy matching module is used to perform fuzzy matching comparison between the calculated hash fingerprint and a pre-built standard license hash fingerprint library to obtain the license identification result corresponding to the license annotation block.

[0048] Furthermore, the multi-dimensional comprehensive scoring module includes:

[0049] A position evaluation unit is used to assign a corresponding position score based on the position of the candidate comment block in the source code file.

[0050] A type evaluation unit is used to assign a corresponding type score based on the form and scope of the candidate annotation block;

[0051] The keyword evaluation unit is used to assign corresponding keyword scores based on the typical license keywords contained in the candidate annotation block;

[0052] The text aggregation evaluation unit is used to aggregate all lines of the candidate annotation block into a complete full-text text, and to perform semantic and format analysis on the full-text text and assign a corresponding text aggregation score.

[0053] The weighted calculation unit is used to perform weighted calculations on the location score, type score, keyword score, and text aggregation score to obtain the credibility score.

[0054] Furthermore, the text normalization module is used for:

[0055] Remove comment markers and meaningless characters from the license comment block, retain the pure license text content, and perform formatting adjustments including merging and line breaks. While maintaining the original semantics of the license comment block text content, unify the format to obtain the standardized license text.

[0056] Furthermore, the hash fingerprint of the license normalization text includes a MinHash signature and a SimHash fingerprint value, and the standard license hash fingerprint library includes the identification information of known standard license templates, the MinHash signature and SimHash fingerprint value of the text corresponding to the standard license template.

[0057] Furthermore, the standard license hash fingerprint library is pre-built in the following manner:

[0058] Collect known standard license templates and perform normalization preprocessing on the text of the standard license templates to remove irrelevant format differences;

[0059] The normalized preprocessed text is divided into several element sets, and multiple sets of hash functions are calculated for each element set to form a set of MinHash signatures; the normalized preprocessed text is then vectorized into feature vectors and SimHash values ​​are calculated.

[0060] The MinHash signature and SimHash value are mapped to the identification information of the standard license template and stored in the standard license hash fingerprint database.

[0061] Furthermore, the fuzzy matching module includes:

[0062] The Jaccard similarity calculation unit is used to calculate the Jaccard similarity based on the MinHash signature of the license normalized text and the MinHash signature of the standard license template.

[0063] The candidate template unit is used to identify the standard license templates whose Jaccard similarity is higher than a set similarity threshold as candidate templates.

[0064] Hamming distance calculation unit is used to calculate Hamming distance based on the SimHash fingerprint value of the license normalized text and the SimHash fingerprint value of the standard license template;

[0065] The matching unit is used to determine the matching standard license template based on the Jaccard similarity and Hamming distance corresponding to each candidate template, so as to obtain the license recognition result.

[0066] Furthermore, the matching unit includes:

[0067] The comprehensive similarity score calculation subunit is used to calculate the comprehensive similarity score based on the Jaccard similarity and Hamming distance corresponding to each candidate template.

[0068]

[0069] Where Score represents the overall similarity score, and J represents the Jaccard similarity score. Indicates Hamming distance, and The weighting coefficients are set;

[0070] The comprehensive similarity score comparison subunit is used to identify candidate templates whose comprehensive similarity score is higher than the set comprehensive score threshold as matching standard license templates, and to use the identification information of the matching standard license templates as the license identification result.

[0071] A computer-readable storage medium for identifying open-source software licenses includes a memory for storing processor-executable instructions that, when executed by the processor, implement the steps of the open-source software license identification method described above.

[0072] An apparatus for identifying open-source software licenses includes at least one processor and a memory storing computer-executable instructions, wherein the processor executes the instructions to implement the steps of the open-source software license identification method.

[0073] The present invention has the following beneficial effects:

[0074] This invention combines abstract syntax tree context with multi-dimensional rules, utilizing the hierarchical information of syntax nodes in the source code and the positional features of comments. It designs a comprehensive judgment model integrating positional weights, comment types, keyword matching, and semantic aggregation scoring, providing a multi-layered, high-precision open-source license detection technology that accurately captures license statements. Compared to existing schemes that rely solely on string matching, it boasts higher recall and precision, and can detect non-standard license statements. In particular, by utilizing structural information such as the position and type of comments, it effectively avoids misidentifying irrelevant comments as licenses, reducing missed detections due to format variations.

[0075] This invention designs a fault-tolerant matching mechanism. By constructing a standard license hash fingerprint database and combining it with a fuzzy matching algorithm, it achieves a high fault-tolerant comparison of license texts. It can identify texts with extremely high similarity but not completely identical belonging to the same license category, which greatly improves the robustness and accuracy of license recognition. Attached Figure Description

[0076] Figure 1 This is a flowchart of the open-source software license identification method of the present invention;

[0077] Figure 2 This is a schematic diagram of the open-source software license identification device of the present invention. Detailed Implementation

[0078] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0079] This invention provides a method for identifying open-source software licenses, such as... Figure 1 As shown, the method includes:

[0080] S100: Receives an abstract syntax tree generated from parsing a source code file, which includes structural nodes of the source code and accompanying comment node information.

[0081] An Abstract Syntax Tree (ABS) is a data structure used to describe the structure of source code. It parses source code into a series of abstract syntactic structures and uses a tree structure to represent the relationships between these syntactic structures. ABS generation is a core step in the process of source code by compilers and code analysis tools, transforming human-readable source code into a tree structure that programs can analyze and manipulate.

[0082] The upstream module in this step parses the source code file, generates an abstract syntax tree containing structural nodes and comment nodes, and passes it to the module corresponding to this step.

[0083] S200: Traverse each annotation node, collect all annotation blocks, and perform preliminary screening on the collected annotation blocks to obtain candidate annotation blocks.

[0084] After collecting the comment blocks, these comment blocks are initially screened according to certain rules, such as prioritizing the selection of comment blocks at the beginning of a file or module as candidate comment blocks to narrow down the scope of subsequent analysis.

[0085] S300: A credibility score is obtained by comprehensively evaluating the candidate annotation block based on its location, annotation type, license keywords, and full-text semantic aggregation.

[0086] This step employs a multi-dimensional strategy algorithm based on abstract syntax trees to extract and identify license comment blocks. The core idea is to utilize the structural information and comment features of the source code to perform a multi-dimensional comprehensive scoring of comment paragraphs that may contain license statements.

[0087] Specifically, for each candidate annotation block, a predefined set of rules is applied for detailed analysis, including calculating its positional weight, determining the annotation type, matching license keywords, and aggregating the full text content for scoring. Multiple factors are combined to form the credibility score of the annotation block. By combining the positional weight of the annotation node, keywords, annotation type, and semantic aggregation score, the system can automatically identify license declaration areas in the file header, before module declarations, and other scattered locations, accurately locating potential license annotation blocks at the structural level.

[0088] S400: Identify candidate annotation blocks with credibility scores higher than the set credibility threshold as license annotation blocks.

[0089] This step is used to identify potential licensed annotation blocks. When the credibility score of a candidate annotation block is higher than the preset credibility threshold, it can be identified as a licensed annotation block. If there is only one candidate annotation block and its credibility score exceeds the credibility threshold, it is directly identified as a licensed annotation block. If multiple candidate annotation blocks reach the credibility threshold, the one with the highest credibility score can be selected, or in scenarios that support multiple licenses, all high-scoring candidate annotation blocks can be retained.

[0090] S500: Normalize the text of the license comment block to obtain the normalized license text.

[0091] The specific operations for text normalization can be as follows: remove comment markers (such as / *, / / , #, etc.) and meaningless characters from the license comment block, retain the clean license text content, and perform formatting adjustments including merging line breaks, removing redundant whitespace, and standardizing capitalization (if necessary). While maintaining the original semantics of the license comment block text content, the format is unified to obtain standardized license text. Through the aforementioned operations, the implicit license declarations in the source code are effectively extracted and standardized, providing basic data support for the entire license compliance detection process.

[0092] S600: Calculate the hash fingerprint of the license normalized text.

[0093] Calculating a hash fingerprint facilitates the unique identification and rapid comparison of the license canonical text. This hash fingerprint can be generated using hash algorithms such as MD5 or SHA-1 to produce a fixed-length digest value, which serves as an identifier for extracting the license canonical text. Since identical license text should produce the same hash fingerprint regardless of the file it originates from, this hash fingerprint can be used to quickly detect duplicate license claims and also provides an index for subsequent matching with a standard license hash fingerprint database.

[0094] S700: The calculated hash fingerprint is compared with a pre-built standard license hash fingerprint library using fuzzy matching to obtain the license identification result corresponding to the license annotation block.

[0095] Specifically, this invention generates a standard license hash fingerprint library in advance (either during system initialization or preprocessing) from a known set of standard license templates (such as the official texts of various open-source protocols). During the actual scanning and identification process, when the hash fingerprint of the license normalized text extracted from upstream is received, this step performs similarity matching between the hash fingerprint and the standard license hash fingerprint library based on a fuzzy hash matching algorithm to identify the closest license category.

[0096] In compliance testing of large software projects, it is necessary to automatically locate license statements (such as open-source license text) contained within source code files to ensure compliance with open-source license agreements. However, license statements are usually embedded in the source code as comments, scattered in various locations and with diverse formats. Traditional keyword-based search methods are easily affected by variations in comment formatting or subtle differences in text, leading to missed or false detections. For example, matching based solely on certain keywords may misinterpret ordinary comments as license statements, or miss actual license text due to differences in line breaks or comment symbols.

[0097] Furthermore, even when using the same license, source code files from different sources may have subtle differences in their comment text. For example, developers might add project names, modify the year, or have differences in line breaks and spaces due to formatting adjustments, or even delete or modify parts of the license text. These variations can cause traditional exact matching methods to fail, often resulting in recognition failures when the text content changes slightly. For instance, simple string comparisons are highly sensitive to minute differences due to the "avalanche effect," failing to identify text with similar but not identical content as belonging to the same license. This poses a challenge to automated compliance checks, potentially leading to misclassifying files that actually contain a license as unlicensed, or identifying license variants as unknown licenses.

[0098] This invention constructs an abstract syntax tree (AST) from the software source code and uses the code structure information provided by the AST to intelligently extract potential license comment blocks. This significantly improves the automation and accuracy of license detection, reduces the workload of manual review, and ensures that no license information embedded in the source code is missed during code compliance reviews. Furthermore, by combining the positional weight of comment nodes, comment type, keywords, and semantic aggregation scoring, this invention can automatically identify license declaration areas, accurately locate potential license comment blocks at the structural level, effectively distinguish between ordinary comments and license declarations, greatly improve detection accuracy, and provide high-quality input for subsequent fuzzy license matching and comparison.

[0099] This invention combines abstract syntax tree context with multi-dimensional rules, utilizing the hierarchical information of syntax nodes in the source code and the positional features of comments. It designs a comprehensive judgment model integrating positional weights, comment types, keyword matching, and semantic aggregation scoring, providing a multi-layered, high-precision open-source license detection technology that accurately captures license statements. Compared to existing schemes that rely solely on string matching, it boasts higher recall and precision, and can detect non-standard license statements. In particular, by utilizing structural information such as the position and type of comments, it effectively avoids misidentifying irrelevant comments as licenses, reducing missed detections due to format variations. Furthermore, it is highly versatile, capable of interfacing with abstract syntax tree parsers for different programming languages, achieving cross-language adaptation. License comment recognition can be performed simply by adjusting the comment extraction rules for each language.

[0100] Furthermore, this invention designs a fault-tolerant matching mechanism. By constructing a standard license hash fingerprint database and combining it with a fuzzy matching algorithm, it achieves a high fault-tolerant comparison of license texts. It can identify texts with extremely high similarity but not completely identical belonging to the same license category, which greatly improves the robustness and accuracy of license recognition.

[0101] As one possible embodiment of the present invention, the aforementioned S300 includes:

[0102] S310: Assign a position score based on the location of the candidate comment block in the source code file.

[0103] For example, comment blocks that appear at the beginning of a source code file (file header) have the highest weight because license statements are usually located at the beginning of the file; conversely, comments that are deeply embedded in functions or modules have lower weight and are less likely to contain license statements. Therefore, this invention assigns different weights to comment blocks based on their position in the file.

[0104] S320: Assign a type score based on the form and scope of the candidate annotation block.

[0105] License statements are often presented as block comments (e.g., / * ... * / multiline comments in C / C++ or docstring comments in Python), so candidates of block or docstring comment types will receive higher priority; while single-line comments are usually used for code explanation, and the probability of a complete license statement appearing in a single-line comment is lower. Therefore, different type scores should be assigned based on the form and scope of the comment.

[0106] S330: Assign corresponding keyword scores based on the typical license keywords contained in the candidate annotation blocks.

[0107] By scanning the comment text, we determine whether it contains typical license-related keywords or phrases, such as "License," "Copyright," "Apache License," "GPL," and "MIT License." Comment blocks containing license terminology are more likely to be license statements. This invention incorporates a common license keyword database and uses keyword matching to initially determine the nature of the comment content and assign corresponding keyword scores.

[0108] S340: Aggregate all lines of the candidate annotation block into the complete full-text text, perform semantic and format analysis on the full-text text, and assign corresponding text aggregation scores.

[0109] This invention merges all lines of the annotation block into a complete text and performs a comprehensive evaluation based on content and format. For example, it analyzes the text's length, syntactic structure, and similarity to common license text templates. Typically, license statement annotations are longer, contain legal terminology, and are formatted correctly (e.g., clear paragraphs, copyright statements, and license conditions), thus receiving higher scores; conversely, brief annotations or those unrelated to the license wording score lower. By aggregating the entire text for semantic and format analysis and assigning corresponding text aggregation scores, the accuracy of identification can be further improved.

[0110] S350: The credibility score is obtained by weighting the location score, type score, keyword score and text aggregation score.

[0111] For each candidate annotation block, a weighted calculation is performed based on the aforementioned multi-dimensional features to generate a comprehensive score that measures the probability of it containing a license statement. The weights can be set according to different levels of importance.

[0112] Before performing license recognition, this invention requires the pre-construction of a standard license hash fingerprint database during system initialization or preprocessing. The specific construction process is as follows:

[0113] 1. Collect known standard license templates and perform normalization preprocessing on the text of the standard license templates to remove irrelevant format differences.

[0114] This step involves template collection and standardization. First, the system collects a large number of known standard license texts as a standard license template library (e.g., the original official texts of common open-source licenses such as GPL, Apache, and MIT). Then, each standard license template undergoes standardization preprocessing to unify encoding and format, and remove irrelevant format differences (such as extra spaces, blank lines at the end of lines, and indentation differences) to ensure that subsequent hash fingerprint extraction focuses on the substantive content of the license itself, rather than format features.

[0115] 2. Divide the normalized preprocessed text into several element sets, and calculate multiple sets of hash functions for each element set to form a set of MinHash signatures; and perform feature vectorization on the normalized preprocessed text and calculate the SimHash value.

[0116] Based on normalization, two unique hash fingerprint features are generated for each license text:

[0117] MinHash signature set: A set of features used to characterize text, which can approximate the Jaccard similarity between two texts.

[0118] SimHash fingerprint value: used to represent the overall semantic features of text, so that similar texts can be mapped to hash results with only minor differences.

[0119] Specifically, the following operations are performed on each normalized preprocessed text:

[0120] The text is divided into several sets of elements (which can be segmented by words or fixed-length string fragments). Multiple hash functions are calculated for each set of elements, and the sequence of minimum values ​​is selected to form a set of MinHash signature vectors (usually a set of several minimum hash values). At the same time, the text content is feature vectorized and SimHash values ​​(which can be 64-bit or 128-bit binary hashes) are calculated.

[0121] 3. Establish a mapping relationship between the MinHash signature and SimHash value and the identification information of the standard license template (such as license name, version number or SPDX ID, etc.), and store them together in the standard license hash fingerprint database.

[0122] This fingerprint database can be viewed as a database table containing license feature entries, used to support subsequent efficient matching queries. Hash tables, inverted indexes, and other structures can be used to accelerate the search. The entire construction process described above can be completed in one go in an offline environment, and then used for fast querying and similarity matching.

[0123] The standard license hash fingerprint database of this invention supports dynamic updates and incremental learning. It can continuously collect new license samples and their variant texts, and continuously expand the database content by automatically generating hash fingerprint features, thus achieving incremental learning and automatic expansion of recognition capabilities. Compared with the limitations of commercial SCA tools that rely on closed databases, this invention can evolve on its own in an open-source environment, improving the recognition coverage of emerging or custom licenses.

[0124] After the standard license hash fingerprint database is established, during the identification phase, the aforementioned normalized license text is used to calculate its hash fingerprint, including the MinHash signature and SimHash fingerprint value, in the same way to ensure a consistent feature structure during matching. Then, a similarity comparison is performed with all license fingerprint entries in the standard license hash fingerprint database. The specific process includes:

[0125] S710: Calculate the Jaccard similarity based on the MinHash signature of the license canonical text and the MinHash signature of the standard license template.

[0126] S720: Select standard license templates with a Jaccard similarity higher than the set similarity threshold as candidate templates.

[0127] This step utilizes a set of MinHash signature features to calculate similarity and perform preliminary candidate screening. Specifically, it compares the MinHash signature of the license normalized text with the MinHash signature of each template in the standard license hash fingerprint database to estimate the Jaccard similarity with each standard license. Templates with similarity higher than a preset similarity threshold (e.g., 0.8) are quickly selected as candidate templates, thereby narrowing the scope of subsequent comparisons and improving matching efficiency while ensuring recall.

[0128] In one possible example, Local Sensitive Hash (LSH) indexing technology can be used to quickly find similar MinHash signatures, exclude obviously dissimilar templates, and accelerate the matching of large-scale fingerprint databases.

[0129] S730: Calculate the Hamming distance based on the SimHash fingerprint value of the license normalization text and the SimHash fingerprint value of the standard license template.

[0130] S740: Determine the matching standard license template based on the Jaccard similarity and Hamming distance corresponding to each candidate template to obtain the license recognition result.

[0131] This step further calculates the SimHash Hamming Distance between the candidate templates filtered by MinHash and the input text for fine-grained comparison, achieving fine-grained difference identification. The smaller the Hamming Distance, the more similar the two are; for example, if the distance is ≤ the distance threshold (e.g., within 3 bits of difference), it is judged as highly similar text; if the distance is slightly higher but still within an acceptable range (e.g., ≤ 5 bits), it can be used as a secondary candidate.

[0132] Based on the comparison results above, candidate templates are sorted and thresholded. If a candidate template simultaneously satisfies both high MinHash similarity and low SimHash distance (or is judged using the comprehensive similarity score described later), the license normalized text is determined to match the template, and its license type is identified as the license represented by that template. If multiple templates are close, the template with the highest similarity is selected as the result. Furthermore, the matching similarity scores (Jaccard similarity and Hamming distance, etc.) are recorded for result interpretation or further manual review. If no template passes the threshold, a "no match" result is output, indicating that no known license type could be identified.

[0133] This invention, by pre-constructing a standard license hash fingerprint database and employing an efficient hash comparison fault-tolerant matching mechanism, enables the automatic license identification process to have strong anti-interference capabilities and practical value. It can handle license comparison of large-scale files while ensuring accuracy, meeting the performance requirements in practical engineering.

[0134] Furthermore, in response to the reality that license texts have multiple variations (such as year changes, format rewriting, project name insertion, etc.), this invention adopts a two-layer hash fingerprint matching mechanism based on the fusion of MinHash (minimum hash) and SimHash (similarity hash). It fully utilizes the advantages of MinHash and SimHash respectively, realizes multi-layer fault-tolerant matching, provides more comprehensive and reliable fault-tolerant matching capabilities, and ensures that the system maintains high accuracy while taking into account a certain matching speed in large-scale code scanning.

[0135] The MinHash algorithm measures the overlap of text content, ensuring that even if the paragraph order of the license text is adjusted or some content is missing, as long as the majority of the content is consistent, it can still be identified as similar. The SimHash algorithm is insensitive to minor modifications to local text details (such as individual character replacements, punctuation changes, etc.), ensuring that small wording changes will not hinder matching. By combining the MinHash global similarity and SimHash local similarity dual-hash fusion comparison mechanism, the similarity between texts is judged more accurately, tolerating various text transformations such as paragraph additions and deletions, line break format differences, and character modifications. As long as the main content of the license statement remains consistent, even if there are rewrites, cuts, or format changes, it can still be correctly matched to the corresponding standard license, improving robustness and significantly reducing the false negative rate. It overcomes the false negative situation that may occur with a single algorithm, ensuring a high recall rate while reducing the false match rate through dual verification, achieving efficient and fault-tolerant recognition of license text.

[0136] In other words, this invention enables the establishment of a standard license fingerprint database and efficient fault-tolerant matching. Throughout the entire process, regardless of any slight changes in the format or content of the license statement in the source code, it can be correctly identified and mapped to the corresponding standard license type through hash fingerprint similarity, thereby significantly improving the reliability and accuracy of automatic license compliance detection.

[0137] In a specific example, the aforementioned S740 includes:

[0138] S741: Calculate the comprehensive similarity score based on the Jaccard similarity and Hamming distance corresponding to each candidate template.

[0139]

[0140] Where Score represents the overall similarity score, and J represents the Jaccard similarity score. Indicates Hamming distance, and The weighting coefficients are set.

[0141] S742: Candidate templates with a comprehensive similarity score higher than the set comprehensive score threshold are identified as matching standard license templates, and the identification information of the matching standard license templates is used as the license identification result.

[0142] In this step, if the overall similarity score is higher than the preset overall score threshold, the input text is determined to match the corresponding license template. If multiple high-scoring candidates appear, the highest-scoring candidate is output as the final result, or several tied candidates are output for manual review if necessary.

[0143] The output license identification results include the matched license type name, identifier, and version number (e.g., "Apache License 2.0"). If no matching template reaches the comprehensive scoring threshold, a "Not Matched" flag is output, prompting manual review or indicating it as a potential new type of license.

[0144] In one possible implementation, the output license identification result may also include additional information such as a similarity score, which may be calculated based on a comprehensive similarity score (out of 100 or a normalized score), or the comprehensive similarity score may be used directly as the output similarity score.

[0145] The final license identification results can be used to generate compliance reports or drive subsequent processes (such as informing users that the file uses a certain license).

[0146] Some existing open-source compliance scanning tools (such as FOSSology) rely on server-side support, which is inconvenient for scenarios that prioritize local data security or offline environments.

[0147] This invention does not rely on a server, making it suitable for local automation and continuous integration. Specifically, by pre-building a fingerprint database and employing efficient hash comparison, it can handle license comparisons of large-scale files while ensuring accuracy, meeting the performance requirements of local operation in practical engineering. At the algorithm level, Locality Sensitive Hash (LSH) indexing is used to accelerate similarity retrieval, maintaining linear time complexity even for large-scale code scanning. The system design supports parallel processing and caching mechanisms, enabling efficient operation in ordinary computing environments.

[0148] The modular architecture of this invention supports local deployment and lightweight embedding, and can be directly integrated into development toolchains or CI / CD processes to perform real-time license detection and risk alerts. It is not only suitable for enterprise compliance platforms, but can also be embedded in IDEs or build scripts to achieve instant compliance assurance through "development-as-you-go" detection.

[0149] In summary, this invention, by combining abstract syntax tree structure analysis with double-hash fault-tolerant matching technology, significantly improves accuracy, fault tolerance, automation, scalability, and runtime performance. This invention not only adapts to open-source code scenarios in different languages ​​and formats but also enables rapid deployment in local environments, achieving an integrated compliance detection system of "structured extraction + intelligent comparison + highly robust recognition," providing a more intelligent, efficient, and universally applicable technical means for software open-source compliance.

[0150] This invention provides an open-source software license identification device, such as... Figure 2 As shown, the device includes:

[0151] Abstract syntax tree acquisition module 100 is used to receive the abstract syntax tree generated from the source code file. The abstract syntax tree includes the structure nodes and comment nodes of the source code.

[0152] The candidate annotation block filtering module 200 is used to traverse each annotation node, collect all annotation blocks, and perform preliminary filtering on the collected annotation blocks to obtain candidate annotation blocks.

[0153] The multi-dimensional comprehensive scoring module 300 is used to perform multi-dimensional comprehensive scoring based on the position of candidate annotation blocks, annotation type, license keywords, and full-text semantic aggregation to obtain a credibility score.

[0154] The license annotation block determination module 400 is used to determine candidate annotation blocks with a credibility score higher than the set credibility threshold as license annotation blocks.

[0155] The text normalization module 500 is used to normalize the text of the license annotation block to obtain the normalized license text.

[0156] The hash fingerprint calculation module 600 is used to calculate the hash fingerprint of the license normalization text.

[0157] The fuzzy matching module 700 is used to perform fuzzy matching comparison between the calculated hash fingerprint and the pre-built standard license hash fingerprint library to obtain the license identification result corresponding to the license annotation block.

[0158] As an improvement to this embodiment of the invention, the multi-dimensional comprehensive scoring module includes:

[0159] The position evaluation unit is used to assign a corresponding position score based on the position of the candidate comment block in the source code file.

[0160] The type evaluation unit is used to assign a type score based on the form and scope of the candidate annotation block.

[0161] The keyword evaluation unit is used to assign corresponding keyword scores based on the typical license keywords contained in the candidate annotation blocks.

[0162] The text aggregation evaluation unit is used to aggregate all lines of the candidate annotation block into a complete full-text text, and to perform semantic and format analysis on the full-text text, assigning a corresponding text aggregation score.

[0163] The weighted calculation unit is used to calculate the credibility score by weighting the location score, type score, keyword score, and text aggregation score.

[0164] Regarding the aforementioned text normalization module, it is specifically used for:

[0165] Remove comment markers and meaningless characters from the license comment block, retain the pure license text content, and perform formatting adjustments, including merging and line breaks. While maintaining the original semantics of the license comment block text content, standardize the format to obtain the license text.

[0166] The hash fingerprint of the license normalization text described above in this invention includes a MinHash signature and a SimHash fingerprint value. Correspondingly, the standard license hash fingerprint database includes the identification information of known standard license templates, the MinHash signature of the text corresponding to the standard license template, and the SimHash fingerprint value.

[0167] This standard license hash fingerprint library is pre-built in the following way:

[0168] Collect known standard license templates and perform normalization preprocessing on the text of the standard license templates to remove irrelevant format differences.

[0169] The normalized preprocessed text is divided into several element sets, and multiple sets of hash functions are calculated for each element set to form a set of MinHash signatures; the normalized preprocessed text is then vectorized into feature vectors and SimHash values ​​are calculated.

[0170] The MinHash signature and SimHash value are mapped to the identification information of the standard license template and stored in the standard license hash fingerprint library.

[0171] As another improvement to this embodiment of the invention, the fuzzy matching module includes:

[0172] The Jaccard similarity calculation unit is used to calculate the Jaccard similarity based on the MinHash signature of the license canonical text and the MinHash signature of the standard license template.

[0173] The candidate template unit is used to identify standard license templates with a Jaccard similarity higher than the set similarity threshold as candidate templates.

[0174] Hamming distance calculation unit is used to calculate Hamming distance based on the SimHash fingerprint value of the license normalization text and the SimHash fingerprint value of the standard license template.

[0175] The matching unit is used to determine the matching standard license template based on the Jaccard similarity and Hamming distance corresponding to each candidate template, and obtain the license recognition result.

[0176] In one example, the matching unit includes:

[0177] The comprehensive similarity score calculation subunit is used to calculate the comprehensive similarity score based on the Jaccard similarity and Hamming distance corresponding to each candidate template.

[0178]

[0179] Where Score represents the overall similarity score, and J represents the Jaccard similarity score. Indicates Hamming distance, and The weighting coefficients are set.

[0180] The comprehensive similarity score comparison sub-unit is used to identify candidate templates with a comprehensive similarity score higher than the set comprehensive score threshold as matching standard license templates, and to use the identification information of the matching standard license templates as the license identification result.

[0181] The apparatus provided in the above embodiments corresponds one-to-one with the embodiments of the aforementioned methods in terms of its implementation principle and the resulting technical effects. For the sake of brevity, any parts of the apparatus not mentioned in the embodiments can be referred to the corresponding content in the embodiments of the aforementioned methods. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the modules and units described in this apparatus can all be referred to the corresponding processes in the embodiments of the aforementioned methods, and will not be repeated here.

[0182] The open-source software license identification method described in the above embodiments of the present invention can implement business logic through a computer program and record it on a storage medium. This storage medium can be read and executed by a computer, achieving the effects of the scheme described in the method embodiments of this specification. Therefore, the embodiments of the present invention also provide a computer-readable storage medium for open-source software license identification, including a memory for storing processor-executable instructions. When these instructions are executed by a processor, they implement the steps of the open-source software license identification method of the foregoing embodiments.

[0183] The storage medium may include a physical device for storing information, typically digitizing the information and then storing it using electrical, magnetic, or optical methods. The storage medium may include: devices that store information using electrical energy, such as various types of memory, like RAM and ROM; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, bubble memory, and USB flash drives; and devices that store information using optical methods, such as CDs or DVDs. Of course, there are other readable storage media, such as quantum memories and graphene memories.

[0184] The storage medium described above may also include other implementation methods according to the description of the method embodiments. The implementation principle and technical effects of this embodiment are the same as those of the foregoing method embodiments. For details, please refer to the description of the relevant method embodiments, which will not be repeated here.

[0185] This invention also provides a device for identifying open-source software licenses. The device can be a standalone computer, or it can include an actual operating device that uses one or more of the methods or embodiments described in this specification. The device for identifying open-source software licenses may include at least one processor and a memory storing computer-executable instructions. When the processor executes the instructions, it implements the steps of any one or more of the above-described open-source software license identification methods.

[0186] The device described above may also include other implementation methods according to the method embodiments. The implementation principle and technical effects of this embodiment are the same as those of the foregoing method embodiments. For details, please refer to the description of the relevant method embodiments, which will not be repeated here.

[0187] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.

Claims

1. A method for identifying open-source software licenses, characterized in that, The method includes: Receive an abstract syntax tree generated from parsing a source code file, the abstract syntax tree including structural nodes and comment nodes of the source code; Traverse each comment node, collect all comment blocks, and perform preliminary screening on the collected comment blocks to obtain candidate comment blocks; A credibility score is obtained by comprehensively evaluating the candidate annotation blocks based on their location, annotation type, license keywords, and full-text semantic aggregation. Candidate annotation blocks whose credibility scores are higher than the set credibility threshold are identified as license annotation blocks; The license annotation block is text normalized to obtain the license normalized text; Calculate the hash fingerprint of the license normalized text; The calculated hash fingerprint is compared with a pre-built standard license hash fingerprint library using fuzzy matching to obtain the license identification result corresponding to the license annotation block.

2. The open-source software license identification method according to claim 1, characterized in that, The credibility score is obtained by performing a multi-dimensional comprehensive evaluation based on the position, annotation type, license keywords, and full-text semantic aggregation of the candidate annotation blocks, including: The candidate comment blocks are assigned corresponding position scores based on their positions in the source code file. The candidate annotation blocks are assigned corresponding type scores based on their form and scope. Assign corresponding keyword scores based on the typical license keywords contained in the candidate annotation blocks; All lines of the candidate annotation block are aggregated into a complete full-text text, and the full-text text is subjected to semantic and format analysis, and a corresponding text aggregation score is assigned. The credibility score is obtained by weighting the location score, type score, keyword score, and text aggregation score.

3. The open-source software license identification method according to claim 1, characterized in that, The text normalization of the license annotation block to obtain the license normalized text includes: Remove comment markers and meaningless characters from the license comment block, retain the pure license text content, and perform formatting adjustments including merging and line breaks. While maintaining the original semantics of the license comment block text content, unify the format to obtain the standardized license text.

4. The open-source software license identification method according to any one of claims 1-3, characterized in that, The hash fingerprint of the license normalized text includes a MinHash signature and a SimHash fingerprint value. The standard license hash fingerprint library includes the identification information of known standard license templates, the MinHash signature of the text corresponding to the standard license template, and the SimHash fingerprint value.

5. The open-source software license identification method according to claim 4, characterized in that, The standard license hash fingerprint database is pre-built in the following manner: Collect known standard license templates and perform normalization preprocessing on the text of the standard license templates to remove irrelevant format differences; The normalized preprocessed text is divided into several element sets, and multiple sets of hash functions are calculated for each element set to form a set of MinHash signatures; the normalized preprocessed text is vectorized into features and SimHash values ​​are calculated; The MinHash signature and SimHash value are mapped to the identification information of the standard license template and stored in the standard license hash fingerprint database.

6. The open-source software license identification method according to claim 5, characterized in that, The step of performing a fuzzy matching comparison between the calculated hash fingerprint and a pre-built standard license hash fingerprint library to obtain the license identification result corresponding to the license annotation block includes: Calculate the Jaccard similarity based on the MinHash signature of the license canonical text and the MinHash signature of the standard license template; The standard license templates with Jaccard similarity scores higher than the set similarity threshold are identified as candidate templates; The Hamming distance is calculated based on the SimHash fingerprint value of the license normalized text and the SimHash fingerprint value of the standard license template. The matching standard license template is determined based on the Jaccard similarity and Hamming distance corresponding to each candidate template, and the license recognition result is obtained.

7. The open-source software license identification method according to claim 6, characterized in that, The process of determining the matching standard license template based on the Jaccard similarity and Hamming distance corresponding to each candidate template to obtain the license recognition result includes: A comprehensive similarity score is calculated based on the Jaccard similarity and Hamming distance corresponding to each candidate template. Where Score represents the overall similarity score, and J represents the Jaccard similarity score. Indicates Hamming distance, and The weighting coefficients are set; Candidate templates whose comprehensive similarity score is higher than the set comprehensive score threshold are identified as matching standard license templates, and the identification information of the matching standard license templates is used as the license identification result.

8. An open-source software license identification device, characterized in that, The device includes: Abstract syntax tree acquisition module, used to receive an abstract syntax tree generated from parsing source code files, the abstract syntax tree including structural nodes and comment nodes of source code; The candidate annotation block filtering module is used to traverse each annotation node, collect all annotation blocks, and perform preliminary filtering on the collected annotation blocks to obtain candidate annotation blocks. The multi-dimensional comprehensive scoring module is used to perform multi-dimensional comprehensive scoring based on the position, annotation type, license keywords and full-text semantic aggregation of the candidate annotation block to obtain a credibility score. The license annotation block determination module is used to determine the candidate annotation blocks whose credibility scores are higher than the set credibility threshold as license annotation blocks; The text normalization module is used to normalize the license annotation block to obtain the license normalized text. A hash fingerprint calculation module is used to calculate the hash fingerprint of the license normalization text; The fuzzy matching module is used to perform fuzzy matching comparison between the calculated hash fingerprint and a pre-built standard license hash fingerprint library to obtain the license identification result corresponding to the license annotation block.

9. A computer-readable storage medium for identifying open-source software licenses, characterized in that, It includes a memory for storing processor-executable instructions, which, when executed by the processor, implement the steps of the open-source software license identification method according to any one of claims 1-7.

10. A device for identifying open-source software licenses, characterized in that, It includes at least one processor and a memory storing computer-executable instructions, wherein the processor, when executing the instructions, implements the steps of the open-source software license identification method according to any one of claims 1-7.

Images