Open source license conflict detection methods, devices, electronic devices and storage media

Abstract

This application provides an open-source license conflict detection method, apparatus, electronic device, and storage medium. The method includes: obtaining the original text and license type of a sample license based on user-inputted technology stack information; preprocessing the original text of the sample license to obtain a first text feature; generating a license rule base based on the first text feature and license type; performing feature matching between the license to be processed and the license rule base to determine the initial license type of the license to be processed; and verifying the license type of the license to be processed based on a pre-trained question-answering model to determine the target license type of the license to be processed. Applying this method, the original text of licenses can be automatically obtained from multiple source platforms through an interface, solving the problem of data dispersion and improving data processing efficiency. Multi-layer semantic analysis technology is used to decompose the original text to obtain stronger semantic analysis capabilities, enhancing recognition accuracy. A dynamic optimization mechanism further improves performance.

Description

Technical Field

[0001] This application relates to the field of license detection, and more particularly to an open-source license conflict detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Open source license data is distributed across multiple platforms, including the Software Package Data Exchange (SPDX) and the Open System Interconnection Reference Model (OSI). Manually collecting and integrating open source licenses is time-consuming and labor-intensive. Furthermore, license terms contain numerous legal terms, making efficient parsing difficult for traditional tools after obtaining the original license text. Existing solutions still rely on static rules for license data matching, failing to handle semantic variations such as synonyms. Moreover, existing tools lack real-time monitoring and rule optimization for license term and version updates, making it difficult to adapt to the dynamic changes in the open source ecosystem. Summary of the Invention

[0003] This application provides an open-source license conflict detection method, apparatus, electronic device, and storage medium to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this application, an open-source license conflict detection method is provided. The method includes: acquiring technology stack information input by a user; acquiring basic information of a sample license based on the technology stack information, the basic information including the original text of the sample license and the license type of the sample license, the license type including open license, weakly infectious license, infectious license, and strongly infectious license; preprocessing the original text of the sample license to obtain a first text feature of the sample license; generating a license rule base based on the first text feature of the sample license and the license type of the sample license; acquiring a license to be processed; performing feature matching between the license to be processed and the license rule base to determine the initial license type of the license to be processed; and reviewing the license type of the license to be processed based on a pre-trained question-answering model to determine the target license type of the license to be processed.

[0005] In one embodiment, the preprocessing of the original text of the sample license to obtain the first text feature of the sample license includes: removing invalid content from the original text using a preset regular expression and standardizing the original text of the sample license to obtain standardized license text; performing word segmentation on the standardized license text; removing stop words based on the result of the word segmentation to obtain keyword features of the sample license; performing sentence segmentation on the standardized license text to obtain sentence features of the sample license; and generating phrases from the standardized license text to obtain phrase features of the sample license. The first text feature of the sample license includes the keyword features, sentence features, and phrase features of the sample license.

[0006] In one possible implementation, generating a license rule base based on the first text features of the sample license and the license type of the sample license includes: mapping the first text features of the sample license to generate license rules based on mapping rules; generating a license rule base based on the license rules, the first text features of the sample license, and the license type of the sample license, wherein the license rule base includes each license rule, a corresponding keyword list, a sentence list, a phrase list, and a license type; vectorizing the first text features of the sample license to obtain the first vector features of the sample license, wherein the first vector features include keyword vectors, sentence vectors, and phrase vectors of the sample license; selecting text features whose cosine similarity with the first vector features meets a preset cosine similarity threshold and / or whose fuzzy matching with the first text features meets a fuzzy matching threshold as candidate text features; and adding the candidate text features to the corresponding positions in the license rule base.

[0007] In one possible implementation, the step of performing feature matching between the license to be processed and the license rule base to determine the initial license type of the license to be processed includes: performing feature processing on the license to be processed to obtain a second text feature and a second vector feature of the license to be processed; matching the second text feature and the second vector feature with license rules in the license rule base, and determining the license rules that satisfy the cosine similarity threshold and the fuzzy matching threshold as target license rules; determining the weight value of the license to be processed based on the number of target license rules, the cosine similarity between the license to be processed and the target license rules, and the priority of the target license rules; and determining the initial license type of the license to be processed based on the weight value.

[0008] In one embodiment, the method further includes: obtaining the verification result of the pre-trained question-answering model verifying the license type of the license to be processed; comparing the verification result with the initial license type to obtain a comparison result; and dynamically adjusting the priority of the license rules according to the comparison result.

[0009] In one embodiment, the method further includes: when it is determined that the target license type of the license to be processed is an infectious license or a highly infectious license, obtaining a human judgment result on the license type of the license to be processed; and correcting the license type of the license to be processed based on the judgment result.

[0010] According to a second aspect of this application, an open-source license conflict detection device is provided. The device includes: an acquisition module, configured to acquire user-inputted technology stack information and, based on the technology stack information, acquire basic information of a sample license, the basic information including the original text of the sample license and the license type of the sample license, the license type including open license, weakly infectious license, infectious license, and strongly infectious license; a preprocessing module, configured to preprocess the original text of the sample license to obtain a first text feature of the sample license; a generation module, configured to generate a license rule base based on the first text feature of the sample license and the license type of the sample license; the acquisition module is further configured to acquire a license to be processed; a matching module, configured to perform feature matching between the license to be processed and the license rule base to determine the initial license type of the license to be processed; and a verification module, configured to verify the license type of the license to be processed based on a pre-trained question-answering model to determine the target license type of the license to be processed.

[0011] In one embodiment, the preprocessing module is specifically used to remove invalid content from the original text using a preset regular expression, and to standardize the original text of the sample license to obtain standardized license text; to perform word segmentation on the standardized license text; to remove stop words based on the result of the word segmentation to obtain keyword features of the sample license; to perform sentence segmentation on the standardized license text to obtain sentence features of the sample license; and to generate phrases from the standardized license text to obtain phrase features of the sample license; the first text feature of the sample license includes the keyword features, sentence features, and phrase features of the sample license.

[0012] In one embodiment, the generation module includes: a mapping submodule, configured to generate license rules by mapping the first text features of the sample license based on mapping rules; a generation submodule, configured to generate a license rule library based on the license rules, the first text features of the sample license, and the license type of the sample license, wherein the license rule library includes each license rule, a corresponding keyword list, a sentence list, a phrase list, and a license type; a first processing submodule, configured to vectorize the first text features of the sample license to obtain the first vector features of the sample license, wherein the first vector features include keyword vectors, sentence vectors, and phrase vectors of the sample license; an acquisition submodule, configured to select text features whose cosine similarity with the first vector features meets a preset cosine similarity threshold and / or whose fuzzy matching with the first text features meets a fuzzy matching threshold as candidate text features; and an addition submodule, configured to add the candidate text features to the corresponding positions in the license rule library.

[0013] In one embodiment, the matching module includes: a second processing submodule, configured to perform feature processing on the license to be processed to obtain a second text feature and a second vector feature of the license to be processed; a matching submodule, configured to match the second text feature and the second vector feature with license rules in a license rule base, and determine the license rules that satisfy the cosine similarity threshold and the fuzzy matching threshold as target license rules; a determining submodule, configured to determine the weight value of the license to be processed based on the number of target license rules, the cosine similarity between the license to be processed and the target license rules, and the priority of the target license rules; the determining submodule is further configured to determine the initial license type of the license to be processed based on the weight value.

[0014] In one embodiment, the device further includes: an adjustment module, configured to obtain the verification result of the pre-trained question-answering model verifying the license type of the license to be processed; compare the verification result with the initial license type to obtain a comparison result; and dynamically adjust the priority of the license rules according to the comparison result.

[0015] In one embodiment, the device further includes: a correction module, configured to, when it is determined that the target license type of the license to be processed is an infectious license or a highly infectious license, obtain a human judgment result on the license type of the license to be processed; and correct the license type of the license to be processed based on the judgment result.

[0016] According to a third aspect of this application, an electronic device is provided, comprising:

[0017] At least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in this application.

[0019] According to a fourth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods described in this application.

[0020] This application discloses an open-source license conflict detection method, apparatus, electronic device, and storage medium. First, it acquires user-inputted technology stack information. Based on this information, it obtains basic information about a sample license, including the original text of the sample license and its license type (open license, weakly infectious license, infectious license, and strongly infectious license). Next, it preprocesses the original text of the sample license to obtain its first textual features, and generates a license rule base based on these features and the license type. Then, it acquires the license to be processed, performs feature matching between the license and the license rule base, determines its initial license type, and finally, uses a pre-trained question-answering model to verify the license type and determine its target license type.

[0021] By applying this method, the original license text can be automatically obtained from multiple source platforms through the interface, which solves the problem of data dispersion and improves the efficiency of data processing. Through multi-layer semantic analysis technology, keywords, sentences and phrases are split to obtain stronger semantic analysis capabilities. When matching, matching is performed from multiple dimensions, which enhances robustness and recognition accuracy. Furthermore, through a dynamic optimization mechanism, performance is further improved.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0023] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0024] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0025] Figure 1This paper illustrates the implementation flow of an open-source license conflict detection method according to an embodiment of this application. Figure 1 ;

[0026] Figure 2 This paper illustrates the implementation flow of an open-source license conflict detection method according to an embodiment of this application. Figure 2 ;

[0027] Figure 3 This paper illustrates the implementation flow of an open-source license conflict detection method according to an embodiment of this application. Figure 3 ;

[0028] Figure 4 This paper illustrates the implementation flow of an open-source license conflict detection method according to an embodiment of this application. Figure 4 ;

[0029] Figure 5 This paper shows a schematic diagram of a module of an open-source license conflict detection device according to an embodiment of this application;

[0030] Figure 6 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0031] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] Figure 1 This paper illustrates the implementation flow of an open-source license conflict detection method according to an embodiment of this application. Figure 1 ,include:

[0033] Step 101: Obtain the technology stack information input by the user, and obtain the basic information of the sample license based on the technology stack information. The basic information includes the original text of the sample license and the license type of the sample license. The license type includes open license, weakly infectious license, infectious license and strongly infectious license.

[0034] The technology stack refers to the collection of all open-source software, libraries, frameworks, and their versions that a project depends on. Users submit the open-source software components used in their project and their associated information to the system, including the following: component name, version number, dependency source, license type, project structure relationship, etc., or the technology stack information entered by the user can be automatically identified by analysis tools.

[0035] Based on the technology stack information, basic information about the sample license is obtained. This basic information includes the original text of the sample license, the license type, the summary of the sample license, and the identifier of the sample license. Among them, the license type is divided into open license, weakly infectious license, infectious license, and strongly infectious license according to its infectivity, which correspond to four levels: low risk, medium risk, high risk, and ultra-high risk, respectively.

[0036] This application utilizes the Application Programming Interface (API) of open-source license databases or web scraping to query the license identifier corresponding to each component. Licenses are obtained from official license repositories such as the Software Package Data Exchange (SPDX) and the Open Systems Interconnection Reference Model (OSI), component repositories, or directly from code repositories like GitHub. The data collection method employed in this application via web scraping is implemented in compliance with the target website's terms of service and robots.txt protocol, and all collected data consists of publicly accessible, non-sensitive information.

[0037] Step 102: Preprocess the original text of the sample license to obtain the first text feature of the sample license.

[0038] The original text of the sample licenses undergoes preprocessing, including text cleaning, format standardization, and feature extraction, to obtain the first textual features of the licenses. Since the original license texts are often long paragraphs, direct processing makes it difficult to capture key information. Preprocessing, through steps such as word segmentation, sentence splitting, and phrase generation, breaks down unstructured text into structured units, allowing for more accurate capture of the core semantics of the licenses. Preprocessing effectively cleanses the data, standardizes the format, and focuses on key information, reducing computational costs and improving data processing efficiency, leading to more accurate subsequent rule generation and classification.

[0039] Step 103: Generate a license rule base based on the first textual features of the sample license and the license type of the sample license.

[0040] The generated first text feature is mapped to a predefined rule template to obtain the license rule corresponding to the first text feature. The license rule, the corresponding first text feature and the license type of the sample license are stored in the database to generate a license rule library. This license rule library can be used for rule matching of subsequent licenses to be processed.

[0041] Step 104: Obtain the license to be processed.

[0042] Step 105: Perform feature matching between the license to be processed and the license rule base to determine the initial license type of the license to be processed.

[0043] Licenses to be processed are obtained through user uploads or system scanning. These licenses also undergo preprocessing and feature extraction. Based on the extracted features, the licenses are matched against a license rule base using methods such as cosine similarity matching and fuzzy matching. The initial license type is determined based on the matching results.

[0044] Step 106: Based on the pre-trained question-answering model, review the license type of the license to be processed and determine the target license type of the license to be processed.

[0045] The pre-trained question-answering model verifies the license type of the license to be processed. The original text of the license to be processed and the contents of the license rule base are input into the question-answering model. After the question-answering model is trained, it outputs the license type of the license to be processed in the form of questions and answers, so as to realize the verification of the initial license type. Based on the verification results, the final target license type of the license to be processed is determined.

[0046] The question-and-answer model can also be used to answer other questions related to the license, such as "Is commercial use permitted?"

[0047] This approach establishes compatibility rules for different license types. For example, open licenses are compatible with any type of license, while highly contagious licenses may conflict with other types. It checks the compatibility of multiple licenses within a project and identifies potential conflicts. By applying this method, license texts are automatically retrieved from multiple platforms via an interface, resolving the data fragmentation issue and improving data processing efficiency. Multi-layer semantic analysis is used to decompose the original texts, resulting in enhanced semantic analysis capabilities. Sample licenses are analyzed to generate a license rule base. Matching licenses to be processed against this rule base allows for faster and more accurate determination of the license type. A dynamic optimization mechanism further enhances performance.

[0048] In one possible implementation, such as Figure 2 As shown, the original text of the sample license is preprocessed to obtain the first textual features of the sample license, including:

[0049] Step 201: Remove invalid content from the original text using a preset regular expression, and standardize the original text of the sample license to obtain the standardized license text.

[0050] Step 202: Perform word segmentation on the standardized license text;

[0051] Step 203: Remove stop words based on the results of word segmentation to obtain the keyword features of the sample license;

[0052] Step 204: Segment the standardized license text into sentences to obtain the sentence features of the sample license;

[0053] Step 205: Phrase generation is performed on the standardized license text to obtain the phrase features of the sample license; the first text features of the sample license include keyword features, sentence features and phrase features.

[0054] After obtaining the original sample license text, it undergoes preprocessing. This preprocessing primarily involves standardizing the format of the original license text and extracting key information. Specifically, this includes: firstly, removing invalid content such as copyright notices and contact information using predefined regular expressions. For example, regular expressions can be used to remove the copyright line (e.g., "Copyright (c) 2023..."); and removing Uniform Resource Locators (URLs), email addresses, and special characters. Then, the license text, after removing invalid content, is standardized by encoding it uniformly to UTF-8 to ensure multilingual compatibility, standardizing the case of all characters, eliminating case differences, and normalizing line breaks to ensure consistent text formatting, resulting in a standardized license text.

[0055] The standardized license text is then processed to obtain the text features of the sample license. This application uses a Natural Language Toolkit (NLTK) to process the standardized license text, employing the word tokenization method in NLTK to segment the standardized license text into individual words. For example, the sentence "Permission is hereby granted" is segmented into ["permission", "is", "hereby", "granted"]. Then, based on the segmentation results, stop words that are meaningless for text analysis and information retrieval, as well as those frequently ignored in text processing and lacking actual semantic meaning, such as "is" in the above example, are removed. After removing stop words, the keyword features of the sample license are obtained ["permission", "granted"]. Furthermore, the punkt sentence segmentation method in NLTK is used to segment the standardized license text into sentences, preserving the logical relationships between sentences to obtain the sentence features of the sample license, such as ["Permission is hereby granted."]. Phrase extraction was performed using the collocations method in NLKT to obtain phrase features of the sample license, including extracted two-word phrases and syntactic phrases. The keyword features, sentence features, and phrase features of the sample license obtained after preprocessing constitute the first text features of the sample license. These first text features include the three core clauses of the sample license: rights, conditions, and restrictions.

[0056] In one possible implementation, such as Figure 3 As shown, a license rule base is generated based on the first textual features of the sample license and the license type of the sample license, including:

[0057] Step 301: Generate license rules by mapping the first text features of the sample license based on the mapping rules;

[0058] Step 302: Generate a license rule base based on the license rules, the first text features of the sample license, and the license type of the sample license. The license rule base includes each license rule, the corresponding keyword list, sentence list, phrase list, and license type.

[0059] Step 303: Vectorize the first text features of the sample license to obtain the first vector features of the sample license. The first vector features include the keyword vector, sentence vector and phrase vector of the sample license.

[0060] Step 304: Select text features whose cosine similarity with the first vector features meets a preset cosine similarity threshold and / or whose fuzzy matching with the first text features meets a fuzzy matching threshold as candidate text features.

[0061] Step 305: Add the candidate text features to the corresponding positions in the license rule base.

[0062] In the process of generating the license rule base, the first step is to map the first text features of the sample license based on the mapping rules to generate license rules. Since the first text features include keyword features, sentence features and phrase features, the mapping rules also include keyword mapping, sentence mapping and phrase mapping.

[0063] Keyword mapping associates the keyword features of the sample license with predefined mapping rules. For example, "permission" maps to "permission to use"; "copyleft" maps to "mandatory open source". Sentence mapping matches the sentence features of the sample license with preset sentence templates. For example, the sentence template "[Subject] is hereby granted" matches to "grant [Subject] rights"; the sentence "You must make the source code available" matches to "mandatory public release of source code". Phrase mapping maps the phrase features of the sample license to specific details, such as "free of charge" mapping to "free use"; "must disclose" mapping to "must disclose source code".

[0064] The generated license rules are integrated with the corresponding first text features and the license types of sample licenses to form a structured license rule base. Each rule in this license rule base includes: a license rule description, a list of associated keywords, a list of associated sentences, a list of associated phrases, and a license type. This structured license rule base is then stored after generation.

[0065] The first text features of the sample license are vectorized. Specifically, the keyword features are vectorized using Term Frequency-Inverse Document Frequency (TF-IDF) to obtain keyword vectors. The sentence features and phrase features are converted into 384-dimensional sentence vectors and phrase vectors using the SentenceTransformer. Converting the keyword features, sentence features and phrase features into corresponding vector features can capture the deep semantics of keywords, sentences and phrases.

[0066] Some keywords, phrases, and sentences can be replaced with synonyms to obtain texts with similar meanings to the aforementioned keywords, sentences, and phrases. These texts with similar meanings are then matched with the first vector feature in the license rule base for similarity. Text features whose cosine similarity with the first vector feature meets a preset cosine similarity threshold are used as candidate text features. The preset cosine similarity can be set according to the actual situation. For example, text features with a cosine similarity greater than or equal to 0.85 with the first vector feature can be used as candidate text features. For example, the cosine similarity between the phrase "at no cost" and "free ofcharge" in the license rule base is 0.92, so the phrase "at no cost" is determined to be a candidate phrase feature. In some cases, the texts may have the same meaning but differ in format due to writing. Therefore, these texts can be fuzzily matched with the first vector features. Text features whose fuzzy match with the first text features meets the fuzzy matching threshold are also considered as candidate text features. Similarly, the fuzzy matching threshold can be set according to the actual situation. For example, if the fuzzy match degree between the keyword "sourcecode" to be analyzed and "source code" in the rule base is 85%, the keyword "sourcecode" is determined to be a candidate keyword feature.

[0067] Candidate text features can be added to the corresponding rule entries in the license rule base to enhance the rule coverage; for example, "at no cost" can be added to the keyword list corresponding to "free of charge". Furthermore, the vector model can be retrained periodically to update the vector representations in the license rule base, ensuring the accuracy of semantic matching.

[0068] In one possible implementation, such as Figure 4 As shown, the license to be processed is matched with the license rule base by feature matching to determine the initial license type of the license to be processed, including:

[0069] Step 401: Perform feature processing on the license to be processed to obtain the second text feature and the second vector feature of the license to be processed.

[0070] Step 402: Match the second text feature and the second vector feature with the license rules in the license rule base, and determine the license rules that meet the cosine similarity threshold and the fuzzy matching threshold as the target license rules;

[0071] Step 403: Determine the weight value of the license to be processed based on the number of target license rules, the cosine similarity between the license to be processed and the target license rules, and the priority of the target license rules.

[0072] Step 404: Determine the initial license type of the license to be processed based on the weight value.

[0073] Based on the license rule base, it is possible to determine the license type of unknown licenses. First, the license to be processed is obtained, and the license to be processed is preprocessed in the same way as the sample license, namely standardization, analysis, removal of stop words, sentence segmentation and phrase generation, etc., to obtain the second text feature of the license to be processed. Then, the second text feature is vectorized to obtain the second vector feature of the license to be processed. The second vector feature includes the keyword vector, sentence vector and phrase vector corresponding to the license to be processed.

[0074] The second text feature and the second vector feature are matched with the first text feature and the first vector feature of each rule in the license rule base. The cosine similarity between the second vector feature and the first vector feature is calculated, and the second text feature and the first text feature are then subjected to fuzzy matching. License rules whose cosine similarity between the second vector feature and the first vector feature meets the cosine similarity threshold and whose fuzzy matching between the second text feature and the first text feature meets the fuzzy matching threshold are determined as target license rules. For example, the sentence "Permission to use is hereby granted" in the license to be processed has a cosine similarity of 0.92 with the "Allow free use" rule in the rule base, and a fuzzy matching degree of 88%. This "Allow free use" rule is determined as the target license rule.

[0075] The weight value of the license to be processed is determined based on the number of target license rules, the cosine similarity between the license to be processed and the target license rules, and the priority of the target license rules. Specifically, it is calculated using the following formula: ;in Indicates the number of target license rules. Let represent the cosine similarity between the license to be processed and the i-th target license rule. This indicates the priority of the i-th target license rule. This represents the weight value of the license to be processed. The cosine similarity between the license to be processed and each target license rule is used as one of the weighting factors; the higher the similarity, the greater the impact on the weight value. The preset rule priority in the license rule base is also used as one of the weighting factors; rules with higher priority have a greater impact on the weight value. Rule priorities can be designed according to actual conditions. For example, the priority of rules corresponding to open licenses is 0.4, and the priority of rules corresponding to infectious licenses is 0.8. Generally, the higher the infectivity, the greater the priority.

[0076] The initial license type for the pending license is ultimately determined based on the calculated weight value of the pending license. Specifically, A value <0.6 indicates a low level of risk for the license, corresponding to an open license type. A value <0.75 indicates a moderate level of risk for the license, corresponding to a weakly infectious license type. 0.75 ≤ A value <0.9 indicates a high level of risk for the license, corresponding to an infectious disease license. A value ≥0.9 indicates a very high level of danger for the license, corresponding to a highly contagious license type. Therefore, the initial license type for a license to be processed can be determined based on which weight range its weight value falls into. The correspondence between the license's danger level and the weight value range can be determined based on actual circumstances or experience.

[0077] This application employs a multi-dimensional matching process when matching pending licenses with license rules, considering keywords, sentences, and phrases simultaneously. The matching is prioritized according to the order of sentence, phrase, and keyword, thus improving the accuracy of determining the license type for pending licenses. Through this process, automated matching of pending licenses with the license rule base can be achieved, accurately determining the initial license type and providing a foundation for subsequent conflict detection and compliance analysis.

[0078] In one embodiment, the method further includes: obtaining the verification result of the pre-trained question-answering model verifying the license type of the license to be processed; comparing the verification result with the initial license type to obtain the comparison result; and dynamically adjusting the priority of the license rules according to the comparison result.

[0079] A pre-trained question-answering model is used to verify the accuracy of the license type determination for the license to be processed. The question-answering model generates targeted questions, such as "Does this license require derivative works to be open source?" and "Does it allow commercial use?", extracts the answers from the original text of the license to be processed, and then performs semantic matching between the answers and the features of the corresponding license type in the license rule base, such as "infectious types require mandatory open source", to generate a verification result of the license type of the license to be processed.

[0080] The initial license type is compared with the review result. If the comparison result indicates that the error between the review result and the initial license type is within the error threshold, then the initial license type is the license type closest to the license to be processed. If the comparison result indicates that the error between the review result and the initial license type exceeds the error threshold, then the initial license type is not the license type closest to the license to be processed.

[0081] Based on the comparison results, the priority of license rules in the license rule base is dynamically adjusted through a reinforcement learning mechanism. When the error between the comparison result indicating the verification result and the initial license type is within the error threshold range, it indicates that the corresponding target license rule is effective in this judgment, and its priority is increased. Specifically, the priority can be adjusted according to the rule's contribution to the judgment. For example, if the initial license type is infectious and the verification result confirms that infectiousness is correct, the priority of the corresponding license rule can be increased from 0.8 to 0.85. When the error between the comparison result indicating the verification result and the initial license type exceeds the error threshold range, it indicates that the corresponding target license rule has a deviation in this judgment, and its priority is decreased. Specifically, the priority can be adjusted according to the degree of deviation of the rule. For example, if the initial license type is "open" and the verification result is "weakly infectious," the priority of the corresponding license rule can be decreased from 0.4 to 0.3.

[0082] After dynamic adjustment, the priority of the corresponding rule in the license rule base is updated, and the comparison results and adjustment records are fed back to the reinforcement learning model to optimize subsequent priority adjustment strategies. Simultaneously, the adjusted rules are applied to the next license matching, forming a closed loop of "matching-reviewing-adjusting-rematching," continuously improving the accuracy of the license rule base. Through semantic verification by the question-answering model and dynamic priority adjustment, the license rule base can undergo deep optimization. Correct rules, due to their high frequency and effectiveness, receive higher priority, while incorrect rules, due to their inefficiency, have lower priority, ultimately achieving a continuous improvement in the accuracy of license type determination and reducing the cost of manual intervention.

[0083] In one embodiment, the method further includes: when it is determined that the target license type of the license to be processed is an infectious license or a highly infectious license, obtaining a manual judgment result of the license type of the license to be processed; and correcting the license type of the license to be processed based on the judgment result.

[0084] When the automated process determines that the target license type for the license to be processed is an infectious or highly infectious license, a manual review is automatically triggered for secondary confirmation to ensure the accuracy of the judgment. The license text to be processed and the judgment basis are pushed to a professional team or open-source compliance experts for manual review to obtain the manual judgment result on the license type of the license to be processed. If the manual judgment result is consistent with the initial license type, the original judgment is maintained; if they are inconsistent, the manual judgment result shall prevail and the license type of the license to be processed shall be corrected.

[0085] In addition, the results of human judgment are used as training data to adjust the question-answering model and improve its ability to identify high-risk clauses; and if human review finds key clauses that the system has not identified, they are added to the license rule base and the priority of the corresponding rules is adjusted.

[0086] Understandably, if the initial permission type is inconsistent with the verification result of the question-and-answer model, manual judgment can also be made.

[0087] Licenses may change with version updates. The Long Short-Term Memory (LSTM) network model can be used to detect license version updates and feed the detection results back to the license rule base, so that the license rule base can be automatically updated accordingly to ensure compatibility with new license versions.

[0088] Figure 5 A schematic diagram of a module of an open-source license conflict detection device according to an embodiment of this application is shown.

[0089] See Figure 5 According to a second aspect of this application, an open-source license conflict detection device is provided. The device includes: an acquisition module 501, used to acquire user-input technology stack information and acquire basic information of a sample license based on the technology stack information. The basic information includes the original text of the sample license and the license type of the sample license, including open license, weakly infectious license, infectious license, and strongly infectious license; a preprocessing module 502, used to preprocess the original text of the sample license to obtain the first text feature of the sample license; a generation module 503, used to generate a license rule base based on the first text feature of the sample license and the license type of the sample license; the acquisition module 501 is also used to acquire a license to be processed; a matching module 504, used to perform feature matching between the license to be processed and the license rule base to determine the initial license type of the license to be processed; and a verification module 505, used to verify the license type of the license to be processed based on a pre-trained question-answering model to determine the target license type of the license to be processed.

[0090] In one embodiment, the preprocessing module 502 is specifically used to remove invalid content in the original text using a preset regular expression, and to standardize the original text of the sample license to obtain a standardized license text; to perform word segmentation on the standardized license text; to remove stop words based on the result of word segmentation to obtain the keyword features of the sample license; to perform sentence segmentation on the standardized license text to obtain the sentence features of the sample license; and to generate phrases from the standardized license text to obtain the phrase features of the sample license; the first text features of the sample license include the keyword features, sentence features, and phrase features of the sample license.

[0091] In one embodiment, the generation module 503 includes: a mapping submodule 5031, used to map the first text features of the sample license based on mapping rules to generate license rules; a generation submodule 5032, used to generate a license rule library based on the license rules, the first text features of the sample license, and the license type of the sample license, the license rule library containing each license rule, a corresponding keyword list, a sentence list, a phrase list, and a license type; a first processing submodule 5033, used to vectorize the first text features of the sample license to obtain the first vector features of the sample license, the first vector features including keyword vectors, sentence vectors, and phrase vectors of the sample license; an acquisition submodule 5034, used to select text features whose cosine similarity with the first vector features meets a preset cosine similarity threshold and / or whose fuzzy matching with the first text features meets a fuzzy matching threshold as candidate text features; and an addition submodule 5035, used to add the candidate text features to the corresponding positions in the license rule library.

[0092] In one embodiment, the matching module 504 includes: a second processing submodule 5041, used to perform feature processing on the license to be processed to obtain a second text feature and a second vector feature of the license to be processed; a matching submodule 5042, used to match the second text feature and the second vector feature with license rules in the license rule base, and determine the license rules that meet the cosine similarity threshold and the fuzzy matching threshold as target license rules; a determining submodule 5043, used to determine the weight value of the license to be processed based on the number of target license rules, the cosine similarity between the license to be processed and the target license rules, and the priority of the target license rules; the determining submodule 5043 is also used to determine the initial license type of the license to be processed based on the weight value.

[0093] In one embodiment, the device further includes: an adjustment module 506, configured to obtain the review result of the pre-trained question-answering model reviewing the license type of the license to be processed; compare the review result with the initial license type to obtain the comparison result; and dynamically adjust the priority of the license rules according to the comparison result.

[0094] In one embodiment, the device further includes: a correction module 507, configured to obtain a manual judgment result of the license type of the license to be processed when it is determined that the target license type of the license to be processed is an infectious license or a highly infectious license; and to correct the license type of the license to be processed based on the judgment result.

[0095] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0096] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0097] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0098] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0099] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as an open-source license conflict detection method. For example, in some embodiments, an open-source license conflict detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of an open-source license conflict detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform an open-source license conflict detection method by any other suitable means (e.g., by means of firmware).

[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0104] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0105] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0106] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0107] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting open-source license conflicts, characterized in that, The method includes: Obtain the technology stack information input by the user, and obtain the basic information of the sample license based on the technology stack information. The basic information includes the original text of the sample license and the license type of the sample license. The license type includes open license, weakly infectious license, infectious license and strongly infectious license. The original text of the sample license is preprocessed to obtain the first text feature of the sample license; A license rule base is generated based on the first textual features of the sample license and the license type of the sample license; Obtain the license to be processed; The initial license type of the license to be processed is determined by feature matching between the license rule base and the license to be processed. The license type of the license to be processed is reviewed based on a pre-trained question-answering model to determine the target license type of the license to be processed. The preprocessing of the original text of the sample license to obtain the first text feature of the sample license includes: removing invalid content from the original text using a preset regular expression and standardizing the original text of the sample license to obtain a standardized license text; performing word segmentation on the standardized license text; removing stop words based on the result of the word segmentation to obtain the keyword feature of the sample license; performing sentence segmentation on the standardized license text to obtain the sentence feature of the sample license; and generating phrases from the standardized license text to obtain the phrase feature of the sample license. The first text feature of the sample license includes the keyword feature, sentence feature, and phrase feature of the sample license. The keyword feature, sentence feature, and phrase feature involved in the first text feature contain three core clauses of the sample license: rights, conditions, and restrictions. The step of generating a license rule base based on the first text features of the sample license and the license type of the sample license includes: mapping the first text features of the sample license to generate license rules based on mapping rules; generating a license rule base based on the license rules, the first text features of the sample license, and the license type of the sample license, wherein the license rule base includes each license rule, a corresponding keyword list, a sentence list, a phrase list, and a license type; vectorizing the first text features of the sample license to obtain the first vector features of the sample license, wherein the first vector features include keyword vectors, sentence vectors, and phrase vectors of the sample license; selecting text features whose cosine similarity with the first vector features meets a preset cosine similarity threshold and / or whose fuzzy matching with the first text features meets a fuzzy matching threshold as candidate text features; and adding the candidate text features to the corresponding positions in the license rule base.

2. The method according to claim 1, characterized in that, The step of performing feature matching between the license to be processed and the license rule base to determine the initial license type of the license to be processed includes: The license to be processed is subjected to feature processing to obtain the second text feature and the second vector feature of the license to be processed; The second text feature and the second vector feature are matched with the license rules in the license rule base, and the license rules that meet the cosine similarity threshold and the fuzzy matching threshold are determined as the target license rules; The weight value of the license to be processed is determined based on the number of target license rules, the cosine similarity between the license to be processed and the target license rules, and the priority of the target license rules. The initial license type of the license to be processed is determined based on the weight value.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the verification results of the pre-trained question-answering model for the license type of the license to be processed; The verification result is compared with the initial license type to obtain the comparison result; The priority of the license rules is dynamically adjusted based on the comparison results.

4. The method according to claim 1, characterized in that, The method further includes: When it is determined that the target license type of the license to be processed is an infectious license or a highly infectious license, obtain the manual judgment result of the license type of the license to be processed; The license type of the license to be processed is corrected based on the judgment result.

5. An open-source license conflict detection device, characterized in that, The device includes: The acquisition module is used to acquire the technology stack information input by the user, and acquire the basic information of the sample license based on the technology stack information. The basic information includes the original text of the sample license and the license type of the sample license. The license type includes open license, weakly infectious license, infectious license and strongly infectious license. The preprocessing module is used to preprocess the original text of the sample license to obtain the first text feature of the sample license; A generation module is used to generate a license rule base based on the first text features of the sample license and the license type of the sample license; The acquisition module is also used to acquire the license to be processed; The matching module is used to perform feature matching between the license to be processed and the license rule base to determine the initial license type of the license to be processed; The review module is used to review the license type of the license to be processed based on a pre-trained question-answering model, and determine the target license type of the license to be processed. The preprocessing module is specifically used to remove invalid content from the original text using preset regular expressions, and to standardize the original text of the sample license to obtain standardized license text; to perform word segmentation on the standardized license text; to remove stop words based on the result of the word segmentation to obtain the keyword features of the sample license; to perform sentence segmentation on the standardized license text to obtain the sentence features of the sample license; and to generate phrases from the standardized license text to obtain the phrase features of the sample license. The first text features of the sample license include the keyword features, sentence features, and phrase features of the sample license. The keyword features, sentence features, and phrase features involved in the first text features contain three core clauses of the sample license: rights, conditions, and restrictions. The generation module includes: a mapping submodule, used to map the first text features of the sample license based on mapping rules to generate license rules; a generation submodule, used to generate a license rule library based on the license rules, the first text features of the sample license, and the license type of the sample license, wherein the license rule library contains each license rule, a corresponding keyword list, a sentence list, a phrase list, and a license type; a first processing submodule, used to vectorize the first text features of the sample license to obtain the first vector features of the sample license, wherein the first vector features include keyword vectors, sentence vectors, and phrase vectors of the sample license; an acquisition submodule, used to select text features whose cosine similarity with the first vector features meets a preset cosine similarity threshold and / or whose fuzzy matching with the first text features meets a fuzzy matching threshold as candidate text features; and an addition submodule, used to add the candidate text features to the corresponding positions in the license rule library.

6. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

Images