Open source license legal clause identification method and apparatus
By using a deep learning text classification algorithm, Bi-LSTM and self-attention mechanism are employed to identify the legal terms and binding tendencies of open source licenses, solving the problem of inaccurate identification of compatibility risks in existing technologies and achieving more accurate risk analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF SOFTWARE - CHINESE ACAD OF SCI
- Filing Date
- 2023-03-01
- Publication Date
- 2026-05-29
Smart Images
Figure CN116150374B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer technology and relates to a method and apparatus for identifying legal terms of open source licenses, and particularly to a deep learning text classification algorithm for identifying the content of legal terms of open source licenses and their binding tendencies. Background Technology
[0002] Currently, open source has become a major trend in software innovation and the development of the software industry. At the same time, the development and use of open source software are accompanied by various risks, the biggest of which is potential intellectual property risk. Typically, software developers choose different open source licenses (licenses that allow the use, modification, or sharing of source code under established legal terms) for their software or components to restrict the conditions of use, protect their intellectual property rights, and safeguard the long-term development of the software.
[0003] Existing open-source license risk analyses mostly focus on compliance and compatibility risks. Compliance risk assessment primarily focuses on a single license. The more restrictions declared in a license, the more difficult it is for software or components using that license to comply with regulations, and the higher the compliance risk. Compatibility risk is assessed between two different licenses. Generally, open-source licenses can be broadly categorized into two types: permissive licenses and copyleft licenses. Apache, MIT, and BSD are permissive licenses, while GPL is a typical copyleft license. The biggest difference between the two types is that copyleft licenses require that modifications and extensions to the software must use the same license; permissive licenses do not have this requirement. Therefore, if an open-source software or component declares that it uses both the Apache and GPL licenses, these two licenses have different binding tendencies regarding the specific legal clause of "using the same license for distribution," posing a compatibility risk and potentially exposing the open-source software or component to intellectual property risks.
[0004] In short, current open-source license risk analysis methods analyze license files, primarily judging compatibility risks between different licenses based on their category. However, there is currently no officially defined open-source license classification standard. Therefore, the basis for judging compatibility risks is not sufficiently reasonable. As the above examples show, whether the binding tendency of specific legal clauses in different open-source licenses is consistent is a more specific and accurate basis for judging whether there is a compatibility risk between two licenses. Using artificial intelligence methods such as deep learning text classification, the semantics of license text can be understood by simulating the human brain's thought process. Specifically, by classifying the statements in the license text, specific legal clauses declared in the license text can be automatically identified and their binding tendency (allow / disallow, require / not require, etc.) can be determined.
[0005] In summary, accurately identifying specific legal terms and their binding tendencies declared in open-source licenses helps in determining whether there are compatibility risks between different licenses. Therefore, this invention provides an open-source license legal term identification device based on a deep learning text classification algorithm. Summary of the Invention
[0006] To address the aforementioned issues, this invention provides a method and apparatus for identifying legal terms of open-source licenses, offering a more accurate and reasonable basis for determining whether there are compatibility risks between different open-source licenses.
[0007] To achieve the above-mentioned objectives, the following technical solution was invented:
[0008] A method for identifying legal terms of an open-source license, the method comprising:
[0009] The license text of open-source software is split into several statements;
[0010] Each statement is categorized using multiple tags for legal clauses to obtain statements containing legal clauses and the legal clauses within those statements;
[0011] The binding tendency of statements containing legal clauses is classified to obtain the binding tendency of the legal clauses in the license text.
[0012] Based on the legal provisions and their corresponding binding tendencies, the identification results of the legal provisions in the license text are obtained.
[0013] Furthermore, the process of splitting the open-source software license text file into several statements includes:
[0014] Obtain the license text for open-source software;
[0015] Based on the periods, semicolons, and colons contained in the license text, the license text is segmented to obtain several statements.
[0016] Furthermore, the multi-label classification of each statement into legal clauses to obtain statements containing legal clauses includes:
[0017] The statement is segmented into words;
[0018] Identify the English description of each legal clause tag and construct the tag sentence vector matrix C;
[0019] The word vector of each word is calculated based on the pre-trained word vector model, and the word vector matrix of the first sentence is constructed.
[0020] The word vector matrix of the first statement is input into a single-layer Bi-LSTM neural network to obtain the sentence matrix H of the statement;
[0021] The sentence matrix H of the statement is used to extract features using a self-attention mechanism to obtain the feature matrix M of the statement. (s) ;
[0022] Based on the labeled sentence vector matrix C, feature extraction using a label attention mechanism is performed on the sentence matrix H of the statement to obtain the feature matrix M of the statement. (l) ;
[0023] According to the feature matrix M (s) and the feature matrix M (l) The fusion matrix M is obtained;
[0024] The fusion matrix M is calculated using the sigmoid function to obtain the legal clause label of the statement; wherein the legal clause label includes K dimensions, each dimension representing the type of a corresponding legal clause.
[0025] Further, the step of inputting the first sentence word vector matrix into a single-layer Bi-LSTM neural network to obtain the sentence matrix H of the sentence includes:
[0026] Calculate the forward propagation state of the p-th timestamp in, W represents the forward propagation state at timestamp p-1. p This represents the word vector of the p-th word in the statement;
[0027] Calculate the backpropagation state for the p-th timestamp. in, This represents the backpropagation state of the (p-1)th timestamp;
[0028] Forward propagation state based on each timestamp Constructing a forward propagation matrix Where n represents the number of words in the statement;
[0029] Backpropagation state based on each timestamp Constructing the backpropagation matrix
[0030] According to the forward propagation matrix and the backpropagation matrix The sentence matrix H of the given statement is obtained.
[0031] Furthermore, a self-attention mechanism is used to extract features from the sentence matrix H of the statement, resulting in the feature matrix M of the statement. (s) ,include:
[0032] Calculate the self-attention score A (s) = softmax(W2tanh(W1H)); where W1 and W2 are weight matrices;
[0033] Based on the self-attention score A (s) The feature matrix M of the statement is obtained from the sentence matrix H of the statement. (s) .
[0034] Furthermore, based on the labeled sentence vector matrix, feature extraction using a label attention mechanism is performed on the sentence matrix H of the statement to obtain the feature matrix M of the statement. (l) ,include:
[0035] Calculate the label attention score A (l) =CH;
[0036] Based on the label attention score A (l) The feature matrix M of the statement is obtained from the sentence matrix H of the statement. (l) .
[0037] Furthermore, the types of legal terms include: commercial use, distribution, modification, private use, patent use, trademark use, open source code, license and copyright notice, same license, modification history, disclaimer and warranty.
[0038] Furthermore, the classification of the binding tendency of statements containing legal clauses to obtain the binding tendency corresponding to the legal clauses in the license text includes:
[0039] The statement containing the legal clause is input into the pre-trained ALBERT model to obtain the feature vector matrix of the second statement;
[0040] The feature vector matrix of the second statement is classified to obtain the constraint tendency corresponding to the legal clauses in the license text.
[0041] An open-source license legal terms identification device, the device comprising:
[0042] The text preprocessing module is used to segment the license text of open-source software into several sentences;
[0043] The multi-label classification module is used to classify each statement into legal clauses using multiple labels to obtain the statement containing legal clauses and the legal clauses in that statement;
[0044] The binary classification module is used to classify the restrictive tendency of statements containing legal clauses to obtain the restrictive tendency corresponding to the legal clauses in the license text; based on the legal clauses and the restrictive tendency corresponding to the legal clauses, the identification result of the legal clauses in the license text is obtained.
[0045] An electronic device, characterized in that it comprises: a processor, and a memory storing computer program instructions; wherein the processor, when executing the computer program instructions, implements any of the above-described open-source license legal terms identification methods.
[0046] Compared with existing technologies, this invention designs and implements a device for identifying legal terms in open-source licenses using a deep learning text classification model. The deep learning text classification model used in this invention can be trained to autonomously learn to identify specific legal terms declared in open-source licenses and determine the binding tendency of these terms, significantly saving manpower and time costs. This invention can be widely applied in the field of open-source license compatibility risk analysis and is available for use by software developers and experts. Attached Figure Description
[0047] The accompanying drawings are provided to aid in a further understanding of the invention and form part of this application. They do not constitute an undue limitation of the invention. In the drawings:
[0048] Figure 1 This is a framework diagram of an open-source license legal terms identification device.
[0049] Figure 2 This is an example image of data samples in the training set of the multi-label classification module.
[0050] Figure 3 This is an example image of data samples in the training set for the binary classification module.
[0051] Figure 4 This is the architecture diagram of the multi-label classification module.
[0052] Figure 5This is the architecture diagram of the binary classification module. Detailed Implementation
[0053] The present invention will be further described below with reference to the accompanying tables and figures, through embodiments.
[0054] This example provides a device for identifying the legal terms of an open-source license, and its overall framework is as follows: Figure 1 As shown, the system specifically includes the following modules: a text preprocessing module, a multi-label classification module, and a binary classification module. The text preprocessing module segments the raw text of the input license document into individual sentences using periods (."), semicolons (;), and colons (:), outputting a set of sentences composed of these individual sentences. The multi-label classification module takes this set of sentences as input, trained on a manually labeled training set, identifies the legal clauses in each sentence, and outputs a set of sentences labeled with the legal clause classification. The binary classification module takes this set of sentences labeled with legal clause classification, trained on a manually labeled training set, identifies the clause constraint tendency of each sentence labeled with legal clause classification, and outputs a set of sentences labeled with both the legal clause and constraint tendency classifications.
[0055] Data samples and model architecture in the training set, such as Figures 2 to 5 As shown, it includes the following steps:
[0056] Step 1: Select representative open-source license legal terms and define the description of each legal term and all possible binding tendencies.
[0057] Step 1 mainly includes the following steps:
[0058] 1a) Twelve legal clauses were selected from the common constraints of open source licenses displayed on the open source license website (https: / / choosealicense.com / appendix / ) as classification labels for the multi-label classification model. They are: Commercial Use, Distribution, Modification, Private Use, Patent Use, Trademark Use, Disclose Source, License and Copyright Notice, Same License, State Changes, Liability, and Warranty.
[0059] 1b) As shown in Table 1, this example defines the description of each legal clause and all possible binding tendencies.
[0060] Table 1. Legal Clause Names and Related Definitions
[0061]
[0062]
[0063]
[0064] Step 2: Construct a training set for a multi-label classification model to identify legal clauses.
[0065] Step 2 mainly includes the following steps:
[0066] 2a) Obtain the open-source license text files to form a set of open-source license files. Separate the original text in each file into individual sentences using ".", ";", and ":". Proceed to 2b);
[0067] 2b) Based on the clause descriptions defined in step 1b), and considering the sentence semantics, label the legal clause to which each statement belongs (category label). Figure 2 Taking data sample 2 as an example, the statement 'in no event shall the copyrightholder or contributors be liable for any damages……' means 'be liable for any damages,' which matches the description of the disclaimer clause in Table 1. Therefore, the legal clause to which this statement belongs is Liability, and the category label is (0,0,0,0,0,0,0,0,0,0,1,0). (Go to 2c);
[0068] 2c) Add each statement and its legal clause category label as a data sample to the training set to obtain the training set {sent i ,label i}
[0069] Step 3: Constructing the training set for the binary classification model.
[0070] Step 3 mainly includes the following steps:
[0071] 3a) Select statements from the training set constructed in step 2c) that have legal clause classification labels (at least one dimension of the label vector has a value of 1). Proceed to 3b);
[0072] 3b) Based on the constraint tendency defined in step 1b), and combined with the sentence semantics, label the legal clause constraint tendency (classification label) of each statement. Figure 3Taking data sample 2 as an example, in the statement 'in no event shall the copyrightholder or contributors be liable for any damages……', 'in no event' means "absolutely not," which aligns with the "no liability" constraint tendency of the disclaimers in Table 1. Therefore, the legal clause constraint tendency of this statement is no liability, and the category label is 0. (Go to 3c);
[0073] 3c) Add each statement with a legal clause classification label and its binding tendency classification label as a data sample to the training set to obtain the training set {sent i ,label i}
[0074] Step 4: Set the learning parameters and train the multi-label classification model.
[0075] The overall architecture of the multi-label classification model is as follows: Figure 4 As shown, the specific training steps are as follows:
[0076] 4a) Obtaining the word vector matrix: Extract the word vector matrix from the training set constructed in step 2). i After word segmentation, the input is fed into a pre-trained word vector model to obtain a word vector matrix X composed of the word vectors of the words in the sentence. i .
[0077] X i ={W1,…,W p ,…,W n}
[0078] Among them, W p Let p be the word vector of the p-th word in the sentence.
[0079] Go to 4b);
[0080] 4b) Sentence matrix acquisition: Obtain the word vector matrix X i The input is fed into a single-layer Bi-LSTM neural network to obtain the sentence matrix H of the statement.
[0081] Specifically, at the p-th timestamp, based on the currently input word vector W... p The state of the previous timestamp
[0082] Current timestamp status The calculation formula is:
[0083]
[0084]
[0085] The sentence matrix H of the statements is represented as follows:
[0086]
[0087]
[0088]
[0089] Go to 4c);
[0090] 4c) Obtaining the sentence vector matrix of the tags.
[0091] This example uses a pre-trained Sentence Transformer model to directly encode the English description of each tag (legal clause), as shown in Table 2, into a word vector W. p Given sentence vectors of the same dimension, concatenate the sentence vectors of dimension k to obtain a sentence vector matrix C. The same label sentence vector matrix is used in both training and testing.
[0092] Table 2 Label Names and Related English Descriptions
[0093]
[0094]
[0095] (Transfer to 4d);
[0096] 4d) Self-Attention Feature Filtering: The sentence matrix H is extracted using the Self-Attention mechanism to obtain the sentence feature matrix M. (s) .
[0097] Specifically, the formula for calculating the Self-Attention Score is as follows:
[0098] A (s) = softmax(W2tanh(W1H)), where W1 and W2 are weight matrices.
[0099] The feature matrix M of the statement (s) Represented as:
[0100] M (s) =A (s) H T
[0101] (Go to 4e);
[0102] 4e) Label-Attention Feature Filtering: The label-attention mechanism is used to extract features from the sentence matrix H, resulting in the sentence feature matrix M. (l) .
[0103] Specifically, the formula for calculating the Label-Attention Score is as follows:
[0104] A (l) =CH, where C is the sentence vector matrix of the tags.
[0105] The feature matrix M of the statement (l) Represented as:
[0106] M (l) =A (l) H T
[0107] Go to 4f);
[0108] 4f) Feature fusion: The feature matrix M obtained by Self-Attention feature filtering is fused together. (s) The feature matrix M obtained by Label-Attention feature selection (l) The fusion matrix M is input into a fully connected ReLU layer.
[0109] The formula for calculating the fusion matrix M is:
[0110] M = αM (s) +βM (l) Where α+β=1,
[0111] Specifically, α = sigmoid(M (s) W3), β = sigmoid(M (l) W4),
[0112] Among them, W3 and W4 are weight matrices.
[0113] (Transfer to 4g);
[0114] 4g) Classification: The sigmoid function is used to calculate the fusion matrix M to obtain the classification label of the statement. Statement category tags The calculation formula is:
[0115] Among them, W5 and W6 are weight matrices.
[0116] Specifically, steps 4a) to 4f) require multiple iterations using the cross-entropy loss function based on the set learning parameters to update the various parameters in the model, ultimately training a multi-label classification model.
[0117] Step 5: Set the learning parameters and train the binary classification model.
[0118] The overall architecture of the binary classification model is as follows: Figure 5 As shown, the specific training steps are as follows:
[0119] 5a) Feature Matrix Acquisition: After word segmentation of the sentences in the training set constructed in step 3), input them into the pre-trained ALBERT word vector model to obtain a feature matrix composed of the feature vectors of the words in the sentences. Go to 5b);
[0120] 5b) Classification: Input the feature matrix into the ALBERT classification model to obtain the classification label of the statement;
[0121] Specifically, steps 5a) to 5b) require multiple iterations using the cross-entropy loss function based on the set learning parameters to update the various parameters in the model, ultimately training a binary classification model.
[0122] Those skilled in the art can modify or make equivalent substitutions to the technical solutions of this invention without departing from the spirit and scope of this invention. The scope of protection of this invention shall be determined by the claims.
Claims
1. A method for identifying legal terms of an open-source license, characterized in that, The method includes: The license text of open-source software is split into several statements; Each statement is categorized using multiple tags for legal clauses to obtain statements containing legal clauses and the legal clauses within those statements; The binding tendency of statements containing legal clauses is classified to obtain the binding tendency of the legal clauses in the license text. Based on the legal provisions and their corresponding binding tendencies, the identification results of the legal provisions in the license text are obtained; The step of performing multi-label classification of legal clauses on each statement to obtain statements containing legal clauses and the legal clauses within those statements includes: The statement is segmented into words; Identify the English description of each legal clause tag and construct a tag sentence vector matrix. ; The word vector of each word is calculated based on the pre-trained word vector model, and the word vector matrix of the first sentence is constructed. The word vector matrix of the first statement is input into a single-layer Bi-LSTM neural network to obtain the sentence matrix of the statement. ; Using a self-attention mechanism to analyze the sentence matrix of the given statements Feature extraction is performed to obtain the feature matrix of the statement. ; Based on the tag sentence vector matrix The sentence matrix of the stated statement Feature extraction is performed using a tag attention mechanism to obtain the feature matrix of the statement. ; According to the feature matrix and the feature matrix The fusion matrix is obtained. ; Use the sigmoid function on the fusion matrix Calculations are performed to obtain the legal clause tags of the statement; wherein, the legal clause tags include Each dimension represents a type of legal clause.
2. The method as described in claim 1, characterized in that, The process of splitting the open-source software license text file into several statements includes: Obtain the license text for open-source software; Based on the periods, semicolons, and colons contained in the license text, the license text is segmented to obtain several statements.
3. The method as described in claim 1, characterized in that, The first sentence word vector matrix is input into a single-layer Bi-LSTM neural network to obtain the sentence matrix of the sentence. ,include: Calculate the first The forward propagation state of each timestamp ;in, Indicates the first The forward propagation state of each timestamp. Indicates the first in the statement Word vectors of 1 word; Calculate the first Backpropagation state of each timestamp ;in, Indicates the first The backpropagation state of each timestamp; Forward propagation state based on each timestamp Construct a forward propagation matrix ;in, Indicates the number of words in the statement; Backpropagation state based on each timestamp Construct the backpropagation matrix ; According to the forward propagation matrix and the backpropagation matrix The sentence matrix of the given statement is obtained. .
4. The method as described in claim 1, characterized in that, Using a self-attention mechanism to analyze the sentence matrix of the given statements Feature extraction is performed to obtain the feature matrix of the statement. ,include: Calculate self-attention score ;in, and This is the weight matrix; Based on the self-attention score and the sentence matrix of the stated statements The feature matrix of the statement is obtained. .
5. The method as described in claim 1, characterized in that, The sentence matrix of the statement is based on the tag sentence vector matrix. Feature extraction is performed using a tag attention mechanism to obtain the feature matrix of the statement. ,include: Calculate the label attention score ; Based on the label attention score and the sentence matrix of the stated statements The feature matrix of the statement is obtained. .
6. The method as described in claim 1, characterized in that, The types of legal terms mentioned include: commercial use, distribution, modification, private use, patent use, trademark use, open source code, license and copyright notice, same license, modification history, disclaimer and warranty.
7. The method as described in claim 1, characterized in that, The classification of the binding tendency of statements containing legal clauses to obtain the binding tendency corresponding to the legal clauses in the license text includes: The statement containing the legal clause is input into the pre-trained ALBERT model to obtain the feature vector matrix of the second statement; The feature vector matrix of the second statement is classified to obtain the constraint tendency corresponding to the legal clauses in the license text.
8. An open-source license legal terms identification device, characterized in that, The device includes: The text preprocessing module is used to segment the license text of open-source software into several sentences; The multi-label classification module is used to classify each statement into legal clauses using multiple labels to obtain the statement containing legal clauses and the legal clauses in that statement; The binary classification module is used to classify the restrictive tendency of statements containing legal clauses to obtain the restrictive tendency of the legal clauses in the license text; based on the legal clauses and the restrictive tendency of the legal clauses, the identification result of the legal clauses in the license text is obtained. Specifically, each statement is categorized using multiple tags for legal clauses to obtain statements containing legal clauses and the legal clauses within those statements, including: The statement is segmented into words; Identify the English description of each legal clause tag and construct a tag sentence vector matrix. ; The word vector of each word is calculated based on the pre-trained word vector model, and the word vector matrix of the first sentence is constructed. The word vector matrix of the first statement is input into a single-layer Bi-LSTM neural network to obtain the sentence matrix of the statement. ; Using a self-attention mechanism to analyze the sentence matrix of the given statements Feature extraction is performed to obtain the feature matrix of the statement. ; Based on the tag sentence vector matrix The sentence matrix of the stated statement Feature extraction is performed using a tag attention mechanism to obtain the feature matrix of the statement. ; According to the feature matrix and the feature matrix The fusion matrix is obtained. ; Use the sigmoid function on the fusion matrix Calculations are performed to obtain the legal clause tags of the statement; wherein, the legal clause tags include Each dimension represents a type of legal clause.
9. An electronic device, characterized in that, include: Processor, and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the open-source license legal terms identification method according to any one of claims 1-7.