Sensitive information identification method and system, readable storage medium and computing device

Abstract

This invention relates to a method, system, readable storage medium, and computing device for sensitive information identification. The method includes the following steps: acquiring code to be identified; determining the language type of the code to be identified; performing a syntax tree transformation on the code to be identified to generate a syntax tree corresponding to the code to be identified; extracting a symbol table related to literals in the code to be identified, and determining the relationships between variables, functions, classes, constants, and comments therein; using a preset data structure to represent the relationships between variables, functions, classes, constants, and comments, wherein the preset data structure is the same for different codes to be identified; for each literal, determining the usage scenario of the literal in the code to be identified, and obtaining relevant information in the usage scenario; and identifying whether the literal is sensitive information. The sensitive information identification method and system of this invention improve the ability to extract sensitive information while solving the problem of a dramatic increase in development workload when accessing code of multiple language types.

Description

Technical Field

[0001] This invention relates to the field of data security technology, and more specifically to a method, system, readable storage medium, and computing device for identifying sensitive information with front-end and back-end separation. Background Technology

[0002] As internet products become increasingly important in users' lives, the protection of user data privacy is receiving more and more attention. Various countries and regions have introduced a series of laws and regulations on data security and privacy protection, requiring internet companies to classify and classify user privacy data in their businesses and protect it at different levels of security.

[0003] Text serves as a primary carrier of user identification data, and its leakage can cause significant losses to users. During the scanning and classification of unknown data within a raw text file, various types of privacy data exhibit different organizational forms due to differences in country and region. Therefore, it is necessary to customize identification methods for various sensitive information based on factors such as type and region.

[0004] As the core carrier of program design, coding languages ​​may hard-code various sensitive information used for testing and connection purposes, such as account passwords, ID cards, email addresses, and mobile phone numbers, in code segments and configuration languages ​​during the writing process. Sensitive information can be mainly divided into two categories: the first category includes information with a clearly defined format and verification method, such as an ID card, which consists of a region code, date of birth, a random code, and a check digit; the second category includes information without a clearly defined format and / or without a verification method, such as account passwords and mobile phone numbers. For the extraction of these two types of data, existing technologies mainly employ regular expressions combined with context and check digit matching. This method is effective for rich text extraction, but it struggles with scenarios involving nested code text, value passing, and scope division. Its main problems are: 1) It struggles to consider contextual information, failing to accurately extract semantics from code segments and configuration files using multiple languages; 2) Regular expression extraction cannot cover complex code language issues such as indirect assignment, computational assignment, function constant parameter passing, complex computational assignment, setter / getter assignment, and language characteristics; 3) When optimizing accuracy for multi-language segmentation, conflicts arise between language segments, and using regular expressions can lead to conflicts between different languages; 4) For context-sensitive scenarios (i.e., usage scenarios), the inconsistent contextual forms of different languages ​​make it difficult to share subsequent extraction parts in regular expression recognition, increasing the difficulty linearly with the number of languages. Furthermore, due to the significant differences in the composition of different code languages, developing corresponding sensitive type extraction methods for each language results in a quadratic increase in development complexity, multiplied by the product of the number of language types and usage scenarios. Summary of the Invention

[0005] One of the objectives of this invention is to provide a sensitive information identification method that improves the ability to extract sensitive information while solving the problem of a surge in development workload when integrating code from multiple language types.

[0006] To achieve the above objectives, the present invention provides a method for identifying sensitive information, comprising the following steps:

[0007] Obtain the code to be recognized;

[0008] Determine the language type of the code to be identified;

[0009] Based on the language type of the code to be identified, a syntax tree transformation is performed on the code to be identified to generate a syntax tree corresponding to the code to be identified;

[0010] Based on the syntax tree, extract the symbol table related to literals in the code to be identified, and determine the relationship between variables, functions, classes, constants and comments in the symbol table;

[0011] A preset data structure is used to represent the relationships between variables, functions, classes, constants and comments in the symbol table. The preset data structure is the same for the code to be identified in different language types.

[0012] For each literal, based on the data structure, the usage scenario of the literal in the code to be identified is determined, and the relevant information of the literal in the usage scenario is obtained;

[0013] For each literal, it is determined whether the literal is sensitive information based on the literal and its relevant information in the usage scenario.

[0014] Furthermore, in some implementations, Antlr is used to perform syntax tree transformation on the code to be identified.

[0015] Furthermore, in some implementations, extracting the symbol table related to literals in the code to be recognized based on the syntax tree includes:

[0016] The literal-related operations are performed through the visitor mechanism of the syntax tree to obtain the literal after the operation, and the symbol table related to the literal after the operation is extracted through the listener mechanism of the syntax tree.

[0017] Furthermore, in some implementations, extracting the symbol table related to literals in the code to be identified based on the syntax tree, and determining the relationships between variables, functions, classes, constants, and comments in the symbol table includes:

[0018] Extract the variable name, variable type, associated variable, and variable scope fields, and generate a unique ID;

[0019] Extract the function name, function variable name, and the defined functions and variables used internally, and define the function's functional score based on the internal variable name and function name, and then define the function's attributes;

[0020] Extract sub-variable fields from subclasses;

[0021] Extract direct literal constants and constants to be calculated, and associate them with the unique IDs of the variable table. Obtain the associated variable names, function calls, class calls, and method calls from the variable table as meta information.

[0022] Comments are functionally categorized into code snippets and text information. Code snippets are extracted a second time, while text information is directly integrated into a general rich text detection tool.

[0023] Furthermore, in some implementations, the use cases include direct assignment, indirect assignment, computational assignment, passing function constants as parameters, passing function variables as parameters, complex computational assignment, language features, and comments / rich text.

[0024] Furthermore, in some implementations, the relevant information of the literal in the use case includes the variables, functions, and classes associated with the literal in the use case, as well as the role of the literal, variables, functions, and classes.

[0025] Furthermore, in some implementations, regular expression matching methods are used to identify whether each literal is sensitive information; and / or,

[0026] Based on the relevant information of each literal in the usage scenario, keyword retrieval or neural network model is used to determine the function of each literal, and then the function of each literal is used to determine whether it is sensitive information.

[0027] Another objective of this invention is to provide a sensitive information identification system that, while improving the ability to extract sensitive information to be identified, solves the problem of a dramatic increase in development workload when integrating code from multiple language types.

[0028] To achieve the above objectives, the present invention also provides a sensitive information identification system, including systems for mutual data communication:

[0029] The acquisition module is set to acquire the code to be recognized;

[0030] The judgment module is set to determine the language type of the code to be recognized.

[0031] The syntax tree transformation module is configured to transform the syntax tree of the code to be recognized based on the language type of the code to be recognized, and generate the syntax tree corresponding to the code to be recognized.

[0032] The first processing module is configured to extract the symbol table related to literals in the code to be identified based on the syntax tree, and determine the relationship between variables, functions, classes, constants and comments in the symbol table;

[0033] The second processing module is configured to use a preset data structure to represent the relationship between variables, functions, classes, constants and comments in the symbol table, wherein the preset data structure is the same for the code to be identified for different language types;

[0034] The third processing module is configured to determine the usage scenario of each literal in the code to be recognized based on the data structure, and obtain the relevant information of the literal in the usage scenario.

[0035] The recognition module is configured to identify whether a literal is sensitive information based on the literal and related information in its usage scenario for each literal.

[0036] Furthermore, in some embodiments, the syntax tree transformation module uses Antlr to perform syntax tree transformation on the code to be identified.

[0037] Furthermore, in some implementations, extracting the symbol table related to literals in the code to be recognized based on the syntax tree includes:

[0038] The literal-related operations are performed through the visitor mechanism of the syntax tree to obtain the literal after the operation, and the symbol table related to the literal after the operation is extracted through the listener mechanism of the syntax tree.

[0039] Furthermore, in some embodiments, the first processing module is further configured as follows:

[0040] Extract the variable name, variable type, associated variable, and variable scope fields, and generate a unique ID;

[0041] Extract the function name, function variable name, and the defined functions and variables used internally, and define the function's functional score based on the internal variable name and function name, and then define the function's attributes;

[0042] Extract sub-variable fields from subclasses;

[0043] Extract direct literal constants and constants to be calculated, and associate them with the unique IDs of the variable table. Obtain the associated variable names, function calls, class calls, and method calls from the variable table as meta information.

[0044] Comments are functionally categorized into code snippets and text information. Code snippets are extracted a second time, while text information is directly integrated into a general rich text detection tool.

[0045] Furthermore, in some implementations, the use cases include direct assignment, indirect assignment, computational assignment, passing function constants as parameters, passing function variables as parameters, complex computational assignment, language features, and comments / rich text.

[0046] Furthermore, in some implementations, the relevant information of the literal in the use case includes the variables, functions, and classes associated with the literal in the use case, as well as the role of the literal, variables, functions, and classes.

[0047] Furthermore, in some embodiments, the identification module employs a regular expression matching method to identify whether each literal is sensitive information; and / or,

[0048] Based on the relevant information of each literal in the usage scenario, keyword retrieval or neural network model is used to determine the function of each literal, and then the function of each literal is used to determine whether it is sensitive information.

[0049] Another object of the present invention is to provide a storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the steps of the sensitive information identification method described above.

[0050] Another object of the present invention is to provide a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it performs the steps of the sensitive information identification method as described above. Attached Figure Description

[0051] Figure 1 A flowchart of a sensitive information identification method according to an embodiment of the present invention;

[0052] Figure 2 This is a structural block diagram of a sensitive information identification system according to another embodiment of the present invention. Detailed Implementation

[0053] The preferred embodiments of the present invention are given below with reference to the accompanying drawings and described in detail.

[0054] like Figure 1 As shown, this embodiment of the invention provides a method for identifying sensitive information with front-end and back-end separation, which includes the following steps:

[0055] 100: Get the code to be identified.

[0056] 200: Determine the language type of the code to be identified.

[0057] The code to be identified can be code text in various languages, including but not limited to Java, C++, Python, Yaml, Jason, and Java Profiles.

[0058] 300: Syntax Tree Transformation: Based on the language type of the code to be recognized, perform syntax tree transformation on the code to be recognized to generate the corresponding syntax tree.

[0059] In some embodiments, Antlr can be used to perform syntax tree transformation on the code to be recognized. Antlr (Another Tool for Language Recognition) is an open-source parser that can automatically generate and visualize syntax trees based on input. Its predecessor was PCCTS, which provides a framework for automatically constructing recognizers, parsers, and translators for various languages, including Java, C++, and C#, based on syntax descriptions. Specifically, an Antlr parsing script can be written to generate the syntax tree corresponding to the code to be recognized. Since it is an open-source parser, the method for writing its parsing script is well-known in the field and will not be elaborated here.

[0060] After the code to be identified is input into Antlr, its language type will be automatically identified and a syntax tree will be generated. Therefore, steps 200 and 300 can be completed simultaneously using Antlr.

[0061] After syntax tree transformation, the syntax tree corresponding to the code to be identified can be obtained, such as Java syntax tree, C++ syntax tree, Python syntax tree, YAML syntax tree, Jason syntax tree, or Java Profiles syntax tree, etc.

[0062] 400: Syntax Tree Parsing: Based on the syntax tree, extract the symbol table related to literals in the code to be identified, and determine the relationships between variables, functions, classes, constants and comments in the symbol table.

[0063] Since sensitive information appears in the code as literals (i.e. strings), during syntax tree parsing, only the information related to the literals needs to be extracted, and then this information is further processed in subsequent steps.

[0064] Because code may contain many operations related to literals, such as language features (including getter / setter assignments and StringBuilder, etc.), calculations (including addition, subtraction, multiplication, and division), and decryption operations, many intermediate literals may appear. These intermediate literals are only present during calculations and are not sensitive information themselves; they exist solely to perform certain operations. Only the final literal obtained from these operations may be sensitive information. Therefore, when extracting the symbol table related to literals, it is necessary to parse the literals, i.e., first perform the literal-related operations to obtain the literal after the operation. For example, in the case of adding two literals, perform the addition operation to obtain the literal after addition, and then extract the symbol table of the literal after addition. Alternatively, literals may have different encoding forms in the code. For example, an ID number may be encrypted, so it needs to be decrypted to obtain the decrypted literal before extracting its symbol table. This makes the identification of sensitive information more accurate. Taking JavaScript as an example, the general regular expression method will incorrectly identify the assignment of "variable = variable" as sensitive information. Out of 5,000 sensitive information identified by the regular expression method, only 2 are correct, making it almost impossible to operate. However, the method of this invention completely avoids this problem because it parses the language characteristics and finally obtains the literal value after operation. The accuracy can be significantly improved.

[0065] In some embodiments, the visitor mechanism of the syntax tree can be used to obtain the literal after the operation.

[0066] The symbol table is an information representation of code. In the symbol table, each identifier in the program's source code is bound to its declaration or usage information, such as its data type, scope, and memory address. This information is generally stored in the system in tabular form, such as constant tables, variable name tables, array name tables, procedure name tables, label tables, etc. The symbol table contains multiple different elements, such as variables, function declarations / calls, class declarations / calls, constants, and comments. During syntax tree parsing, the information of each element is aggregated. For example, all variables are aggregated together, all function declarations / calls are aggregated together, all class declarations / calls are aggregated together, all constants are aggregated together, and all comments are aggregated together. During the aggregation process, contextual information association (i.e., attribute association) is also required to clarify the names, positions, and relationships of various elements, that is, to determine the relationships between all variables, functions, classes, constants, and comments related to literals. For example, if the symbol table contains assignment-related variables "a = b" and "c = a", then the contextual information for variable c is variables a and b. Determining the relationship between variables, functions, classes, constants, and comments means obtaining the assignment relationship between variables a, b, and c. Similarly, for function declarations / calls, if the code contains an insecure encrypted function, it can be extracted from the function declarations / calls in the symbol table. This function needs to use a variable A, which can be obtained from the variable table in the symbol table. This function needs to use variable A for decryption. In this case, obtaining and determining the relationship between variables, functions, classes, constants, and comments in the symbol table means obtaining the function, variable A, and the interaction relationship between the function and variable A from the symbol table. That is, processing the content extracted from the symbol table to obtain the content needed for subsequent sensitive information identification.

[0067] In some embodiments, step S400 further includes:

[0068] Extract the variable name, variable type, associated variable, and variable scope fields, and generate a unique ID to improve query performance;

[0069] Extract the function name, function variable name, and the defined functions and variables used internally, and define the function's functional score based on the internal variable name and function name, and then define the function's attributes;

[0070] Obtaining the intermediate representation of a class declaration / call includes: extracting sub-variable fields of the subclass for defining the association between the class and sensitive types. Obtaining the intermediate representation of a constant includes: extracting direct literal constants (i.e., literals), constants to be calculated, associating them with the variable's unique ID, and obtaining the associated variable name, function call, class call, and method call as meta information.

[0071] Comments are functionally categorized into code snippets and explanatory text information. Code snippets are extracted a second time, while explanatory text information is directly integrated into rich text general detection.

[0072] In some embodiments, the extraction of the symbol table associated with literals can be accomplished through the syntax tree listener mechanism. The listener mechanism is provided by Antlr for traversing the syntax tree, and it can automatically traverse the syntax tree. By default, Antlr uses a built-in iterator to access the generated syntax tree and generates a syntax tree listener interface for each event that may be triggered during traversal. Users can implement their own logic by overriding the enter and exit functions.

[0073] 500: A preset data structure is used to represent the relationship between variables, functions, classes, constants and comments in the symbol table, wherein the preset data structure is the same for the code to be identified for different language types.

[0074] Because different language types have different implementation logics for their symbol tables, extracting the relationships between variables, functions, classes, constants, and comments requires different processing methods to identify sensitive information, making it impossible to uniformly identify sensitive information. Therefore, planning is necessary. In this embodiment, after determining the relationships between variables, functions, classes, constants, and comments, they are formed into a single pre-defined data structure. This standardizes the relationships between literal-related variables, functions, classes, constants, and comments in different language types, facilitating subsequent unified processing.

[0075] The data structure includes the names, locations, and relationships of all variables, functions, classes, constants, and comments in the code to be identified. It serves as the information source for sensitive information identification, and subsequent sensitive information identification needs to be carried out based on the data structure.

[0076] Steps 100-500 are unique to each language type of code. That is, for each language type of code to be identified, steps 100-500 need to be performed separately to obtain the same data structure. This data structure contains literal information related to the code to be identified in different language types, and uses this as the information source for sensitive information identification. Therefore, through steps 100-500, a universal information source can be extracted.

[0077] 600: For each literal, based on the data structure, determine the usage scenario of the literal in the code to be recognized, and obtain the relevant information of the literal in the usage scenario.

[0078] Since the data structure is only obtained when acquiring all information related to literals, such as variables, functions, classes, etc., without knowing the role of literals in this information, it is impossible to directly identify sensitive information through the data structure. Therefore, in step 600, information aggregation is performed on the data structure using literals as the aggregation unit. That is, starting from the literal, all information such as variables and functions associated with the literal is found in the data structure, and the scenarios are subdivided to form different usage scenarios. This clarifies the relevant information of literals in different usage scenarios, including the role of literals in different usage scenarios and the role of variables and functions associated with the literals. This information is used as auxiliary information to determine whether the literal is sensitive information. For example, in the code, there is a function `login("username", "mypassword")` that accesses a database using two variables, `username` and `mypassword`. When obtaining the data structure in step 500, we can obtain the `login` function and its calling relationship with the variables `username` and `mypassword`. However, we cannot know what the function does. It is impossible to determine whether a literal is sensitive information based solely on the function name and variable name. Therefore, it is necessary to clarify the purpose of literals, functions, and variables through scenario segmentation in order to determine whether a literal is sensitive information.

[0079] In some embodiments, different use cases include, but are not limited to, direct assignment, indirect assignment, computational assignment, passing function constants as parameters, passing function variables as parameters, complex computational assignment, language features (such as getters / setters), and comments / rich text. Scenario segmentation involves parsing the literal to obtain the literal and its related information in different use cases.

[0080] In one exemplary embodiment, the following use cases exist in the code:

[0081] Direct assignment: mypassword = "123456"; Indirect assignment: temp = "123456", mypassward = temp, mypassward = temp = "123456"; Calculated assignment: mypassward = "123" + "456"; Function constant parameter passing: response = login("abcdef", "123456"); Language-independent complex calculation assignment: String temp1 = "123"; String temp2 = "456"; String mypassword = temp1 + temp2; Language-related complex calculation assignment: StringBuilder strBuilder = new StringBuilder(); for(int i = 1; i < 7; i++) { str.Builder.append(i);} String mypassword = strBuilder.toString().

[0082] After segmenting the scenarios, we can obtain the literal 123456 and its related information in these usage scenarios, such as variable names mypassword, temp, temp1, temp2, function name login, variable name abcdef, function login, and their functions. In subsequent recognition, this related information can be used to determine whether the literal 123456 is sensitive information. For example, if the keyword search finds mypassword, then 123456 can be considered a password, and the variable name abcdef can be considered a username. It logs in or accesses a system through the login function, so the literal 123456 is sensitive information. If this related information is not available, the literal 123456 might be considered a timestamp and not sensitive information.

[0083] It is understandable that different literals have different use cases. Some may only exist in one or a few of the above use cases, while others may have other use cases, such as method assignment. When the use cases are subdivided, the corresponding use cases will be abstracted for different literals according to the data structure.

[0084] By segmenting scenarios, different use cases can be divided, facilitating expansion and avoiding the problem of neglecting one aspect for another. Scenario segmentation also covers assignment, calculation, parameter passing, and language features, which can significantly improve recall rate and operational efficiency.

[0085] In some embodiments, step 600 can be performed separately for the code to be identified in a certain language type. That is, for each code to be identified in a certain language type, steps 100-600 are performed sequentially to obtain all literals and their related information in different usage scenarios. Thus, if there are N types of code and each type of code has M usage scenarios, then in order to obtain the literals of all codes and their related information in different usage scenarios, N×M usage scenarios need to be implemented. In other embodiments, the code to be identified in multiple language types can also be uniformly subdivided into scenarios, i.e., step 600. That is, steps 100-500 are executed for each type of code to obtain their data structures. Since the data structures have the same format, scenario subdivision can be performed on the data structures of all codes simultaneously in step 600. In this way, for N types of code, each type of code has M usage scenarios. When obtaining the data structures of different codes, it is necessary to implement N times (once for each type of code). Then, when obtaining the literals of all codes and their related information in different usage scenarios, it is necessary to implement M times (i.e., once for each usage scenario). Finally, the total number of implementations is N+M times, which can greatly reduce the implementation difficulty.

[0086] After step 600, all literals in the code to be identified (one or more) and their related information in different usage scenarios can be extracted. These can be combined into a candidate set (i.e. sensitive information to be identified). The literals in this candidate set are not necessarily sensitive information, but need to be identified to determine whether they are sensitive information.

[0087] Because a syntax tree, a preset data structure, and scene segmentation are introduced in steps 100-600, the sensitive information to be identified in the code to be identified can be extracted efficiently and accurately, so as to facilitate the subsequent identification of sensitive information.

[0088] 700: For each literal, identify whether the literal is sensitive information based on the literal and related information in its usage scenario.

[0089] Steps 100-600 are primarily used for the general extraction of the candidate set, while step 700 is used to identify the literals in the extracted candidate set, determining whether each literal in the candidate set is sensitive information, thereby completing the identification of sensitive information. In some embodiments, the extraction of the candidate set and the identification of sensitive information can be separated (decoupled). That is, for the code to be identified in different language types, a candidate set can first be obtained through steps 100-600. The candidate set includes all literals of all the code to be identified and their relevant information in the usage scenario. Then, in step 700, the same identification algorithm is used to identify each literal in the candidate set to determine whether the literal is sensitive information. In this way, when a new language type of code needs to be added, it is only necessary to perform a general extraction in steps 100-600 and then identify it in step 700, without developing a new identification algorithm for sensitive information identification. This solves the problem of a surge in the development workload of multi-language access and reduces the implementation cost of new language access and optimization.

[0090] In some embodiments, existing regular expression matching methods can be used to identify sensitive information in the candidate set. For example, for data with a clear format and verification method, such as ID card numbers, a regular expression pattern with the same format as ID card numbers can be set. Then, this regular expression pattern is used to match literals in the candidate set. If a match is found, the literal is extracted and determined to be sensitive information. Otherwise, it indicates that it does not belong to data with a clear format and verification method, such as ID card numbers or mobile phone numbers, and further judgment is needed based on relevant information in the usage scenario of the literal. For data without a clear format and verification method, such as account passwords and private keys, the function of the literal can be determined based on relevant information in the usage scenario, thereby determining whether the literal is sensitive information. In some embodiments, the function of a literal can be determined by keyword retrieval. For example, keywords such as account, password, private key, etc. are set, and relevant information of all literals in the candidate set in the usage scenario is retrieved using these keywords. If the relevant information of a certain literal contains keywords, then the function of the literal is account, password or private key, etc., and it can be considered as sensitive information.

[0091] In other embodiments, a neural network model can be used to determine the function of a literal, thereby determining whether it belongs to sensitive information. For example, word2vec can be used to process the variable names in the relevant information of the literal in the usage scenario to generate word vectors. Then, the word vectors are matched with standard word vectors for similarity, and the function of the literal is determined by the standard word vectors. Alternatively, code vectorization techniques such as code2vec can be used to convert the function bodies used by the variables in the relevant information of the literal in the usage scenario into function vector representations. Then, similarity matching is performed with standard function vector representations, and the function of the literal is determined by the function of the standard function vectors, thereby determining whether it belongs to sensitive information. If the function of the literal is a password, private key, etc., it can be considered sensitive information. In some embodiments, the confidence level of the literal being sensitive information can also be determined based on the degree of similarity matching. For example, if the neural network model determines that the function of the literal is a private key, but the similarity level during similarity matching is 90%, then the probability that it belongs to sensitive information is 90%, that is, the confidence level is 90%.

[0092] Sensitive information can also be identified using other existing methods, such as sensitive information identification algorithms for structured data, and this invention does not limit this to any particular method.

[0093] Step 700 may further include: saving relevant information about literals in the candidate set that are useful for operations in different use cases for governance decisions, such as saving the URL used by the account password connection to confirm its activity level and clarify the risk level and / or saving the context information of the account password to confirm its type and business purpose (e.g., MySQL account password used to read ID card information to clarify its risk domain and association level) and pairing and identifying the occurrence of sensitive information in the candidate set in pairs based on the relevant information of the literals in different use cases (e.g., scope, context relevance, etc.), such as pairing and identifying account passwords.

[0094] The sensitive information identification method of this invention decouples the general extraction of the candidate set (i.e., steps 100-600) from the identification of sensitive information (i.e., step 700). While improving parsing capabilities compared to regular expression matching, it solves the problem of a surge in development workload when integrating code from multiple language types, reducing the implementation cost of integrating and optimizing code from new language types. The method of this invention can simultaneously cover usage scenarios such as assignment, calculation, parameter passing, and language characteristics, thereby significantly improving recall rate and operational efficiency. By designing a unified data format, code sharing across multiple language types can be achieved, significantly improving the general extraction capability of sensitive information from multi-language code and increasing accuracy.

[0095] like Figure 2 As shown, another embodiment of the present invention provides a sensitive information identification system, which includes a data acquisition module 10 for mutual data communication, a judgment module 20, a syntax tree transformation module 30, a first processing module 40, a second processing module 50, a third processing module 60, and an identification module 70.

[0096] Module 10 is set to acquire the code to be identified.

[0097] The judgment module 20 is set to determine the language type of the code to be recognized.

[0098] The code to be identified can be code text in various languages, including but not limited to Java, C++, Python, Yaml, Jason, and Java Profiles.

[0099] The syntax tree transformation module 30 is configured to transform the syntax tree of the code to be recognized according to the language type of the code to be recognized, and generate the syntax tree corresponding to the code to be recognized.

[0100] In some embodiments, Antlr can be used to perform syntax tree transformation on the code to be identified. Since Antlr automatically identifies the language type of the code and generates the corresponding syntax tree, in some embodiments, the judgment module 20 can also be implemented using Antlr.

[0101] After syntax tree transformation, the syntax tree corresponding to the code to be identified can be obtained, such as Java syntax tree, C++ syntax tree, Python syntax tree, YAML syntax tree, Jason syntax tree, or Java Profiles syntax tree, etc.

[0102] The first processing module 40 is configured to extract the symbol table related to literals in the code to be identified based on the syntax tree, and determine the relationship between variables, functions, classes, constants and comments in the symbol table.

[0103] Since sensitive information appears in the code as literals (i.e., strings), during syntax tree parsing, only the information related to the literals needs to be extracted, and then this information is further processed. In some embodiments, the literals in the symbol table related to the literals are the literals after performing the relevant operations. For example, the visitor mechanism of the syntax tree can be used to obtain the literals after the operations, which can make the identification of sensitive information more accurate.

[0104] In some embodiments, the first processing module 40 is further configured as follows:

[0105] Extract the variable name, variable type, associated variable, and variable scope fields, and generate a unique ID to improve query performance;

[0106] Extract the function name, function variable name, and the defined functions and variables used internally, and define the function's functional score based on the internal variable name and function name, and then define the function's attributes;

[0107] Obtaining the intermediate representation of a class declaration / call includes: extracting sub-variable fields of the subclass for defining the association between the class and sensitive types. Obtaining the intermediate representation of a constant includes: extracting direct literal constants (i.e., literals), constants to be calculated, associating them with the variable's unique ID, and obtaining the associated variable name, function call, class call, and method call as meta information.

[0108] Comments are functionally categorized into code snippets and explanatory text information. Code snippets are extracted a second time, while explanatory text information is directly integrated into rich text general detection.

[0109] In some embodiments, the extraction of the symbol table associated with literals can be accomplished through the syntax tree listener mechanism, which is a mechanism provided by Antlr for traversing the syntax tree and can automatically traverse the syntax tree.

[0110] The second processing module 50 is configured to use a preset data structure to represent the relationship between variables, functions, classes, constants and comments in the symbol table, wherein the preset data structure is the same for the code to be identified for different language types.

[0111] The data structure includes the names, locations, and relationships of all variables, functions, classes, constants, and comments in the code to be identified. It serves as the information source for sensitive information identification, and subsequent sensitive information identification needs to be performed based on this data structure. Since the same data structure is used for code in different language types, it can be processed uniformly.

[0112] The third processing module 60 is configured to determine the usage scenario of each literal in the code to be recognized based on the data structure, and obtain the relevant information of the literal in the usage scenario.

[0113] Since the data structure is only obtained when acquiring information related to literals, such as variables, functions, classes, etc., without knowing the role of literals in this information, it is impossible to directly identify sensitive information through the data structure. Therefore, in the third processing module 60, information aggregation is performed on the data structure using literals as the aggregation unit. That is, starting from the literal, all information such as variables and functions associated with the literal is found in the data structure, and the scenarios are subdivided to form different usage scenarios. This clarifies the relevant information of literals in different usage scenarios, including the role of literals in different usage scenarios and the role of variables and functions associated with the literals. This information is then used as auxiliary information to determine whether the literal is sensitive information.

[0114] In some embodiments, different use cases include, but are not limited to, direct assignment, indirect assignment, computational assignment, passing function constants as parameters, passing function variables as parameters, complex computational assignment, language features (such as getters / setters), and comments / rich text. Scenario segmentation involves parsing the literal to obtain the literal and its related information in different use cases.

[0115] It is understandable that different literals have different use cases. Some may only exist in one or a few of the above use cases, while others may have other use cases, such as method assignment. When the use cases are subdivided, the corresponding use cases will be abstracted for different literals according to the data structure.

[0116] By segmenting scenarios, different use cases can be divided, facilitating expansion and avoiding the problem of neglecting one aspect for another. Scenario segmentation also covers assignment, calculation, parameter passing, and language features, which can significantly improve recall rate and operational efficiency.

[0117] The third processing module 60 can extract all literals in the code to be identified (one or more) and their related information in different usage scenarios, which can be combined into a candidate set (i.e. sensitive information to be identified). The literals in the candidate set are not necessarily sensitive information, but need to be identified to determine whether they are sensitive information.

[0118] Since the preset data structure is the same for different language types of codes to be identified, the third processing module 60 can simultaneously subdivide the data structure of different language types of codes to be identified into scenarios, and obtain a candidate set composed of codes to be identified in all language types, so as to achieve efficient and accurate extraction of the candidate set, so as to facilitate the subsequent identification of sensitive information.

[0119] The recognition module 70 is configured to identify whether a literal is sensitive information based on the literal and related information in its usage scenario for each literal.

[0120] The acquisition module 10, judgment module 20, syntax tree transformation module 30, first processing module 40, second processing module 50, and third processing module 60 are mainly used for the general extraction of the candidate set, which is equivalent to the front end. The recognition module 70 is used to recognize the literals in the extracted candidate set and determine whether each literal in the candidate set is sensitive information, thereby completing the recognition of sensitive information, which is equivalent to the back end. In some embodiments, the extraction of the candidate set and the recognition of sensitive information (i.e., the front end and the back end) can be separated (decoupled). That is, for the code to be recognized in different language types, the candidate set of the code to be recognized in different language types can be obtained through the front end first. The candidate set includes all literals of all the code to be recognized and their relevant information in the usage scenario. Then, the same recognition algorithm is used by the back end to recognize each literal in the candidate set and determine whether the literal is sensitive information. In this way, when a new language type of code needs to be added, it is only necessary to perform general extraction on the front end and then recognize it on the back end, without developing a new recognition algorithm to recognize sensitive information. This solves the problem of the surge in development workload for multi-language access and reduces the implementation cost of new language access and optimization.

[0121] In some embodiments, the identification module 70 can be configured to identify sensitive information in the candidate set using existing regular expression matching methods. For example, for data with a clear composition format and verification method, such as ID card numbers, a regular expression pattern with the same format as ID card numbers can be set. Then, this regular expression pattern is used to match literals in the candidate set. If a match is found, the literal is extracted and determined to be sensitive information. Otherwise, it indicates that it does not belong to data with a clear composition format and verification method, such as ID card numbers or mobile phone numbers, and further judgment is needed based on relevant information in the usage scenario of the literal. For data without a clear composition format and verification method, such as account passwords and private keys, the function of the literal can be determined based on relevant information in the usage scenario of the literal, thereby determining whether the literal is sensitive information. In some embodiments, the function of a literal can be determined by keyword retrieval. For example, keywords such as account, password, private key, etc. are set, and relevant information of all literals in the candidate set in the usage scenario is retrieved using these keywords. If the relevant information of a certain literal contains keywords, then the function of the literal is account, password or private key, etc., and it can be considered as sensitive information.

[0122] In other embodiments, the identification module 70 can also be configured to use a neural network model to determine the function of the literal, thereby determining whether it belongs to sensitive information based on the function of the literal. For example, word2vec can be used to process the variable names in the relevant information of the literal in the usage scenario to generate word vectors, and then the word vectors are matched with standard word vectors for similarity, and then the function of the literal is determined by the standard word vectors; or, code2vec and other code vectorization techniques can be used to convert the function bodies used by the variables in the relevant information of the literal in the usage scenario into function vector representations, and then the similarity is matched with standard function vector representations, and then the function of the literal is determined by the function of the standard function vectors, thereby determining whether it belongs to sensitive information based on the function of the literal. If the function of the literal is a password, private key, etc., it can be considered as sensitive information. In some embodiments, the confidence level of the literal being sensitive information can also be determined based on the degree of similarity matching. For example, if the neural network model determines that the function of the literal is a private key, but the similarity level during similarity matching is 90%, then the probability that it belongs to sensitive information is 90%, that is, the confidence level is 90%.

[0123] Sensitive information can also be identified using other existing methods, such as sensitive information identification algorithms for structured data, and this invention does not limit this to any particular method.

[0124] The identification module 70 can also be configured to: save relevant information of literals useful for operation in different usage scenarios in the candidate set for governance decisions, such as saving the URL used by the account password connection to confirm its activity level and clarify the risk level and / or saving the context information of the account password to confirm its type and business purpose (e.g., MySQL account password used to read ID card information to clarify its risk domain and association level) and pair and identify the occurrence of sensitive information in pairs in the candidate set based on the relevant information of the literals in different usage scenarios (e.g., scope, context relevance, etc.), such as pairing and identifying account passwords.

[0125] The sensitive information identification system of this invention decouples the general extraction of candidate sets (i.e., front-end) from the identification of sensitive information (i.e., back-end). While improving parsing capabilities compared to regular expression matching, it solves the problem of a surge in development workload when integrating code from multiple language types, reducing the implementation cost of integrating and optimizing code from new language types. The method of this invention can simultaneously cover usage scenarios such as assignment, calculation, parameter passing, and language characteristics, thereby significantly improving recall rate and operational efficiency. By designing a unified data format, code sharing across multiple language types can be achieved, significantly improving the general extraction capability of sensitive information from multi-language code and increasing accuracy.

[0126] Another embodiment of the present invention provides a readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the steps of the sensitive information identification method in the above embodiments of the present invention.

[0127] Another embodiment of the present invention provides a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it performs the steps of the sensitive information identification method in the above embodiments of the present invention.

[0128] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. Various variations can be made to the above embodiments of the present invention. That is, all simple and equivalent changes and modifications made based on the claims and description of this invention fall within the protection scope of the claims of this patent. All aspects not described in detail in this invention are conventional technical content.

Claims

1. A method for identifying sensitive information, comprising the following steps: Obtain the code to be recognized; Determine the language type of the code to be identified; Based on the language type of the code to be identified, a syntax tree transformation is performed on the code to be identified to generate a syntax tree corresponding to the code to be identified; Based on the syntax tree, extract the symbol table related to literals in the code to be identified, and determine the relationship between variables, functions, classes, constants and comments in the symbol table; A pre-defined data structure is used to represent the relationships between variables, functions, classes, constants, and comments in the symbol table, wherein... The pre-defined data structure is the same for codes to be recognized in different language types; For each literal, based on the data structure, the usage scenario of the literal in the code to be identified is determined, and the relevant information of the literal in the usage scenario is obtained; the usage scenario includes direct assignment, indirect assignment, computational assignment, passing function constant parameters, passing function variable parameters, complex computational assignment, language features, and comments / rich text; the relevant information of the literal in the usage scenario includes the variables, functions, and classes associated with the literal in the usage scenario, as well as the role of the literal, variables, functions, and classes; For each literal, it is determined whether the literal is sensitive information based on the literal and its relevant information in the usage scenario.

2. The sensitive information identification method according to claim 1, wherein Antlr is used to perform syntax tree transformation on the code to be identified.

3. The sensitive information identification method according to claim 2, wherein extracting the symbol table related to literals in the code to be identified based on the syntax tree includes: The literal-related operations are performed through the visitor mechanism of the syntax tree to obtain the literal after the operation, and the symbol table related to the literal after the operation is extracted through the listener mechanism of the syntax tree.

4. The sensitive information identification method according to claim 1, wherein extracting the symbol table related to literals in the code to be identified based on the syntax tree, and determining the relationship between variables, functions, classes, constants, and comments in the symbol table includes: Extract the variable name, variable type, associated variable, and variable scope fields, and generate a unique ID; Extract the function name, function variable name, and the defined functions and variables used internally, and define the function's functional score based on the internal variable name and function name, and then define the function's attributes; Extract sub-variable fields from subclasses; Extract direct literal constants and constants to be calculated, and associate them with the unique IDs of the variable table. Obtain the associated variable names, function calls, class calls, and method calls from the variable table as meta information. Comments are functionally categorized into code snippets and text information. Code snippets are extracted a second time, while text information is directly integrated into a general rich text detection tool.

5. The sensitive information identification method according to claim 1, wherein a regular expression matching method is used to identify whether each literal is sensitive information; and / or, Based on the relevant information of each literal in the usage scenario, keyword retrieval or neural network model is used to determine the function of each literal, and then the function of each literal is used to determine whether it is sensitive information.

6. A sensitive information identification system, comprising mutual data communication: The acquisition module is set to acquire the code to be recognized; The judgment module is set to determine the language type of the code to be recognized. The syntax tree transformation module is configured to transform the syntax tree of the code to be recognized based on the language type of the code to be recognized, and generate the syntax tree corresponding to the code to be recognized. The first processing module is configured to extract the symbol table related to literals in the code to be identified based on the syntax tree, and determine the relationship between variables, functions, classes, constants and comments in the symbol table; The second processing module is configured to use a preset data structure to represent the relationship between variables, functions, classes, constants and comments in the symbol table, wherein the preset data structure is the same for the code to be identified for different language types; The third processing module is configured to, for each literal, determine the usage scenario of the literal in the code to be recognized based on the data structure, and obtain the relevant information of the literal in the usage scenario; the usage scenario includes direct assignment, indirect assignment, computational assignment, function constant parameter passing, function variable parameter passing, complex computational assignment, language features, and comments / rich text; the relevant information of the literal in the usage scenario includes the variables, functions, and classes associated with the literal in the usage scenario, as well as the role of the literal, variables, functions, and classes; The recognition module is configured to identify whether a literal is sensitive information based on the literal and related information in its usage scenario for each literal.

7. In the sensitive information identification system according to claim 6, the syntax tree transformation module uses Antlr to perform syntax tree transformation on the code to be identified.

8. The sensitive information identification system according to claim 7, wherein extracting the symbol table related to literals in the code to be identified based on the syntax tree includes: The literal-related operations are performed through the visitor mechanism of the syntax tree to obtain the literal after the operation, and the symbol table related to the literal after the operation is extracted through the listener mechanism of the syntax tree.

9. The sensitive information identification system according to claim 6, wherein the first processing module is further configured as follows: Extract the variable name, variable type, associated variable, and variable scope fields, and generate a unique ID; Extract the function name, function variable name, and the defined functions and variables used internally, and define the function's functional score based on the internal variable name and function name, and then define the function's attributes; Extract sub-variable fields from subclasses; Extract direct literal constants and constants to be calculated, and associate them with the unique IDs of the variable table. Obtain the associated variable names, function calls, class calls, and method calls from the variable table as meta information. Comments are functionally categorized into code snippets and text information. Code snippets are extracted a second time, while text information is directly integrated into a general rich text detection tool.

10. The sensitive information identification system according to claim 6, wherein the identification module uses a regular expression matching method to identify whether each literal is sensitive information; and / or, Based on the relevant information of each literal in the usage scenario, keyword retrieval or neural network model is used to determine the function of each literal, and then the function of each literal is used to determine whether it is sensitive information.

11. A readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the steps of the sensitive information identification method as described in any one of claims 1-5.

12. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it performs the steps of the sensitive information identification method as described in any one of claims 1-5.

Images