A code-level sensitive information detection method, an electronic device, and a storage medium

Abstract

The application relates to the technical field of code detection, in particular to a code-level sensitive information detection method, an electronic device and a storage medium, the method comprises the following steps: extracting a candidate string constant from an abstract syntax tree corresponding to a code file to be detected, dynamically determining a corresponding determination threshold according to the character length and the character set type of the candidate string constant, screening the candidate string constant with information entropy higher than the corresponding determination threshold as a to-be-detected string constant, extracting the context semantic features of each to-be-detected string constant from the abstract syntax tree to form a corresponding context feature vector, inputting the context feature vector of each to-be-detected string constant into a preset classifier, and finally generating a sensitive information detection result; the application can realize high-precision and low-false-alarm identification of sensitive information by dynamically adjusting the determination threshold and combining the context semantic feature extraction with a lightweight classifier.

Description

Technical Field

[0001] This invention relates to the field of code detection technology, and in particular to a method, electronic device and storage medium for detecting sensitive code-level information. Background Technology

[0002] In modern software development, sensitive information such as passwords, API keys, access tokens, and personal identification information often exists in the code, constituting serious security vulnerabilities. Current mainstream detection methods mainly rely on rule-based matching methods, such as regular expressions, static entropy-based detection methods, and machine learning-based identification methods. However, these methods all have significant limitations. Rule-based matching methods have a high false positive rate and struggle to identify emerging sensitive information types without predefined formats, making them easily bypassed by simple code obfuscation techniques. Static entropy-based detection methods perform poorly with short keys and non-random strings such as simple passwords, and easily misclassify random, non-sensitive data such as UUIDs and hash values ​​as sensitive information. Machine learning-based methods heavily rely on the quality and quantity of training data, have poor model interpretability, and their accuracy needs improvement.

[0003] To address the aforementioned issues, there is an urgent need for a code-level sensitive information detection method that boasts high accuracy in identifying sensitive information and low false alarm rate. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a code-level sensitive information detection method, electronic device, and storage medium. By dynamically adjusting the judgment threshold and combining contextual semantic feature extraction with a lightweight classifier, it is possible to achieve high-precision and low-false-report identification of sensitive information.

[0005] According to a first aspect of the present invention, a method for detecting code-level sensitive information is provided, comprising the following steps:

[0006] S1 parses the code file to be detected, generates an abstract syntax tree, and extracts candidate string constants from the abstract syntax tree.

[0007] S2, calculate the information entropy of each candidate string constant, and dynamically determine the corresponding judgment threshold according to the character length and character set type of the candidate string constant. Select the candidate string constants with information entropy higher than the corresponding judgment threshold as the string constants to be detected.

[0008] S3 extracts the contextual semantic features of each string constant to be detected from several nodes of the abstract syntax tree to form the contextual feature vector of each string constant to be detected.

[0009] S4, input the context feature vector of each string constant to be detected into a pre-trained preset classifier, the preset classifier outputs the confidence score of each string constant to be detected as sensitive information, and generates a sensitive information detection result based on the confidence score; the sensitive information detection result includes the sensitive information judgment result, the corresponding code location and risk level.

[0010] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the above-described code-level sensitive information detection method.

[0011] According to a third aspect of the present invention, an electronic device is provided, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0012] The present invention has at least the following beneficial effects:

[0013] This invention provides a code-level sensitive information detection method. First, candidate string constants are extracted from the abstract syntax tree (AST) of the code file to be detected. Introducing the AST achieves preliminary screening of string constants, avoiding invalid detections. Then, a corresponding judgment threshold is dynamically determined based on the character length and character set type of the candidate string constants. Candidate string constants with information entropy higher than the corresponding judgment threshold are selected as string constants to be detected. By introducing a dynamic judgment threshold, targeted judgment of different strings is achieved, significantly reducing the false alarm rate and accurately filtering non-sensitive information strings. Next, the contextual semantic features of each string constant to be detected are extracted from the AST to form a corresponding contextual feature vector. The contextual feature vector of each string constant to be detected is input into a preset classifier, finally generating sensitive information detection results. By introducing the contextual semantic features of the string constants to be detected and training the preset classifier, complex contextual relationships are comprehensively considered, enabling the system to judge based on the actual use and environment of the string constant in the code. This overcomes the limitations of simply relying on text pattern matching, significantly improving the accuracy of sensitive information recognition while reducing the false alarm rate. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1A flowchart of a code-level sensitive information detection method provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] This invention provides a method for detecting sensitive information at the code level, such as... Figure 1 As shown, the method includes the following steps:

[0018] S1 involves parsing the code file to be tested, generating an abstract syntax tree (AST), and extracting candidate string constants from the AST. For example, using an existing ANTLR parsing library to perform lexical and syntactic analysis on the code file generates an AST, which can accurately identify code elements such as string constants, variable assignments, and function call parameters.

[0019] Specifically, step S1 also includes the following steps:

[0020] S101, the extracted candidate string constants are matched with a preset regular expression rule library; each rule in the regular expression rule library corresponds to a preset information type pattern and is associated with a sensitivity score; it can be understood that when a candidate string constant successfully matches a certain rule, the candidate string constant is determined as the information type pattern corresponding to the successfully matched rule. For example, the string constant "password = null" may successfully match a "password assignment pattern", but it is obviously not a real password and needs to be filtered later to solve the false alarm.

[0021] Specifically, information type patterns include, but are not limited to, passwords, API keys, database connection strings, and email addresses.

[0022] Specifically, the sensitivity score refers to the sensitivity risk value pre-set for the corresponding information type pattern.

[0023] S102, for the candidate string constants that match successfully, query the corresponding static whitelist for filtering; this can be understood as: if it is in the static whitelist, the candidate string constant is considered to be non-sensitive information and is filtered out.

[0024] S103, the filtered string constants are used as new candidate string constants, and the sensitivity score associated with each new candidate string constant is added to the context semantic features corresponding to the candidate string constant itself.

[0025] As described above, by parsing the code to generate an abstract syntax tree and extracting candidate string constants, precise structured analysis of the code is achieved. This can accurately distinguish between executable code and non-execution paths such as comments and log output, avoiding invalid detection from the source and greatly improving the targeting and efficiency of detection. Furthermore, by matching the regular expression rule base and filtering the whitelist, common false alarm strings can be filtered out, which is conducive to improving the accuracy of subsequent identification of sensitive information.

[0026] S2 calculates the information entropy of each candidate string constant and dynamically determines the corresponding judgment threshold based on the character length and character set type of the candidate string constant. Candidate string constants with information entropy higher than the corresponding judgment threshold are selected as string constants to be detected. For example, character set types include pure numbers, numbers + letters, hexadecimal character set, Base64 character set, etc.

[0027] Specifically, the information entropy H of the candidate string constants meets the following conditions:

[0028] Where n is the number of unique characters in the candidate string constant, and P i Let be the probability of the i-th character appearing in the candidate string constant. For example, the string "abca" has 3 unique character types.

[0029] Furthermore, the judgment threshold corresponding to the candidate string constant meets the following conditions:

[0030] K = log₂N × α × β + γ, where K is the judgment threshold corresponding to the candidate string constant, N is the character set size of the character set type corresponding to the candidate string constant, α is the preset length adaptive coefficient, β is the preset length penalty factor, and γ is the basic threshold offset value set based on the character set type; where β is positively correlated with the character length L corresponding to the candidate string constant. For example, when L < 6, β = 0.5; when 6 ≤ L < 12, β = 0.7; and when L ≥ 12, β = 1.

[0031] Specifically, Where u is the preset attenuation coefficient related to the character set type of the candidate string constant, and L min This is the minimum valid length for the character set type corresponding to the candidate string constant. For example, keys are usually no less than 8 characters, with a default attenuation factor of 0.3 for pure numeric character sets and 0.1 for Base64 character sets.

[0032] Furthermore, the base threshold offset is inversely proportional to the complexity of the character set type. For example, pure numeric characters are a simple character set, so γ is set to 0.5, while Base64 is a complex character set, so γ is set to 0.2.

[0033] The above-mentioned method dynamically determines the judgment threshold by calculating information entropy and combining it with string length and character set type, realizing intelligent evaluation of string randomness. Log2N is used to set different judgment baselines for different character set types. On this basis, α is set to set different sensitivities for strings of different lengths to compensate for the insufficient entropy value of short strings. β is set to prevent the situation where excessively short strings are misjudged even if they have high entropy. γ is used for fine-tuning. Overall, this method achieves a high detection rate for high-risk, low-entropy sensitive information such as short keys and simple passwords, while effectively filtering out high-entropy non-sensitive information such as long hashes and random identifiers, significantly reducing the false alarm rate and achieving a dual improvement in detection rate and accuracy.

[0034] In a further embodiment, the method further includes the following steps:

[0035] T1 retrieves the historical sensitive information judgment results obtained from the most recent M judgments based on the judgment threshold, and calculates the false positive rate and false negative rate of the M historical sensitive information judgment results. Those skilled in the art are familiar with the specific calculation methods of the false positive rate and false negative rate, and will not elaborate further here.

[0036] T2, based on the false alarm rate and false negative rate, adjust the subsequently determined judgment threshold according to a preset adjustment strategy; the preset adjustment strategy is as follows:

[0037] If the false alarm rate is greater than the first preset threshold and the false negative rate is not greater than the second preset threshold, then K is updated to (K+ΔK), where ΔK is the preset adjustment step size. For example, ΔK is set to 0.1.

[0038] If the false alarm rate is not greater than the first preset threshold and the false negative rate is greater than the second preset threshold, then K is updated to (K-ΔK).

[0039] If the false alarm rate is greater than the first preset threshold and the false negative rate is greater than the second preset threshold, a prompt will be sent to the user so that the user can change the method of obtaining the judgment threshold.

[0040] If the false alarm rate is not greater than the first preset threshold and the false negative rate is not greater than the second preset threshold, then K remains unchanged.

[0041] The above-mentioned approach achieves adaptive optimization of entropy by obtaining the false alarm rate and false negative rate in actual detection and dynamically adjusting the judgment threshold. This enables the system to automatically balance the detection rate and accuracy during use, and to proactively issue warnings when both indicators deteriorate simultaneously, prompting parameter calibration. This improves the robustness of the adjustment strategy while ensuring the accuracy of the detection system.

[0042] S3 extracts the contextual semantic features of each string constant to be detected from several nodes of the abstract syntax tree to form the contextual feature vector of each string constant to be detected.

[0043] Specifically, the contextual semantic features include at least: variable name features containing preset keywords, type features of parent nodes in the abstract syntax tree, function call context features, and code location features. For example, whether the variable name assigned to the string constant to be detected contains preset keywords such as key, secret, token, pwd; whether the direct parent node of the string constant to be detected is a variable declaration, function argument, or return value; if the string constant to be detected is a parameter of a function, then analyze the name and source of the function, and analyze whether its source is a known encryption library or logging API; analyze whether the string constant to be detected is in a configuration file such as .json or .yaml, or in the logical path of the source code.

[0044] As described above, by obtaining the contextual semantic features of the string constants to be detected from multiple dimensions in the abstract syntax tree, the combination of structured information and semantic information of the code is realized, enabling the system to understand the actual use and environment of the string constants to be detected in the code, overcoming the one-sidedness of simply relying on text pattern matching, and significantly improving the recognition accuracy of sensitive information.

[0045] S4: Input the context feature vector of each string constant to be detected into a pre-trained preset classifier. The preset classifier outputs the confidence score of each string constant to be detected as sensitive information, and generates a sensitive information detection result based on the confidence score. This can be understood as follows: when the output confidence score is greater than a preset probability threshold, the string constant to be detected is determined to be sensitive information. For example, sensitive information can be keys, tokens, passwords, personal identification information, etc.

[0046] Specifically, the sensitive information detection results include the sensitive information determination results, the corresponding code location, and the risk level.

[0047] Preferred, the preset classifier adopts any one of the following: logistic regression model, gradient boosting tree model, or neural network model.

[0048] As described above, the context feature vector is input into a preset classifier and the confidence score is output. This comprehensively considers the complex contextual relationships, effectively distinguishes between seemingly similar sensitive and non-sensitive information, and supports the generation of accurate judgment reports containing risk levels with quantified confidence scores. This significantly reduces the false alarm rate and improves the operability of the results.

[0049] Specifically, the training process of the preset classifier is as follows:

[0050] S01, for each obtained code file sample, execute steps S1-S3 to extract the context feature vector corresponding to each string constant sample, and use the pre-annotated sensitive information judgment result of each string constant sample as a label to construct a training sample set.

[0051] S02, the preset machine learning model is trained using the training sample set. By optimizing the loss function in the preset machine learning model, the preset machine learning model learns the correlation between the context feature vector and the sensitive information, and the initial classifier is obtained.

[0052] S03. The initial classifier is evaluated using a pre-built validation dataset. The model parameters of the initial classifier are tuned based on the evaluation results until the preset performance index is met, thus obtaining the trained preset classifier. This can be understood as: meeting the preset performance index means that the classification error is less than the preset value.

[0053] Furthermore, the risk level of the string constant to be detected is determined through the following steps:

[0054] S401, based on the confidence level of the constant string to be detected output by the preset classifier as sensitive information, obtain the confidence level factor corresponding to the confidence level; wherein, the confidence level factor is positively correlated with the confidence level. In one embodiment, the confidence level factor is segmented according to the numerical range of the confidence level. For example, when the confidence level is less than 0.5, the confidence level factor is 0.6; when the confidence level is not less than 0.5 but less than 0.8, the confidence level factor is 1; and when the confidence level is not less than 0.8, the confidence level factor is 1.5.

[0055] S402, calculate the sensitivity risk score of the string constant to be detected based on the confidence level factor and the information type of the string constant to be detected identified based on the preset regular expression rule base; the sensitivity risk score S of the string constant to be detected meets the following conditions:

[0056] S = W × (1 + ζ × P) × d, where W is the preset basic risk weight corresponding to the information type of the string constant to be detected, ζ is the preset confidence weight coefficient, P is the confidence level, and d is the confidence level factor. For example, the basic risk weight of a plaintext private key is greater than the basic risk weight of an email address.

[0057] It should be noted that ζ is used to adjust the magnitude of the influence of confidence level on the final sensitive risk score, d is equivalent to a non-linear regulator, used to amplify the risk of high confidence results or reduce the risk of low confidence results, making the risk classification more discriminative, and (1+ζ×P) is used to ensure that even if the model confidence level is 0, its corresponding value is not less than 1, thus preserving the influence of the basic risk weight.

[0058] S403, based on a preset sensitive risk score range, map the risk level of the string constant to be detected from several preset risk levels; the several preset risk levels include high risk, medium risk and low risk.

[0059] The above describes how a quantitative comprehensive risk scoring model is constructed by combining rule matching information, confidence level, and nonlinear adjustment factor. This model enables refined classification of detection results, accurately distinguishing between high-risk genuine keys and low-risk suspected strings. By transforming subjective risk assessment into a reasonable and accurate calculation process, the final risk level is more precise and reliable.

[0060] Furthermore, the method also includes the following steps:

[0061] S10: Based on the sensitive information detection results, receive feedback from the user confirming or correcting the sensitive information detection results.

[0062] S20. Based on the feedback results, identify the sensitive strings that were missed and the non-sensitive strings that were falsely reported, and use them as positive and negative examples, respectively.

[0063] S30, add the positive sample, negative sample, and their corresponding context feature vectors to the training sample set for incremental training of the preset classifier.

[0064] As described above, by introducing an incremental learning mechanism based on user feedback, the detection system achieves self-optimization and continuous evolution. It can automatically transform real false positives and false negatives into high-quality training samples, driving the classifier to continuously adapt to new code patterns, effectively improving the sustainability and practicality of detection accuracy.

[0065] Embodiments of the present invention also provide a non-transitory computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the code-level sensitive information detection method provided in the above embodiments.

[0066] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0067] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A method for detecting sensitive information at the code level, characterized in that, The method includes the following steps: S1: Parse the code file to be detected, generate an abstract syntax tree, and extract candidate string constants from the abstract syntax tree; S2, calculate the information entropy of each candidate string constant, and dynamically determine the corresponding judgment threshold based on the character length and character set type of the candidate string constant. Filter candidate string constants with information entropy higher than the corresponding judgment threshold as string constants to be detected. The judgment threshold corresponding to the candidate string constant meets the following condition: K = log2N × α × β + γ, where K is the judgment threshold corresponding to the candidate string constant, N is the character set size of the character set type corresponding to the candidate string constant, α is a preset length adaptive coefficient, β is a preset length penalty factor, and γ is the basic threshold offset value set based on the character set type. β is positively correlated with the character length corresponding to the candidate string constant. S3, extract the contextual semantic features of each string constant to be detected from several nodes of the abstract syntax tree to form the contextual feature vector of each string constant to be detected; the contextual semantic features include at least: variable name features containing preset keywords, type features of the parent node in the abstract syntax tree, function call context features, and code position features; S4, input the context feature vector of each string constant to be detected into a pre-trained preset classifier, the preset classifier outputs the confidence score of each string constant to be detected as sensitive information, and generates a sensitive information detection result based on the confidence score; the sensitive information detection result includes the sensitive information judgment result, the corresponding code location and risk level.

2. The code-level sensitive information detection method according to claim 1, characterized in that, Step S1 also includes the following steps: S101, the extracted candidate string constants are matched with a preset regular expression rule library; each rule in the regular expression rule library corresponds to a preset information type pattern and is associated with a sensitivity score; S102, for the candidate string constants that match successfully, query the corresponding static whitelist for filtering; S103, the filtered string constants are used as new candidate string constants, and the sensitivity score associated with each new candidate string constant is added to the context semantic features corresponding to the candidate string constant itself.

3. The code-level sensitive information detection method according to claim 1, characterized in that, The method further includes the following steps: T1, obtain the historical sensitive information judgment results obtained after the most recent M judgments based on the judgment threshold, and calculate the false alarm rate and false negative rate of the M historical sensitive information judgment results; T2, based on the false alarm rate and false negative rate, adjust the subsequently determined judgment threshold according to the preset adjustment strategy; The preset adjustment strategy is as follows: If the false alarm rate is greater than the first preset threshold and the false negative rate is not greater than the second preset threshold, then K is updated to (K+ΔK), where ΔK is the preset adjustment step size; If the false alarm rate is not greater than the first preset threshold and the false negative rate is greater than the second preset threshold, then K is updated to (K-ΔK). If the false alarm rate is greater than the first preset threshold and the false negative rate is greater than the second preset threshold, a prompt will be sent to the user so that the user can change the method of obtaining the judgment threshold. If the false alarm rate is not greater than the first preset threshold and the false negative rate is not greater than the second preset threshold, then K remains unchanged.

4. The code-level sensitive information detection method according to claim 1, characterized in that, The training process of the preset classifier is as follows: S01, For each code file sample obtained, execute steps S1-S3 to extract the context feature vector corresponding to each string constant sample, and use the pre-annotated sensitive information judgment result of each string constant sample as a label to construct a training sample set; S02, the preset machine learning model is trained using the training sample set. By optimizing the loss function in the preset machine learning model, the preset machine learning model learns the relationship between the context feature vector and the sensitive information, and the initial classifier is obtained. S03. The initial classifier is evaluated using a pre-built validation dataset. The model parameters of the initial classifier are tuned based on the evaluation results until the preset performance indicators are met, thus obtaining the trained preset classifier.

5. The code-level sensitive information detection method according to claim 1, characterized in that, The risk level of the string constant to be detected is determined through the following steps: S401, based on the confidence level of the constant string to be detected output by the preset classifier as sensitive information, obtain the confidence level factor corresponding to the confidence level; wherein, the confidence level factor is positively correlated with the confidence level; S402, calculate the sensitivity risk score of the string constant to be detected based on the confidence level factor and the information type of the string constant to be detected identified based on the preset regular expression rule base; the sensitivity risk score S of the string constant to be detected meets the following conditions: S = W × (1 + ζ × P) × d, where W is the preset basic risk weight corresponding to the information type of the string constant to be detected, ζ is the preset confidence weight coefficient, P is the confidence level, and d is the confidence level factor. S403, based on a preset sensitive risk score range, map the risk level of the string constant to be detected from several preset risk levels; the several preset risk levels include high risk, medium risk and low risk.

6. The code-level sensitive information detection method according to claim 1, characterized in that, The method further includes the following steps: S10, Based on the sensitive information detection results, receive feedback from the user regarding confirmation or correction of the sensitive information detection results; S20, based on the feedback results, identify the sensitive strings that were missed and the non-sensitive strings that were falsely reported, and use them as positive and negative examples respectively; S30, add the positive sample, negative sample, and their corresponding context feature vectors to the training sample set for incremental training of the preset classifier.

7. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program segment, characterized in that, The at least one instruction or the at least one program segment is loaded and executed by the processor to implement the code-level sensitive information detection method as described in any one of claims 1-6.

8. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 7.

Images