Automatic discovery method and device for interactive interface and potential sensitive data in typical scene, storage medium and electronic equipment
By building a knowledge base of business needs and combining rule engines and machine learning algorithms, interactive interfaces and sensitive data are automatically discovered, solving the problems of low identification efficiency and high maintenance costs in the existing technology, and achieving efficient and accurate identification and adaptability renew.
Patent Information
- Application Number
- CN202411007794.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
AI Technical Summary
Prior art When identifying interactive interfaces and sensitive data, manual writing rule sets is difficult to cover all patterns, and static analysis technology lacks flexibility and adaptability, resulting in low recognition efficiency and high maintenance costs.
By building a knowledge base based on business needs, combining rule engines and machine learning algorithms, the interactive interfaces and sensitive data are automatically discovered, and cluster analysis and similarity analysis confirm the recognition results to reduce manual maintenance work.
Improve identification efficiency and accuracy, adapt to dynamically changing system environments, and reduce system maintenance costs and complexity.
Smart Images

Figure CN118964632A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and is a method, device, storage medium and electronic device for automatically discovering interactive interfaces and potential sensitive data in typical scenarios. Background Art
[0002] In the prior art, methods for identifying interactive interfaces and sensitive data usually rely on manually written rules and static analysis techniques. These methods mainly use predefined rule sets and pattern matching techniques to scan and parse the data flow in the system to identify potential interactive interfaces and sensitive data. These methods can provide a certain degree of accuracy under certain conditions, but their effectiveness is usually limited by the completeness of the rule set and the maintenance cost.
[0003] However, existing technologies face some major problems when dealing with dynamic and changing system environments. First, manually written rule sets are difficult to cover all possible interactive interfaces and sensitive data patterns, which can easily lead to missed reports and false positives. Second, static analysis technology lacks sufficient flexibility and adaptability when facing complex and real-time changing data streams, and cannot update and optimize recognition rules in a timely manner. In addition, maintaining and updating rule sets requires a lot of manpower investment, which increases the complexity and cost of system maintenance. Summary of the invention
[0004] The present invention provides a method, device, storage medium and electronic device for automatically discovering interactive interfaces and potential sensitive data in typical scenarios, which overcomes the shortcomings of the above-mentioned prior art and can effectively solve the problem that the prior art only relies on rule engines to identify interactive interfaces and sensitive data, and the rule update requires a lot of manpower investment and is not timely, resulting in low recognition efficiency.
[0005] One of the technical solutions of the present invention is achieved by the following measures: a method for automatically discovering interactive interfaces and potential sensitive data in a typical scenario, comprising:
[0006] Based on preset business requirements and application scenarios, a knowledge base including multiple typical interaction scenario categories is constructed, wherein the knowledge base includes interface features and data features of each typical interaction scenario category;
[0007] Scan and identify the target system based on the rule engine and recognition model to obtain the initial interactive interface and potential sensitive data. The rule engine is formed by an initial rule set established by a knowledge base including multiple typical interactive scenario categories, and the recognition model is obtained by machine learning using a number of real interface and potential sensitive data samples in various business scenarios;
[0008] Perform similarity analysis on the initial interaction interface and potential sensitive data with the knowledge base, and classify them into corresponding typical interaction scenarios according to the similarity scores;
[0009] Cluster analysis is used to analyze the classification results and confirm the target interactive interface and sensitive data.
[0010] The following are further optimizations and / or improvements to the above technical solutions:
[0011] Based on the preset business needs and application scenarios, a knowledge base including multiple typical interaction scenario categories is constructed, including:
[0012] Collect historical business log data for each typical interaction scenario category;
[0013] Based on historical business log data, extract interface features and data features of each typical interaction scenario category. Interface features include API path, request method, request parameter format, and response data format. Data features include potential sensitive data types, data field names, data formats, and data lengths.
[0014] The interface features and data features of each typical interaction scenario category are optimized using the expert database, and a knowledge base including multiple typical interaction scenario categories is formed based on the optimized interface features and data features of each typical interaction scenario category.
[0015] The target system is scanned and identified based on the rule engine and recognition model to obtain the initial interactive interface and potential sensitive data, including:
[0016] Building an initial rule set based on the knowledge base, and using the initial rule set to preliminarily scan the target system to obtain a first candidate interface and a set of potential sensitive data, wherein the initial rule set includes rules for identifying common interface types and potential sensitive data patterns;
[0017] The target system is fully scanned using the recognition model to obtain the second candidate interface and potential sensitive data set, wherein the recognition model is obtained by machine learning using a number of real interfaces and potential sensitive data samples in various business scenarios;
[0018] The first candidate interface and the potential sensitive data set are merged with the second candidate interface and the potential sensitive data set, and a weighted voting mechanism is used to generate a corresponding comprehensive result;
[0019] Confidence scoring is performed on the comprehensive results, and the candidate interfaces and potential sensitive data are divided into a preliminary identification result set and a result set to be verified based on the scoring results;
[0020] Using the result set to be verified, the initial rule set and the recognition model are updated, and the target system is scanned and recognized again using the updated initial rule set and the recognition model;
[0021] The above steps are repeated until the confidence scores of all results to be verified exceed the preset threshold or the maximum number of iterations is reached, and the preliminary recognition result set is output to obtain the final preliminary recognition interaction interface and potential sensitive data.
[0022] The above-mentioned updating of the initial rule set and the recognition model using the result set to be verified, and re-scanning and recognizing the target system using the updated initial rule set and the recognition model, includes:
[0023] Generate test cases based on interface features and data features corresponding to each result to be verified in the result set to be verified;
[0024] Execute test cases on the target system and obtain execution results and system responses;
[0025] Analyze the execution results of test cases and update the parameters of the recognition model and the rules of the rule engine.
[0026] The above-mentioned similarity analysis is performed on the initial interaction interface and potential sensitive data with the knowledge base, and they are classified into corresponding typical interaction scenarios according to the similarity scores, including:
[0027] The cosine similarity algorithm is used to calculate the similarity between the initial interaction interface and potential sensitive data and multiple typical interaction scenario categories in the knowledge base;
[0028] Based on the similarity scores, the initial interaction interfaces and potential sensitive data are classified into the most matching typical interaction scenario categories.
[0029] The above cluster analysis is used to analyze the classification results and confirm the target interactive interface and sensitive data, including:
[0030] The K-means clustering algorithm is used to perform cluster analysis on the classification results to obtain multiple cluster groups;
[0031] Perform feature analysis on each cluster group to identify and mark the target interaction interfaces and sensitive data.
[0032] The second technical solution of the present invention is achieved by the following measures: a method for automatically discovering interactive interfaces and potential sensitive data in a typical scenario, comprising:
[0033] A knowledge base construction unit, based on preset business requirements and application scenarios, constructs a knowledge base including multiple typical interaction scenario categories, wherein the knowledge base includes interface features and data features of each typical interaction scenario category;
[0034] The preliminary identification unit scans and identifies the target system based on the rule engine and the identification model to obtain the initial interaction interface and potential sensitive data. The rule engine is formed by an initial rule set established by a knowledge base including multiple typical interaction scenario categories, and the identification model is obtained by machine learning using a number of real interface and potential sensitive data samples in various business scenarios.
[0035] The classification analysis unit performs similarity analysis on the initial interaction interface and potential sensitive data with the knowledge base, and classifies them into corresponding typical interaction scenarios according to the similarity scores;
[0036] The final identification unit uses cluster analysis to analyze the classification results and confirm the target interaction interface and sensitive data.
[0037] The following are further optimizations and / or improvements to the above technical solutions:
[0038] The above-mentioned preliminary identification unit includes:
[0039] A first scanning module constructs an initial rule set based on the knowledge base, and uses the initial rule set to perform a preliminary scan on the target system to obtain a first candidate interface and a potential sensitive data set, wherein the initial rule set includes rules for identifying common interface types and potential sensitive data patterns;
[0040] The second scanning module uses the recognition model to perform a comprehensive scan on the target system to obtain a second candidate interface and a set of potential sensitive data, wherein the recognition model is obtained by machine learning using a number of real interfaces and potential sensitive data samples in various business scenarios;
[0041] A fusion module, which fuses the first candidate interface and the potential sensitive data set with the second candidate interface and the potential sensitive data set, and generates a corresponding comprehensive result by using a weighted voting mechanism;
[0042] The scoring module performs confidence scoring on the comprehensive results and divides the candidate interfaces and potential sensitive data into a preliminary identification result set and a result set to be verified based on the scoring results;
[0043] The iteration module updates the initial rule set and the recognition model using the result set to be verified, and re-scans and recognizes the target system using the updated initial rule set and the recognition model;
[0044] The output module loops the above steps until the confidence scores of all results to be verified exceed the preset threshold or reach the maximum number of iterations, outputs the preliminary recognition result set, and obtains the final preliminary recognition interaction interface and potential sensitive data.
[0045] The beneficial effects of the present invention include:
[0046] (1) By building a knowledge base based on business needs and application scenarios, it is possible to quickly identify and classify interactive interfaces and sensitive data, greatly improving recognition efficiency.
[0047] (2) Combining the rule engine and machine learning algorithm for scanning can not only identify known interfaces and data features, but also discover unknown potential sensitive data through model training, thereby improving the accuracy and comprehensiveness of identification.
[0048] (3) Using data mining technology to analyze the matching and classification results can adapt to the dynamically changing system environment, update the recognition rules in a timely manner, and ensure the real-time and reliability of the recognition results.
[0049] (4) Through automated algorithms and data mining technology, the workload of manual rule writing and maintenance is reduced, and the system maintenance cost and complexity are reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Attached Figure 1 It is a schematic diagram of the method flow of the present invention.
[0051] Attached Figure 2 Schematic diagram of the construction process of the knowledge base in the present invention.
[0052] Attached Figure 3 It is a schematic diagram of the process of scanning and identifying the target system in the present invention.
[0053] Attached Figure 4 It is a schematic diagram of the device structure of the present invention. DETAILED DESCRIPTION
[0054] The present invention is not limited by the following embodiments, and specific implementation methods can be determined based on the technical solution of the present invention and actual conditions.
[0055] The present invention will be further described below in conjunction with embodiments and drawings:
[0056] Embodiment 1: As attached Figure 1 As shown, the embodiment of the present invention discloses a method for automatically discovering interactive interfaces and potential sensitive data in a typical scenario, including:
[0057] Step S110, building a knowledge base including multiple typical interaction scenario categories based on preset business requirements and application scenarios, wherein the knowledge base includes interface features and data features of each typical interaction scenario category;
[0058] Step S120, scanning and identifying the target system based on a rule engine and a recognition model to obtain preliminary interaction interfaces and potential sensitive data, wherein the rule engine is formed by an initial rule set established by a knowledge base including multiple typical interaction scenario categories, and the recognition model is obtained by machine learning using a number of real interface and potential sensitive data samples in various business scenarios;
[0059] Step S130, performing similarity analysis on the initially identified interaction interface and the potential sensitive data with the knowledge base, and classifying them into corresponding typical interaction scenarios according to the similarity scores;
[0060] Step S140: Analyze the classification results using cluster analysis to confirm the target interactive interface and sensitive data.
[0061] The present invention discloses a method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios. Through a knowledge base including multiple typical interactive scenario categories, the interactive interfaces and potential sensitive data can be quickly identified and classified, greatly improving the recognition efficiency; combining a rule engine and a machine learning algorithm to perform two scans of the target system, not only can known interfaces and data features be identified, but also unknown potential sensitive data can be discovered through model training, thereby improving the accuracy and comprehensiveness of recognition; cluster analysis is used to analyze the classification results, which can adapt to the dynamically changing system environment, timely update the recognition rules, and ensure the real-time and reliability of the recognition results.
[0062] Embodiment 2: The embodiment of the present invention discloses a method for automatically discovering interactive interfaces and potential sensitive data in a typical scenario, including:
[0063] Step S210: construct a knowledge base including a plurality of typical interaction scenario categories based on preset business requirements and application scenarios, wherein the knowledge base includes interface features and data features of each typical interaction scenario category.
[0064] As attached Figure 2 As shown, the above step S210 specifically includes:
[0065] Step S211, collecting historical business log data of each typical interaction scenario category;
[0066] In this embodiment, business requirements and application scenarios are preset. Business requirements include types of potentially sensitive data that need to be protected in the system, common business operation processes, and interaction methods of different business modules; application scenarios cover the operating modes of the system in various business environments, such as user registration, login, data query, transaction processing, etc.
[0067] In this embodiment, the historical business log data collected can be obtained from system logs, API call records and database logs. The historical business log data provides detailed records of various interactions in the actual operation of the system, including user requests and system responses.
[0068] Step S212: extracting interface features and data features of each typical interaction scenario category based on historical business log data, wherein interface features include API path, request method, request parameter format, and response data format; and data features include potential sensitive data types, data field names, data formats, and data lengths;
[0069] In this embodiment, the interactive interface information includes API path, request method (such as GET, POST), request parameter format (such as JSON, XML) and response data format (such as JSON, XML); data field information includes potentially sensitive data types (such as ID number, credit card number), data field names (such as "ID number" field, "credit card number" field), data format (such as string, number) and data length.
[0070] Furthermore, based on historical business log data, after extracting the interface features and data features of each typical interaction scenario category, data cleaning and standardization can be performed to ensure a unified format and eliminate noise and redundant information.
[0071] Step S213: Optimize the interface features and data features of each typical interaction scenario category using the expert database, and form a knowledge base including multiple typical interaction scenario categories based on the optimized interface features and data features of each typical interaction scenario category.
[0072] In this embodiment, in order to further improve the accuracy and practicality of the knowledge base, it is necessary to use the expert database to optimize the interface features and data features of each typical interaction scenario category, specifically including:
[0073] (1) Experts review the extracted interface features and data features;
[0074] (2) Reanalyze and discuss controversial features;
[0075] (3) Through repeated iterations, the feature description in the knowledge base is finally confirmed and optimized.
[0076] The expert group then re-analyzed and discussed the controversial features, and confirmed and corrected the errors and omissions in the preliminary feature information by comparing multiple data sources and actual business needs. Based on the expert review, the extracted features were repeatedly optimized. In each round of iteration, the expert group re-evaluated and confirmed the revised features to ensure that the feature description was highly consistent with the actual business scenario. In this process, the expert group can also introduce more actual business data and scenarios to further enrich and improve the knowledge base content.
[0077] Step S220, scan and identify the target system based on the rule engine and the recognition model to obtain preliminary interactive interfaces and potential sensitive data, wherein the rule engine is formed by an initial rule set established by a knowledge base including multiple typical interactive scenario categories, and the recognition model is obtained by machine learning using a number of real interface and potential sensitive data samples in various business scenarios.
[0078] As attached Figure 3 As shown, the above step S220 specifically includes:
[0079] Step S221, constructing an initial rule set based on the knowledge base, using the initial rule set to perform a preliminary scan on the target system to obtain a first candidate interface and a set of potentially sensitive data, wherein the initial rule set includes rules for identifying common interface types and potentially sensitive data patterns.
[0080] In this embodiment, the rules in the initial rule set are used to identify common interface types and potentially sensitive data patterns. To build the initial rule set, a series of rules need to be defined based on business requirements and system characteristics. These rules may include regular expression matching, keyword recognition, data format verification, etc. For example, for identifying API paths, rules can be defined to match paths in the format of " / api / *"; for potentially sensitive data fields, rules can be defined to match data in specific formats such as ID numbers and credit card numbers.
[0081] In this embodiment, the target system is initially scanned using the constructed initial rule set. This scanning process traverses the logs, API requests and response data in the target system, and identifies the first candidate interface and potential sensitive data set according to the predefined rule set. The target system data is then checked line by line through the rule engine, and the matching interfaces and potential sensitive data are recorded.
[0082] Step S222, using the recognition model to comprehensively scan the target system to obtain a second candidate interface and a set of potentially sensitive data, wherein the recognition model is obtained by machine learning using a number of real interfaces and potentially sensitive data samples in various business scenarios.
[0083] The above recognition model is obtained by machine learning using a number of real interfaces and potentially sensitive data samples in various business scenarios, wherein the machine learning model used includes a feature extraction module, a primary classifier module, a sequence analysis module, an adaptive weight adjustment module, and a final discrimination module;
[0084] A feature extraction module, applying a feature extraction algorithm to extract feature vectors from the original data stream of the target system and generate a feature vector set; wherein the feature vectors include data packet length, request frequency, data format and field type;
[0085] Specifically, the feature extraction algorithm needs to analyze multiple attributes of the data packet, such as packet length, request frequency, data format, and field type. The packet length can reflect the scale of data transmission, and the request frequency can reveal the system workload and user behavior patterns. The data format and field type provide structural information about the data content. For example, if the data format is JSON, the field type is string and integer, etc., the feature extraction module will extract this information and convert it into a feature vector, forming a feature vector set containing all relevant information.
[0086] The primary classifier module processes the feature vector set through a multi-layer perceptron to obtain a preliminary classification result; wherein the preliminary classification result includes a list of potential interactive interfaces and potential sensitive data;
[0087] Specifically, the primary classifier module processes the feature vector set through a multi-layer perceptron (MLP). The multi-layer perceptron is a classic neural network structure suitable for processing high-dimensional feature vectors. The primary classifier generates preliminary classification results by performing nonlinear transformation and learning on the feature vectors. These results include a list of potential interactive interfaces and potential sensitive data. Specifically, the MLP processes the input feature vector layer by layer through multiple hidden layers, and finally gives the probability of each feature vector belonging to a certain interactive interface or potential sensitive data at the output layer.
[0088] A sequence analysis module, using a recursive neural network to analyze the time series pattern of the preliminary classification results to obtain a time series analysis result;
[0089] Specifically, the sequence analysis module uses recurrent neural networks (RNNs), especially long short-term memory networks (LSTMs), to analyze the time series patterns of preliminary classification results. RNNs are good at processing sequence data and can capture the temporal dependencies between data. By analyzing the temporal change patterns of preliminary classification results, the sequence analysis module can obtain more accurate time series analysis results. This process involves time series modeling of preliminary classification results, identifying the temporal correlations and sequential patterns between data packets, and thus more accurately identifying potential interaction interfaces and potential sensitive data.
[0090] The adaptive weight adjustment module dynamically adjusts the weights of the primary classifier and the sequence analysis module based on historical accuracy and confidence scores; based on the adjusted weights, the weighted comprehensive results are obtained based on the preliminary classification results and the time series analysis results;
[0091] Specifically, the adaptive weight adjustment module dynamically adjusts the weights of the primary classifier and sequence analysis module based on historical accuracy and confidence scores. The key is to continuously optimize the model using the feedback mechanism. According to the performance of the primary classifier and sequence analysis module in historical data, the adaptive weight adjustment module sets a dynamic adjustment mechanism to assign different weight ratios to the two to achieve the best overall effect. The adjusted weights make the weighted comprehensive results more accurate and reliable. Specifically, the weight adjustment can dynamically adjust the influence of each module in each prediction by calculating the accuracy of each module in the past predictions.
[0092] The final discrimination module, based on the weighted comprehensive results, identifies the target interactive interface and potential sensitive data through the support vector machine.
[0093] Specifically, support vector machines (SVMs) are used to identify target interactive interfaces and potentially sensitive data. SVM is a powerful classification algorithm, especially suitable for high-dimensional data classification tasks. By processing the weighted synthesis results, SVM can find the optimal classification boundary in the feature space and accurately identify which interactive interfaces and data fields are sensitive. The final discrimination module uses the efficient classification ability of SVM to perform the final classification of the weighted synthesis results and output a list of target interactive interfaces and potentially sensitive data.
[0094] In the above embodiment, the reference implementation code of the automatic discovery algorithm based on the combination of rule engine and machine learning is as follows:
[0095]
[0096]
[0097]
[0098]
[0099]
[0100] This code implements the steps of the automatic discovery algorithm based on the combination of rule engine and machine learning in detail. It builds an initial rule set for preliminary scanning, uses a machine learning model for comprehensive scanning, combines sequence analysis and adaptive weight adjustment, and finally uses a support vector machine to judge the results.
[0101] Step S223: The first candidate interface and the potential sensitive data set are merged with the second candidate interface and the potential sensitive data set, and a weighted voting mechanism is used to generate a corresponding comprehensive result.
[0102] In this embodiment, the weights of the rule engine and the recognition model are dynamically adjusted according to their historical accuracy. The weighted voting mechanism refers to assigning weights according to the confidence of each recognition result, and then combining the results from the two sources to generate a more accurate recognition result. In specific implementation, the confidence score can be calculated for each recognition result, and the weights of the rule engine and the machine learning model can be adjusted according to the historical accuracy, and finally the result with the highest comprehensive score is output.
[0103] Step S224, performing confidence scoring on the comprehensive results, and dividing the candidate interfaces and potential sensitive data into a preliminary identification result set and a result set to be verified based on the scoring results.
[0104] In this embodiment, the fused comprehensive results are scored for confidence, and the candidate interfaces and potential sensitive data corresponding to the high-confidence comprehensive results are directly included in the preliminary identification result set, while the candidate interfaces and potential sensitive data corresponding to the low-confidence comprehensive results are included in the result set to be verified.
[0105] Step S225, using the result set to be verified to update the initial rule set and the recognition model, and using the updated initial rule set and the recognition model to re-scan and recognize the target system;
[0106] In this embodiment, the initial rule set and the recognition model are updated by using the result set to be verified, and the target system is scanned and recognized again by using the updated initial rule set and the recognition model, including:
[0107] (1) Generate test cases based on the interface features and data features corresponding to each result to be verified in the result set to be verified; the test cases here are intended to further verify and confirm the accuracy of the low-confidence results. Generate test cases to verify the system's response to these results by simulating actual API requests and responses, or constructing specific data inputs.
[0108] (2) Execute the test case on the target system to obtain the execution results and system response. Here, the execution of the test case simulates actual operations, checks how the system processes these inputs, and collects the system's response data and log information. The collected execution results and system responses provide basic data for the next step of analysis.
[0109] (3) Analyze the execution results of the test cases and update the parameters of the recognition model and the rules of the rule engine. Here, the analysis of the execution results can identify the patterns and anomalies in the system by comparing the expected results with the actual responses, and adjust the parameters of the machine learning model and the rules of the rule engine accordingly to make them better adapt to the actual system environment.
[0110] Step S226, loop the above steps until the confidence scores of all the results to be verified exceed the preset threshold or reach the maximum number of iterations, output the preliminary identification result set, and obtain the final preliminary identification interaction interface and potential sensitive data. Based on the maximum number of iterations, it is possible to obtain multiple rounds of optimization and contain all high-confidence interaction interfaces and potential sensitive data in the target system, thereby providing comprehensive and accurate identification information, which provides an important basis for subsequent security management and data protection.
[0111] Step S230 , performing similarity analysis on the initially identified interaction interface and the potential sensitive data with the knowledge base, and classifying them into corresponding typical interaction scenarios according to the similarity scores.
[0112] The purpose of the above step S230 is to match and classify the initial interaction interface and the potential sensitive data, which is accomplished by combining similarity calculation and classification algorithms. Specifically, it includes:
[0113] (1) Standardize the initial interaction interface and potential sensitive data. This step ensures the consistency of the data format for subsequent matching and classification operations. For example, the interface path may contain dynamic parameters or variables, which are converted into a standardized path format through regularization; similarly, for data fields, ensure that their data types and lengths are consistent with the descriptions in the knowledge base.
[0114] (2) Perform feature vectorization on the standardized initial interaction interfaces and potential sensitive data. This means converting each interaction interface and data field into a feature vector, where each dimension of the vector represents a specific feature (such as interface path, request method, parameter type, etc.). This vectorization process makes subsequent similarity calculations more efficient and accurate.
[0115] (3) Use a similarity algorithm to calculate the similarity between the initial interaction interface and potential sensitive data and each typical interaction scenario category in the knowledge base. Commonly used similarity algorithms include cosine similarity, Jaccard similarity, etc. Specifically, a dot product operation can be performed on the feature vector of each recognition result and the feature vector of the scenario category in the knowledge base to calculate the cosine similarity, thereby quantifying the degree of similarity between the two.
[0116] Among them, the cosine similarity algorithm is a method to measure the similarity between two vectors. It determines their similarity by calculating the cosine value of the angle between the two vectors. In specific implementation, the dot product operation can be performed on the initially identified feature vector and the feature vector of each typical interaction scenario category in the knowledge base, and then divided by the product of the modulus lengths of the two vectors to obtain the cosine similarity value. The closer the similarity value is to 1, the more similar the two vectors are.
[0117] (4) Based on the similarity score, the initial interaction interface and potential sensitive data are classified into the most matching typical interaction scenario category. That is, a similarity threshold is set, and only matching results with similarity scores exceeding the threshold are accepted, while results below the threshold require further manual verification or more automated analysis.
[0118] The above classification process is to assign each recognition result to the scene category with the highest similarity score. For example, if a preliminarily recognized API path has the highest similarity score with a scene category, the API path is classified into the scene category. Similarly, if a sensitive data field has the highest similarity score with a scene category, the data field is classified into the scene category.
[0119] Step S240: Analyze the classification results using cluster analysis to confirm the target interactive interface and sensitive data.
[0120] The above steps specifically include:
[0121] (1) Use K-means clustering algorithm to perform cluster analysis on the classification results and obtain multiple cluster groups;
[0122] Before this step, the classification results can be preprocessed to generate feature vectors, including standardizing the data format, cleaning data noise, and converting data types to ensure that all data conform to a unified format to facilitate the application of subsequent analysis tools.
[0123] The standardization process involves subtracting the mean of the feature from the original feature value and then dividing it by the standard deviation of the feature to ensure that different features have the same dimension, as follows:
[0124] Each feature is normalized to zero mean according to the following formula:
[0125]
[0126] Among them, x is the original eigenvalue; μ is the mean of the feature; σ is the standard deviation of the feature; and z is the result after zero mean standardization.
[0127] The goal of cluster analysis in this step is to group similar interactive interfaces and sensitive data together, thereby identifying common patterns and anomalies.
[0128] Specifically include:
[0129] Set the K value to indicate the number of groups you want to divide into, and use the K-means++ algorithm to select the initial center point. This step improves the efficiency of selecting the initial center point by selecting the point farthest from the currently selected center point as the next center point.
[0130] After the initial center point is selected, K-means clustering is performed and the center point is iteratively updated until the position of the center point no longer changes significantly. In each iteration, the distance from each data point to each center point is calculated, and the data point is assigned to the group to which the nearest center point belongs. Then the center point position of each group is recalculated. This process continues until the position of the center point converges or the change is less than the preset threshold.
[0131] The specific steps include:
[0132] a: Randomly select a point from the feature vector set as the first initial center point;
[0133] b: For each eigenvector x i , according to the following formula 2, calculate the distance Dx between it and the nearest center point among all current center points i :
[0134]
[0135] Among them, C is the currently selected center point set;
[0136] c: Select distance Dx i The probability and Dx i 2 The point proportional to the initial center point is selected as the next initial center point, and the process is repeated until K initial center points are selected.
[0137] d: Use the selected initial center point to perform traditional K-means clustering and iteratively update the center point until the center point no longer changes or the change is less than the preset threshold;
[0138] In the clustering process, according to the following formula 3, an improved method based on the fuzzy C-means algorithm is used in combination to dynamically adjust the membership of the data points in each iteration:
[0139]
[0140] Among them, u ij Represents data point x i For cluster center c j The membership degree of ; m is the fuzzy index, and its value is greater than 1;
[0141] Among them, the cluster center c j Calculate according to the following formula 4:
[0142]
[0143] Where N is the total number of data points;
[0144] Perform feature analysis on the center point of each cluster to identify the common features of the clusters, where the center point feature analysis includes calculating the weighted mean μ1 and variance σ1 of each center point feature according to the following formulas 5 and 6:
[0145]
[0146] Among them, w i is the weight of the feature vector; x i is the eigenvalue; n is the number of eigenvectors.
[0147] The following is the reference implementation code for clustering analysis of matching and classification results:
[0148]
[0149]
[0150]
[0151]
[0152]
[0153] (2) Perform feature analysis on each cluster group to identify and mark the target interaction interface and sensitive data.
[0154] The main task of feature analysis is to identify common sensitive data patterns in each group. The specific steps include:
[0155] (1) Count the frequency of interactive interfaces and sensitive data fields in each cluster group. For example, count the number of times a certain API path appears in the group, or count the frequency of certain sensitive data (such as ID card number, credit card number, etc.). Through frequency statistics, we can identify features that appear frequently. These high-frequency features are usually common and representative features within the group.
[0156] Here, you can also build a dictionary or hash table to store the frequency statistics results. For each interactive interface path, use it as the key and store the number of occurrences as the value. Similarly, for sensitive data fields (such as ID card number, credit card number, etc.), frequency statistics are also performed separately. Each time a data point is processed, the corresponding frequency statistics result is updated once.
[0157] (2) After completing the frequency statistics, analyze the features that appear frequently and identify common sensitive data patterns. For example, if an API path appears frequently in a group and is accompanied by specific request parameters and response data formats, it can be identified as a common sensitive data pattern. Similarly, for frequently appearing sensitive data fields, their common usage patterns can be identified by analyzing their context and format.
[0158] In order to identify these high-frequency features, a frequency threshold can be set. Only features that appear more than the threshold are considered high-frequency features. This method helps to filter out noise data and low-frequency features and focus on analyzing representative features.
[0159] During the analysis process, the contextual information of the features can be further combined. For example, for frequently appearing API paths, their corresponding request methods, parameter formats, etc. can be analyzed to identify their common usage patterns. Similarly, for frequently appearing sensitive data fields, their specific locations and formats can be analyzed to identify their common storage and transmission methods.
[0160] Based on the results of the above feature analysis, the target interactive interfaces and sensitive data are identified and marked. By comparing the frequently occurring features and common sensitive data patterns with predefined security standards and specifications, it is possible to identify which interactive interfaces and data fields belong to the target interactive interfaces and sensitive data, and mark them. These marks can be used for subsequent security monitoring and management to ensure that the system can identify and protect sensitive data in a timely manner.
[0161] Embodiment 3: As attached Figure 4 As shown, the embodiment of the present invention discloses a method for automatically discovering interactive interfaces and potential sensitive data in a typical scenario, including:
[0162] A knowledge base construction unit, based on preset business requirements and application scenarios, constructs a knowledge base including multiple typical interaction scenario categories, wherein the knowledge base includes interface features and data features of each typical interaction scenario category;
[0163] The preliminary identification unit scans and identifies the target system based on the rule engine and the identification model to obtain the initial interaction interface and potential sensitive data. The rule engine is formed by an initial rule set established by a knowledge base including multiple typical interaction scenario categories, and the identification model is obtained by machine learning using a number of real interface and potential sensitive data samples in various business scenarios.
[0164] The classification analysis unit performs similarity analysis on the initial interaction interface and potential sensitive data with the knowledge base, and classifies them into corresponding typical interaction scenarios according to the similarity scores;
[0165] The final identification unit uses cluster analysis to analyze the classification results and confirm the target interaction interface and sensitive data.
[0166] Among them, the preliminary identification unit includes:
[0167] A first scanning module constructs an initial rule set based on the knowledge base, and uses the initial rule set to perform a preliminary scan on the target system to obtain a first candidate interface and a potential sensitive data set, wherein the initial rule set includes rules for identifying common interface types and potential sensitive data patterns;
[0168] The second scanning module uses the recognition model to perform a comprehensive scan on the target system to obtain a second candidate interface and a set of potential sensitive data, wherein the recognition model is obtained by machine learning using a number of real interfaces and potential sensitive data samples in various business scenarios;
[0169] A fusion module, which fuses the first candidate interface and the potential sensitive data set with the second candidate interface and the potential sensitive data set, and generates a corresponding comprehensive result by using a weighted voting mechanism;
[0170] The scoring module performs confidence scoring on the comprehensive results and divides the candidate interfaces and potential sensitive data into a preliminary identification result set and a result set to be verified based on the scoring results;
[0171] The iteration module updates the initial rule set and the recognition model using the result set to be verified, and re-scans and recognizes the target system using the updated initial rule set and the recognition model;
[0172] The output module loops the above steps until the confidence scores of all results to be verified exceed the preset threshold or reach the maximum number of iterations, outputs the preliminary recognition result set, and obtains the final preliminary recognition interaction interface and potential sensitive data.
[0173] Embodiment 4: The embodiment of the present invention discloses a storage medium, on which is stored a computer program that can be read by a computer, and the computer program is configured to execute a method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios when running.
[0174] The above storage medium may include, but is not limited to: a USB flash drive, a read-only memory, a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.
[0175] Embodiment 5: The embodiment of the present invention discloses an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement a method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios.
[0176] The above processor may be a central processing unit (CPU), a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. It may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The memory may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory, a mobile hard disk, a magnetic disk or an optical disk.
[0177] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0178] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0179] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0180] The above technical features constitute the best embodiment of the present invention, which has strong adaptability and best implementation effect. Non-essential technical features can be added or reduced according to actual needs to meet the requirements of different situations.
Claims
1. A method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios, characterized in that: include: Based on preset business requirements and application scenarios, a knowledge base including multiple typical interaction scenario categories is constructed, wherein the knowledge base includes interface features and data features of each typical interaction scenario category; Scan and identify the target system based on the rule engine and recognition model to obtain the initial interactive interface and potential sensitive data. The rule engine is formed by an initial rule set established by a knowledge base including multiple typical interactive scenario categories, and the recognition model is obtained by machine learning using a number of real interface and potential sensitive data samples in various business scenarios; Perform similarity analysis on the initial interaction interface and potential sensitive data with the knowledge base, and classify them into corresponding typical interaction scenarios according to the similarity scores; Cluster analysis is used to analyze the classification results and confirm the target interactive interface and sensitive data.
2. The method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios according to claim 1 is characterized in that: Based on the preset business requirements and application scenarios, a knowledge base including multiple typical interaction scenario categories is constructed, including: Collect historical business log data for each typical interaction scenario category; Based on historical business log data, extract interface features and data features of each typical interaction scenario category. Interface features include API path, request method, request parameter format, and response data format. Data features include potential sensitive data types, data field names, data formats, and data lengths. The interface features and data features of each typical interaction scenario category are optimized using the expert database, and a knowledge base including multiple typical interaction scenario categories is formed based on the optimized interface features and data features of each typical interaction scenario category.
3. The method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios according to claim 1 or 2, characterized in that: The target system is scanned and identified based on the rule engine and the identification model to obtain the initial interaction interface and potential sensitive data, including: Building an initial rule set based on the knowledge base, and using the initial rule set to perform a preliminary scan on the target system to obtain a first candidate interface and a set of potential sensitive data, wherein the initial rule set includes rules for identifying common interface types and potential sensitive data patterns; The target system is fully scanned using the recognition model to obtain the second candidate interface and potential sensitive data set, wherein the recognition model is obtained by machine learning using a number of real interfaces and potential sensitive data samples in various business scenarios; The first candidate interface and the potential sensitive data set are merged with the second candidate interface and the potential sensitive data set, and a weighted voting mechanism is used to generate a corresponding comprehensive result; Confidence scoring is performed on the comprehensive results, and the candidate interfaces and potential sensitive data are divided into a preliminary identification result set and a result set to be verified based on the scoring results; Using the result set to be verified, the initial rule set and the recognition model are updated, and the target system is scanned and recognized again using the updated initial rule set and the recognition model; The above steps are repeated until the confidence scores of all results to be verified exceed the preset threshold or the maximum number of iterations is reached, and the preliminary recognition result set is output to obtain the final preliminary recognition interaction interface and potential sensitive data.
4. The method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios according to claim 3 is characterized in that: The method of updating the initial rule set and the recognition model by using the result set to be verified, and re-scanning and recognizing the target system by using the updated initial rule set and the recognition model, includes: Generate test cases based on interface features and data features corresponding to each result to be verified in the result set to be verified; Execute test cases on the target system and obtain execution results and system responses; Analyze the execution results of test cases and update the parameters of the recognition model and the rules of the rule engine.
5. The method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios according to any one of claims 1 to 4, characterized in that: The similarity analysis of the initial interaction interface and potential sensitive data with the knowledge base is performed, and they are classified into corresponding typical interaction scenarios according to the similarity scores, including: The cosine similarity algorithm is used to calculate the similarity between the initial interaction interface and potential sensitive data and multiple typical interaction scenario categories in the knowledge base; Based on the similarity scores, the initial interaction interfaces and potential sensitive data are classified into the most matching typical interaction scenario categories.
6. The method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios according to any one of claims 1 to 5, characterized in that: The cluster analysis is used to analyze the classification results to confirm the target interactive interface and sensitive data, including: The K-means clustering algorithm is used to perform cluster analysis on the classification results to obtain multiple cluster groups; Perform feature analysis on each cluster group to identify and mark the target interaction interfaces and sensitive data.
7. A method for automatically discovering interactive interfaces and potential sensitive data in a typical scenario using the method as claimed in any one of claims 1 to 6, characterized in that: include: A knowledge base construction unit, based on preset business requirements and application scenarios, constructs a knowledge base including multiple typical interaction scenario categories, wherein the knowledge base includes interface features and data features of each typical interaction scenario category; The preliminary identification unit scans and identifies the target system based on the rule engine and the identification model to obtain the initial interaction interface and potential sensitive data. The rule engine is formed by an initial rule set established by a knowledge base including multiple typical interaction scenario categories, and the identification model is obtained by machine learning using a number of real interface and potential sensitive data samples in various business scenarios. The classification analysis unit performs similarity analysis on the initial interaction interface and potential sensitive data with the knowledge base, and classifies them into corresponding typical interaction scenarios according to the similarity scores; The final identification unit uses cluster analysis to analyze the classification results and confirm the target interaction interface and sensitive data.
8. The automatic discovery device for interactive interfaces and potential sensitive data in typical scenarios according to claim 7, characterized in that: The preliminary identification unit comprises: A first scanning module constructs an initial rule set based on the knowledge base, and uses the initial rule set to perform a preliminary scan on the target system to obtain a first candidate interface and a potential sensitive data set, wherein the initial rule set includes rules for identifying common interface types and potential sensitive data patterns; The second scanning module uses the recognition model to perform a comprehensive scan on the target system to obtain a second candidate interface and a set of potential sensitive data, wherein the recognition model is obtained by machine learning using a number of real interfaces and potential sensitive data samples in various business scenarios; A fusion module, which fuses the first candidate interface and the potential sensitive data set with the second candidate interface and the potential sensitive data set, and generates a corresponding comprehensive result by using a weighted voting mechanism; The scoring module performs confidence scoring on the comprehensive results and divides the candidate interfaces and potential sensitive data into a preliminary identification result set and a result set to be verified based on the scoring results; The iteration module updates the initial rule set and the recognition model using the result set to be verified, and re-scans and recognizes the target system using the updated initial rule set and the recognition model; The output module loops the above steps until the confidence scores of all results to be verified exceed the preset threshold or reach the maximum number of iterations, outputs the preliminary recognition result set, and obtains the final preliminary recognition interaction interface and potential sensitive data.
9. A storage medium, characterized in that: The storage medium stores a computer program that can be read by a computer, and the computer program is configured to execute the method for automatically discovering interactive interfaces and potential sensitive data in a typical scenario as described in any one of claims 1 to 6 when running.
10. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the method for automatically discovering interactive interfaces and potential sensitive data in typical scenarios as described in any one of claims 1 to 6.
Citation Information
Cited By
Intelligent palm rattan identification and classification method and system
CN119992194A
Intelligent rattan palm identification and classification method and system
CN119992194B
Processing method and system of data element credible circulation technology
CN120147012A
Automatic identification and desensitization processing method and system for sensitive data in communication log
CN122113164A
