Black box computing environment-oriented data risk detection method and system

Abstract

The invention belongs to the technical field of data risk detection, and provides a black-box computing environment-oriented data risk detection method and system, and the method comprises the steps: intercepting a calculation result of a black-box computing environment, carrying out the analysis of the calculation result, and extracting a data size feature, a statistical distribution feature, a semantic sensitive feature and a model specific feature; combining the extracted features into a feature vector, performing evaluation by using a pre-trained risk evaluation model to obtain a quantitative risk score, and determining a risk level of a calculation result according to a set score threshold interval; triggering a corresponding disposal strategy according to the risk level, and blocking a calculation result of which the risk level exceeds a set level; and performing encryption and evidence storage on the calculation result allowed to be output. According to the method, the potential security risk in the output data of the black-box computing environment can be automatically, accurately and efficiently identified and handled.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data risk detection, and particularly relates to a data risk detection method and system for a black-box computing environment. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Privacy computing technology has become a key infrastructure for the circulation of data elements. Technologies such as Data Sandbox and Trusted Execution Environment (TEE) ensure that raw data does not leave the domain during the computing process by building a "black box" environment, effectively solving the risk of raw data leakage.

[0004] However, there is a universally overlooked security blind spot in the existing technology: the security of the black-box computing environment itself cannot guarantee the security of its output results (such as statistical reports, machine learning models, query results, etc.) in 100%. Attackers may use advanced methods such as differential attacks, membership inference attacks, and model reverse attacks to infer sensitive individual information from seemingly harmless output data. For example, an overly accurate aggregate statistical value may leak individual characteristics, and an over-fitted machine learning model may remember the privacy details in the training data.

[0005] Currently, the industry lacks a risk review scheme for the output results of the black-box environment, and the existing static rule filtering method is generally based on simple keyword and regular expression matching, which cannot cope with complex and variable privacy leakage patterns, and has high false positive and false negative rates. The completely manual review method is manually reviewed by data security experts, which is costly, inefficient, difficult to scale, and heavily dependent on the personal experience of experts, lacking consistency and measurability. SUMMARY

[0006] To solve the above problems, the present application proposes a data risk detection method and system for a black-box computing environment, which can automatically, accurately and efficiently identify and dispose of potential security risks in the output data of the black-box computing environment.

[0007] According to some embodiments, the present application adopts the following technical solutions: A data risk detection method for a black-box computing environment, comprising the following steps: intercepting the computing results of the black-box computing environment, analyzing them, and extracting some of the data volume features, statistical distribution features, semantic sensitive features and model-specific features; The extracted features are combined into a feature vector, the pre-trained risk assessment model is used for evaluation, a quantitative risk score is obtained, and according to a set score threshold interval, the risk level of the calculation result is determined; According to the risk level, the corresponding treatment strategy is triggered, and the calculation result whose risk level exceeds the set level is blocked; The calculation result allowed to be output is encrypted and stored.

[0008] As an optional implementation, it further includes the following steps: periodically randomly sampling samples from the calculation results allowed to be output according to a preset proportion, reviewing the samples, and if there is an inaccurate risk level in the review, using the calculation result and the reviewed risk level as new training samples to feed back to the risk assessment model for retraining.

[0009] As an optional implementation, the data volume feature includes the number of data records, the total size of data, and the number of fields / attributes, which are used to judge the granularity of data output. The data volume exceeding the set value represents a possible over-detailed information leakage.

[0010] As an optional implementation, the statistical distribution feature includes statistical values of numerical fields calculated for structured data, and is compared with the overall distribution of the calculation result. The calculation result deviating from the overall distribution by more than a set range represents a possible individual sensitive information.

[0011] As an optional implementation, the semantic sensitive feature includes the result of sensitive information identification of text type output content by using natural language processing technology. Through a pre-defined sensitive information pattern library, the pattern library stores some regular expressions or keywords of identity card number, mobile phone number, bank card number, name, and address. It is detected whether the calculation result contains explicit sensitive data.

[0012] As an optional implementation, the model specificity feature is obtained by extracting the complexity index and performance index of the model. The model complexity or performance exceeding the predetermined value represents a possible overfitting, and there is a risk of memorizing the training data privacy.

[0013] As an optional implementation, the risk assessment model is a machine learning model, which is trained by supervised learning using labeled historical data.

[0014] As an optional implementation, according to the set score threshold interval, the process of determining the risk level of the calculation result includes: if the score is lower than the threshold T1, it is determined to be basically safe, and the calculation result data is allowed to be directly output to the requester; If the score is between threshold T1 and T2, it is determined that there is a suspicious risk, an artificial review task is created, and a designated data security administrator is pushed through a message notification or a task list. If the score is higher than the threshold T2, the output of the corresponding calculation result data is automatically and in real time blocked, and a high-level security alarm is immediately generated to inform the security team to handle the emergency.

[0015] As an alternative embodiment, the process of encrypting and storing the calculation results allowed to be output includes: generating a unique digital fingerprint of the calculation result data through a digital digest technology; binding the digital fingerprint with the metadata of the current calculation task to form a complete audit log; After encrypting the audit log, it is stored in the blockchain for storage and provides traceability.

[0016] Further, the metadata of the calculation task includes some of the task ID, timestamp, data requester identity, black box environment identifier, risk assessment score, disposal action and reviewer ID.

[0017] A data risk detection system for a black box computing environment includes: The feature extraction engine is used to intercept the calculation results of the black box computing environment, analyze them, and extract some of the data volume features, statistical distribution features, semantic sensitive features and model specific features; The AI risk assessment engine is used to combine the extracted features into a feature vector, use a pre-trained risk assessment model for evaluation, obtain a quantitative risk score, and determine the risk level of the calculation result according to the set score threshold interval; The risk disposal engine is used to trigger the corresponding disposal strategy according to the risk level, and block the calculation results whose risk level exceeds the set level; The encryption storage module is used to encrypt and store the calculation results allowed to be output.

[0018] Compared with the prior art, the beneficial effects of the present application are: The present application systematically solves the security risk of result data: it systematically proposes an end-to-end risk review scheme for the output data of the black box computing environment, filling the key security gap in the landing link of privacy computing, data sandbox and other black box computing environments.

[0019] The present application uses AI models to replace a large amount of repetitive manual labor, has high review efficiency, and can identify complex risk patterns that cannot be found by traditional rules.

[0020] The present application has continuous learning ability through the "sampling review-feedback optimization" closed loop, can cope with the evolving data security threats, realizes the upgrade from "static defense" to "dynamic immunity", and realizes dynamic self-adaptation.

[0021] The risk grading treatment mechanism of the application ensures that high-value artificial resources focus on the most critical minority suspicious data, and realizes the best balance between security control and business agility.

[0022] The application provides an unalterable audit trail for data external process through the encrypted evidence storage mechanism, and supports compliance audit.

[0023] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are used for detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings constituting a part of the specification of the application are used to provide further understanding of the application, the schematic embodiments of the application and the description thereof are used to explain the application, and do not constitute improper limitation on the application.

[0025] Figure 1 is a flowchart provided by an embodiment; Figure 2 is a system schematic diagram provided by an embodiment. DETAILED DESCRIPTION

[0026] The application will be further described below in combination with the drawings and embodiments.

[0027] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0028] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0029] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0030] Embodiment one A result data risk detection method for a black box computing environment, as shown in Figure 1 The specific steps are as follows: Step S100: output data interception and multi-dimensional feature extraction.

[0031] When the black-box computing environment (such as data sandbox, TEE) completes the calculation, its output data is first intercepted by the system of the present application. The system parses the output data and extracts one or more of the following dimensions of features: Data volume features: including the number of data records, the total size of data, the number of fields / attributes. Used to judge the granularity of data output, an abnormally large data volume may mean an excessive detailed information leakage.

[0032] Statistical distribution features: for structured data, calculate the statistical values of numerical fields (such as maximum value, minimum value, average value, median, standard deviation, variance, unique value count, etc.), and compare with the overall distribution of input data. The output that significantly deviates from the overall distribution may contain individual sensitive information.

[0033] Semantic sensitive features: use natural language processing techniques to identify sensitive information in text-type output content. Through a pre-defined sensitive information pattern library (such as regular expressions or keywords of ID number, mobile phone number, bank card number, name, address), detect whether the output contains explicit sensitive data.

[0034] Model-specific features (when the output is a machine learning model): extract the complexity indicators of the model (such as the depth of tree model, the number of layers and parameter size of neural network), performance indicators (such as accuracy, AUC value on test set). Excessive model complexity or performance may imply overfitting, which poses a risk of memorizing training data privacy.

[0035] Step S200: Automated risk preliminary screening based on AI risk assessment model.

[0036] Combine all the features extracted in step S100 into a feature vector, input into a pre-trained risk assessment model, which outputs a quantitative risk score (for example, a probability value between 0 and 1), and according to the set score threshold interval, divide the output data into different risk levels.

[0037] Risk assessment model: can be implemented using machine learning algorithms such as gradient boosting decision tree, random forest, deep learning, etc. The model is trained through a large number of labeled historical data (including "safe" and "at risk" output samples) through supervised learning, and can learn complex, non-linear risk patterns. Compared with static rules, AI models have stronger generalization ability and potential for detecting unknown threats.

[0038] Step S300: Hierarchical disposal and human-machine collaborative review based on risk level.

[0039] The system triggers the corresponding disposal strategy according to the risk level obtained in step S200: Low risk (score below threshold T1): Determine as basically safe, allow data to be directly output to the requester. The process goes to step S400.

[0040] Medium-high risk (score between thresholds T1 and T2): Determine as suspicious risk. The system automatically creates a manual review task, which is pushed to the designated data security administrator through message notification or task list. The review interface highlights the main risk features identified by the AI model to assist the administrator in making quick decisions. The administrator can give the final instruction of "approve release", "reject output" or "release after specifying desensitization strategy".

[0041] Extremely high risk (score above threshold T2): The system automatically and in real time blocks the output of this batch of data, and immediately generates a high-level security alert to notify the security team for emergency handling.

[0042] Step S400: Output data encryption storage.

[0043] In order to meet the requirements of data traceability and compliance audit, all the data finally allowed to be output (whether automatically released or released after manual review) are stored in the blockchain: Generate a unique digital fingerprint of this batch of output data through digital digest technology (for example, calculate the hash value using SHA-256 algorithm).

[0044] Bind the digital fingerprint with the metadata of this calculation task (including but not limited to: task ID, timestamp, data requester identity, black box environment identifier, risk assessment score, disposal action, reviewer ID) to form a complete audit log.

[0045] After encrypting the audit log, store it in the blockchain for storage and provide traceability function.

[0046] Step S500: Periodic sampling review and model feedback optimization.

[0047] In order to ensure the long-term effectiveness of the system and continuously reduce the "false negative" rate, the system performs a closed-loop optimization process: Sampling review: The system periodically (such as every week or every month) randomly selects samples from the "low risk" data that have been released according to the preset proportion. These samples are sent to an independent deep review pool for more detailed manual review by experts to find potential risks missed by the AI model.

[0048] Model optimization: The newly discovered "risk" samples in the sampling review and the results of daily manual review are fed back to the training set of the risk assessment model in step S200. The system re-trains the model regularly or triggered to make it evolve and improve the accuracy of model evaluation.

[0049] Embodiment Two A system for implementing the above method, as shown, includes the following modules: Figure 2 Data interface module: for connecting with the upstream black-box computing environment and downstream data users, responsible for receiving output data and forwarding the results of risk review and disposal to data users. Data interface module: for connecting with the upstream black-box computing environment and downstream data users, responsible for receiving output data and forwarding the results of risk review and disposal to data users.

[0050] Feature extraction engine: for performing step S100, parsing and feature calculation on the received output data.

[0051] AI risk assessment engine: for performing step S200, built-in trained risk assessment model, responsible for calculating risk score.

[0052] Risk disposal engine: for performing step S300, according to the risk score evaluation data, data release, blocking output or pushing artificial review module review.

[0053] Artificial review module: provides a graphical human-computer interaction interface, highlights the main risk feature points identified by the AI model, and assists administrators in making quick decisions. Administrators can make final instructions such as "approve release", "reject output" or "release after specifying desensitization strategy".

[0054] Encrypted evidence and audit module: for performing step S400, responsible for generating data digest and audit log, encrypting and block chain storing audit log, guaranteeing the integrity of audit data.

[0055] Sampling and feedback learning module: for performing step S500, managing sampling strategy, collecting feedback data, providing expert review pool, supporting data annotation based on review results, and starting model retraining process.

[0056] In this embodiment, the data interface module receives the output data of the black-box settlement environment; The feature extraction engine performs multi-dimensional feature extraction on the output data to obtain a feature vector, and the feature dimensions include but are not limited to: Data volume feature: including data record number, data total size, field / attribute number. Used to judge the granularity of data output, and abnormally large data volume may mean excessive detailed information leakage.

[0057] Statistical distribution feature: for structured data, calculate the statistical values of numerical fields (such as maximum value, minimum value, average value, median, standard deviation, variance, unique value count, etc.), and compare with the overall distribution of input data. The output that significantly deviates from the overall distribution may contain individual sensitive information.

[0058] Semantic-sensitive features: Utilize natural language processing techniques to identify sensitive information in text-based output. By pre-defining a library of sensitive information patterns (such as regular expressions or keywords for ID numbers, mobile numbers, bank card numbers, names, addresses), detect whether the output contains explicit sensitive data.

[0059] Model-specific features (when the output is a machine learning model): Extract the model's complexity indicators (such as the depth of tree models, the number of layers and parameter size of neural networks), performance indicators (such as accuracy, AUC value on the test set). Excessive model complexity or performance may indicate overfitting, which poses a risk of memorizing training data privacy.

[0060] AI risk assessment engine receives the feature vector, inputs into the pre-set risk assessment model for risk assessment, outputs a quantitative risk score (e.g. probability value between 0 and 1), and according to the set score threshold interval, divides the output data into different risk levels.

[0061] Risk assessment model: Can be implemented using machine learning algorithms such as gradient boosting decision trees, random forests, deep learning, etc. This model is trained through a large number of labeled historical data (including "safe" and "at risk" output samples) through supervised learning, and can learn complex, non-linear risk patterns. Compared with static rules, AI models have stronger generalization ability and potential for detecting unknown threats.

[0062] Risk handling engine receives the evaluation results of the AI risk assessment engine, according to the risk level, triggers the corresponding handling strategy: Low risk (score below threshold T1): Determine as basically safe, allow data to be directly output to the requester.

[0063] High risk (score between thresholds T1 and T2): Determine as having suspicious risk. The system automatically creates a manual review task and pushes it to the manual review module for manual review.

[0064] Extremely high risk (score above threshold T2): The system automatically and in real-time blocks the output of this batch of data, and immediately generates a high-level security alert to notify the security team for emergency handling.

[0065] Manual review module receives manual review tasks and pushes them to designated data security administrators through message notification or task list. The review interface highlights the main risk features identified by the AI model, assisting administrators in making quick decisions. Administrators can issue final instructions such as "approve release", "reject output" or "release after specifying desensitization strategy".

[0066] The storage and audit module generates a unique digital fingerprint of the batch of output data (e.g. using SHA-256 algorithm to calculate hash value) through digital digest technology for the output data determined as low risk by both the risk handling engine and the manual review module. The digital fingerprint is bound with the metadata of the current computing task (including but not limited to: task ID, timestamp, data requester identity, black box environment identifier, risk assessment score, handling action, reviewer ID) to form a complete audit log. The audit log is encrypted and stored in the blockchain for storage and evidence, and the traceability function is provided.

[0067] The data released with low risk and the handling results of medium and high risk are forwarded to the data user through the data interface module.

[0068] The sampling and feedback learning module periodically (e.g. weekly or monthly) randomly samples samples from the "low risk" data released according to a preset proportion. These samples are sent to an independent deep review pool for more detailed manual review by experts to find potential risks missed by the AI model. The newly discovered "risk" samples in the sampling review and the results of the daily manual review are used as new labeled data to feedback to the training set of the risk assessment model, and the model is retrained periodically or triggered to evolve and improve the accuracy of model evaluation Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD ROM , optical storage, etc.) containing computer usable program code.

[0069] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowchart and / or block diagram. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0070] ​These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The flow or flows and / or blocks Figure 1 The flow or flows and / or blocks

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The flow or flows and / or blocks Figure 1 The flow or flows and / or blocks

[0072] The above description is only preferred embodiments of the present application and is not intended to limit the present application. The present application can be variously changed and modified by those skilled in the art without departing from the spirit and scope of the present application. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art without departing from the spirit and scope of the present application should be included in the scope of the present application.

Claims

1. A data risk detection method for black-box computing environments, characterized in that, Includes the following steps: Intercept the computation results from the black-box computing environment, analyze them, and extract several of the following features: data volume characteristics, statistical distribution characteristics, semantically sensitive characteristics, and model-specific characteristics. The extracted features are combined into a feature vector, which is then evaluated using a pre-trained risk assessment model to obtain a quantified risk score. Based on the set score threshold range, the risk level of the calculated result is determined. Based on the risk level, the corresponding response strategy is triggered to block the calculation result where the risk level exceeds the set level; Encrypt and preserve the allowed calculation results.

2. The data risk detection method for black-box computing environments as described in claim 1, characterized in that, It also includes the following steps: Periodically, samples are randomly selected from the allowed output calculation results according to a preset ratio, and the samples are reviewed. If the review reveals inaccuracies in the risk level, the calculation results and the reviewed risk level are used as new training samples and fed back into the risk assessment model for retraining.

3. The data risk detection method for black-box computing environments as described in claim 1, characterized in that, The data volume characteristics include the number of data records, the total data size, and the number of fields / attributes, which are used to determine the granularity of the data output. Data volume exceeding the set value indicates that there may be excessively detailed information leakage.

4. The data risk detection method for black-box computing environments as described in claim 1, characterized in that, The statistical distribution characteristics include the statistical values ​​of numerical fields calculated for structured data, and compared with the overall distribution of the calculation results. Calculation results that deviate from the overall distribution by more than a set range may contain sensitive individual information.

5. A data risk detection method for black-box computing environments as described in claim 1, characterized in that, The semantically sensitive features include the results of sensitive information identification in text-based output content using natural language processing technology. Through a predefined sensitive information pattern library, which stores several regular expressions or keywords of ID card number, mobile phone number, bank card number, name, and address, the detection and calculation results are used to determine whether they contain explicit sensitive data.

6. The data risk detection method for black-box computing environments as described in claim 1, characterized in that, The model-specific features are obtained by extracting the model's complexity and performance metrics. Model complexity or performance exceeding predetermined values ​​may indicate overfitting and pose a risk to the privacy of training data.

7. A data risk detection method for black-box computing environments as described in claim 1, characterized in that, The risk assessment model is a machine learning model that is trained using labeled historical data through supervised learning.

8. The data risk detection method for black-box computing environments as described in claim 1, characterized in that, The process of determining the risk level of the calculation result based on the set scoring threshold range includes: if the score is lower than the threshold T1, it is judged to be basically safe, and the calculation result data is allowed to be directly output to the requester; If the score is between the thresholds T1 and T2, it is determined that there is a suspicious risk. A manual review task is created and pushed to the designated data security administrator via message notification or task list. If the score is higher than the threshold T2, the output of the corresponding calculation result data will be automatically and in real time blocked, and a high-level security alarm will be generated immediately to notify the security team to handle the emergency.

9. A data risk detection method for black-box computing environments as described in claim 1, characterized in that, The process of encrypting and storing the allowed output calculation results includes: generating a unique digital fingerprint of the calculation result data through digital digest technology; binding the digital fingerprint with the metadata of this calculation task to form a complete audit log; The audit logs are encrypted and stored on the blockchain for evidence preservation, and traceability is provided. The metadata of a computation task includes several of the following: task ID, timestamp, data requester identity, black-box environment identifier, risk assessment score, action taken, and reviewer ID.

10. A data risk detection system for black-box computing environments, characterized in that, include: The feature extraction engine is used to intercept the computation results of the black-box computing environment, parse them, and extract several of the following features: data volume features, statistical distribution features, semantically sensitive features, and model-specific features. The AI ​​risk assessment engine is used to combine the extracted features into a feature vector, use a pre-trained risk assessment model to evaluate the risk, obtain a quantitative risk score, and determine the risk level of the calculated result based on the set score threshold range. The risk management engine is used to trigger corresponding management strategies based on the risk level, and to block calculation results where the risk level exceeds the set level. The encrypted evidence storage module is used to encrypt and store the allowed calculation results.

Images