Data processing method and device, storage medium and electronic equipment

Abstract

The present disclosure relates to a data processing method, device, storage medium and electronic equipment, the method comprising: obtaining full-volume original data, and storing the full-volume original data in a data sandbox set by a blockchain; performing desensitization processing on the full-volume original data to obtain sample data; obtaining a sensitive data review index based on the full-volume original data, the sensitive data review index being used to evaluate the desensitization degree of the sample data; and reviewing the sample data according to the sensitive data review index to obtain open data. The present disclosure can ensure that the value characteristics of the data are less lost while ensuring that the desensitization degree is high enough.

Description

Technical Field

[0001] This disclosure relates to the field of data security technology, and more specifically, to a data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the development of big data, artificial intelligence and Internet of Things technologies, how to ensure that personal privacy data and sensitive corporate data are only provided to authorized data users and not obtained by other participants is an urgent technical problem to be solved. Summary of the Invention

[0003] The purpose of this disclosure is to provide a data processing method, apparatus, storage medium, and electronic device to at least solve some of the aforementioned related technical problems.

[0004] To achieve the above objectives, according to a first aspect of the present disclosure, a data processing method is provided, the method comprising:

[0005] Obtain all raw data and store it in the data sandbox set up by the blockchain;

[0006] The original data was anonymized to obtain sample data.

[0007] Sensitive data review indicators are obtained based on the full set of original data. These sensitive data review indicators are used to evaluate the degree of anonymization of the sample data.

[0008] The sample data is reviewed according to the aforementioned sensitive data review indicators to obtain open data.

[0009] Optionally, obtaining sensitive data review indicators based on the full set of original data includes:

[0010] The information change rate is obtained based on the full amount of original data, and the information change rate is the amount of information lost after performing the desensitization process.

[0011] Obtain a first indicator, which is used to represent the feature diversity of the data after performing the de-identification process;

[0012] The sensitive data review indicator is obtained based on the information change rate and the first indicator.

[0013] Optionally, obtaining the first indicator includes:

[0014] Obtain the weight of each feature corresponding to the full set of original data, and obtain the number of first equivalence classes for each feature;

[0015] The first indicator is obtained based on the weight of each feature, the number of the first equivalence classes, and the total number of data entries in the full original data.

[0016] Optionally, obtaining sensitive data review indicators based on the information change rate and the first indicator includes:

[0017] Extract the set of quasi-identifiers from the full set of original data, and obtain the number of second equivalence classes corresponding to the full set of quasi-identifiers.

[0018] A second indicator is obtained based on the number of the second equivalence classes and the total number of data entries in the full original data. The second indicator is used to represent the data diversity of the full original data.

[0019] Sensitive data review indicators are obtained based on the information change rate, the first indicator, and the second indicator.

[0020] Optionally, obtaining the information change rate based on the full set of original data includes:

[0021] Obtain the first information entropy of the full set of original data, where the first information entropy is the amount of information carried by all features corresponding to the full set of original data;

[0022] Obtain the second information entropy of the full original data after de-identification processing. The second information entropy is the amount of information carried by all features corresponding to the full original data after de-identification processing.

[0023] The rate of information change is determined based on the first and second information entropies.

[0024] Optionally, the method further includes:

[0025] When a data user is detected to have a data usage requirement, the identity and usage permissions of the data user are verified, and an approval instruction is sent to the data user after the verification is successful.

[0026] Optionally, the method further includes:

[0027] The identities of each member in the data sandbox are recorded on the blockchain. These members include data providers, data auditors, and data users.

[0028] According to a second aspect of the present disclosure, a data processing apparatus is provided, comprising:

[0029] The data acquisition module is configured to acquire all raw data and store the raw data in a data sandbox set up by the blockchain;

[0030] The desensitization module is configured to perform desensitization processing on the full set of original data to obtain sample data;

[0031] The indicator acquisition module is configured to acquire sensitive data review indicators based on the full set of original data. The sensitive data review indicators are used to evaluate the degree of desensitization of the sample data.

[0032] The review module is configured to review the sample data based on the sensitive data review indicators to obtain open data.

[0033] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0034] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0035] A memory on which computer programs are stored;

[0036] A processor for executing the computer program in the memory to implement the steps of the method described in the first aspect above.

[0037] This disclosed embodiment ensures a sufficiently high level of data anonymization while minimizing the loss of data value characteristics through sensitive data review indicators. Specifically, it acquires all raw data and stores it in a data sandbox set up by the blockchain. Anonymization processing is then performed on the raw data to obtain sample data. Based on this, sensitive data review indicators are obtained from the raw data. These indicators are used to evaluate the anonymization level of the sample data. Finally, the sample data is reviewed according to the sensitive data review indicators to obtain open data. This approach simultaneously considers both data value and sensitivity.

[0038] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0039] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0040] Figure 1 This is a flowchart illustrating a data processing method according to an exemplary embodiment of this disclosure.

[0041] Figure 2 This is an example diagram illustrating the process of uploading the identities of each member in a data sandbox to the blockchain in an exemplary embodiment of this disclosure.

[0042] Figure 3 This is a schematic diagram illustrating the review and approval of open data in a data sandbox in an exemplary embodiment of this disclosure.

[0043] Figure 4 This is a flowchart illustrating another data processing method as shown in an exemplary embodiment of this disclosure.

[0044] Figure 5 This is an example diagram illustrating another data processing method for obtaining open data as shown in an exemplary embodiment of this disclosure.

[0045] Figure 6 This is a block diagram of a data processing apparatus shown in an exemplary embodiment of the present disclosure.

[0046] Figure 7 This is a block diagram of an electronic device shown in an exemplary embodiment of the present disclosure. Detailed Implementation

[0047] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0048] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0049] In the description of this disclosure, terms such as "first" and "second" are used to distinguish similar objects and should not be construed as indicating a specific order or sequence. Furthermore, unless otherwise stated, in the description with reference to the accompanying drawings, the same reference numerals in different drawings denote the same elements.

[0050] Although operations or steps are described in a specific order in the accompanying drawings in the embodiments of this disclosure, it should not be construed as requiring these operations or steps to be performed in the specific order or serial order shown, or requiring all of the shown operations or steps to be performed to obtain the desired result. In the embodiments of this disclosure, these operations or steps may be performed serially; they may be performed in parallel; or a portion of these operations or steps may be performed.

[0051] The essence of a data sandbox is security, and its core is computation. Through a data sandbox, data analysis and computation can be performed in a way that ensures "data is usable but not visible" when the data is physically or logically centralized and does not leave the local domain.

[0052] Currently, the requirements for data sandboxes include ensuring secure computation of data while guaranteeing the protection of privacy and preventing data leakage, thereby maximizing the value of the data. Based on this, ensuring the security and compliance of open data, formulating review and approval rules, and ensuring a transparent and traceable open process have become crucial issues that data sandbox technology must address.

[0053] The relevant technologies for data security auditing systems are still imperfect, especially when it comes to auditing the operation logs of data users. It is often difficult to ensure that the operations of each data user are compliant. Therefore, it is necessary to establish an effective auditing mechanism to protect data security.

[0054] Furthermore, regarding the compliance of open data, related technologies often employ sensitive word or feature detection methods. While these methods strictly guarantee the format of open data, data modeling typically relies on the integrity of data features. Therefore, even after data anonymization, in actual modeling, it's still necessary to consider preserving the data's characteristic attributes and their relationships to avoid compromising its analytical value. Common anonymization methods can all potentially lead to the loss of valuable data features, especially overly strict review mechanisms, which may result in the loss of data features, thereby affecting the effectiveness of data analysis.

[0055] To address the aforementioned problems, embodiments of this disclosure illustrate, as follows: Figure 1 The data processing method shown is achieved by... Figure 1 It is known that the method may include steps S110 to S140.

[0056] In step S110, the full amount of raw data is obtained and stored in the data sandbox set by the blockchain.

[0057] In some implementations, the entire set of raw data can be generated by a data provider, which can be an organization that requires data openness. For example, a data provider could be a data operation platform or data middleware platform of a government or medical institution. It is evident that data providers are organizations that require data openness; they typically possess a large amount of raw data, but most lack the capability for data processing, analysis, and compliant data disclosure.

[0058] Therefore, data providers can store all their raw data in a data sandbox set up by the blockchain, so that data auditors can review the data and, after the review is passed, data users can better process and analyze the data.

[0059] Here, the data approver can be a data compliance verification body, meaning it can be a dedicated verification agency of the data provider or an authoritative third-party organization. For example, the data approver could be a security department, information and data security center, etc. It is clear that the data approver is primarily responsible for verifying the compliance of the processed data. Once approved, the data is provided to the data user (data developer) for analysis and development.

[0060] Optionally, the data user can be an organization specializing in data analysis or data model training. Alternatively, the data user can be the data provider's internal data analysis department or a third-party outsourced code development organization. It should be noted that the data disclosed to the data user can be anonymized, secure, and compliant data.

[0061] In summary, this disclosure embodiment can use distributed digital identity to issue and verify the digital identities of each member in the data sandbox, that is, to put the identities of each member in the data sandbox on the blockchain, so as to jointly construct a consortium blockchain network based on the identities of each member. Each member can include a data provider, a data auditor, and a data user. To better understand the relationship between these three parties, this disclosure embodiment provides the following... Figure 2 The example diagram is shown.

[0062] pass Figure 2 Knowing that, before data processing, this embodiment of the disclosure can first put the identities of data sandbox members on the blockchain, that is, add the identity information (DID) of each member to blockchain 201. Specifically, the identity information of data providers A1 and A2 can be added to the first block of blockchain 201; the identity information of data reviewer B1 can be added to the second block of blockchain 201, and the identity information of data reviewer B2 can be added to the third block of blockchain 201; the identity information of data user C1 can be added to the fourth block of blockchain 201, and the identity information of data users C2 and C3 can be added to the fifth block of blockchain 201. Thus, by leveraging the consortium blockchain composed of data sandbox members and the on-chain digital identities, this embodiment of the disclosure can ensure that the identities of members are trustworthy and verifiable.

[0063] Optionally, embodiments of this disclosure can use blockchain digital identity technology to upload multiple identities to the blockchain, and can also upload approval processes to the blockchain to ensure identity security and the traceability of approval processes. The approval process can include which party provides the data, which party reviews the data, and which party uses the data.

[0064] in addition, Figure 2A1 and A2 in the sandbox can be different data providers. For example, data providers A1 / A2 can connect their own data source (full original data) to the sandbox. Based on this, they can perform anonymization sampling on the full original data to obtain sample data, which leads to step S120. After that, data approvers B1 and / or B2 can review the anonymized sample data. Finally, data users C1, C2, or C3 can use the reviewed data.

[0065] In step S120, desensitization processing is performed on all the original data to obtain sample data.

[0066] In some implementations, after a data provider connects its data source to the sandbox, it can perform de-identification sampling on the entire original data to obtain sample data. This means that de-identification processing is performed on the entire original data to obtain sample data. Here, the de-identification processing is used to generalize the entire original data to ensure data security.

[0067] As an example, different k values ​​can be selected for N desensitization operations to obtain different desensitized data, thus obtaining multiple sample data.

[0068] In step S130, sensitive data audit indicators are obtained based on the full amount of raw data.

[0069] As described above, after processing the full set of raw data to obtain sample data, the data reviewer needs to review the sample data to obtain open data. In this process, this embodiment of the disclosure can also first obtain sensitive data review indicators. These sensitive data review indicators can be used to assess the degree of anonymization of the sample data. Here, the sensitive data review indicators can be obtained based on information change rate, a first indicator, and a second indicator, etc. The following embodiments will detail how to obtain these sensitive data review indicators.

[0070] In step S140, the sample data is reviewed according to the sensitive data review indicators to obtain open data.

[0071] As an optional approach, after obtaining the sensitive data review indicators, this embodiment of the disclosure can review the sample data according to the sensitive data review indicators to obtain open data. Specifically, it is determined whether the sensitive data review indicators meet preset conditions. If it is determined that the sensitive data review indicators meet the preset conditions, the sample data is used as open data.

[0072] For example, determine whether the sensitive data review indicators exceed preset values. If the sensitive data review indicators exceed the preset values, the sample data is deemed qualified and can be used as open data. Conversely, if the sensitive data review indicators do not exceed the preset values, the sample data is deemed unqualified. In this case, the entire original data can be desensitized again to obtain new sample data, and the above operation can be repeated.

[0073] In summary, after obtaining the sample data, the data reviewer can review it and obtain the open data upon approval. This review of the sample data can involve compliance checks and data quality checks to ensure the accuracy of subsequent data analysis.

[0074] As an alternative approach, when a data user has a data usage request, the data provider can verify the user's identity and usage permissions, and send an approval instruction to the user upon successful verification. During this process, the data user can apply for access to the open data by sending a permission request to the data provider. At this point, the data provider can verify and approve the user's digital identity and data usage permissions, and upon approval, provide the open data to the user so that they can perform data analysis or modeling code development based on the open data. For detailed procedures, please refer to [link to relevant documentation]. Figure 3 .

[0075] As described above, in this disclosure, the use of any open data requires an application from the data user, followed by strict approval from the data provider based on the user's digital identity and data permissions. Only after approval is granted can the user have the right to use the data and perform data analysis. The entire process includes review, application, and approval operations, all of which are stored on the blockchain to ensure that any data leakage issues are traceable and accountable.

[0076] This disclosed embodiment ensures a sufficiently high level of data anonymization while minimizing the loss of data value characteristics through sensitive data review indicators. Specifically, it acquires all raw data and stores it in a data sandbox set up by the blockchain. Anonymization processing is then performed on the raw data to obtain sample data. Based on this, sensitive data review indicators are obtained from the raw data. These indicators are used to evaluate the anonymization level of the sample data. Finally, the sample data is reviewed according to the sensitive data review indicators to obtain open data. This approach simultaneously considers both data value and sensitivity.

[0077] This disclosure illustrates embodiments such as Figure 4 Another data processing method shown is through Figure 4Knowing that the method may include steps S410 to S460.

[0078] In step S410, the full amount of raw data is obtained and stored in the data sandbox set by the blockchain.

[0079] In step S420, desensitization processing is performed on all the original data to obtain sample data.

[0080] The specific implementation methods of steps S410 to S420 have been described in detail in the above embodiments, and will not be repeated here.

[0081] As described above, in order to ensure the security and controllability of the review and approval process, this embodiment of the disclosure can upload the identities of each member in the data sandbox to the blockchain, that is, upload the identities of data sandbox members to the blockchain, such as... Figure 5 As shown. Based on this, the data auditor can obtain sensitive data audit indicators and review and approve open data based on these indicators. These operations can all be stored on the blockchain, thus ensuring the evidence preservation of the open data approval process. In the process of obtaining sensitive data audit indicators, this embodiment of the disclosure can first obtain the information change rate, i.e., proceed to step S430.

[0082] In step S430, the information change rate is obtained based on the full amount of original data.

[0083] In some implementations, the information change rate can be the amount of information lost after the anonymization process. In obtaining the information change rate, embodiments of this disclosure can first obtain two information entropies. Specifically, a first information entropy is obtained from the full original data, where the first information entropy can be the amount of information carried by all features corresponding to the full original data. Next, a second information entropy is obtained from the full original data after the anonymization process, where the second information entropy can be the amount of information carried by all features corresponding to the full original data after the anonymization process. Based on this, the information change rate is determined based on the first and second information entropies.

[0084] It should be noted that the first information entropy can be obtained based on the full set of original data or on sampled data. Here, the sampled data can be a portion of the original data selected from the full set of original data, and it must be the data before anonymization processing. Similarly, the second information entropy can be obtained based on the full set of original data after anonymization processing or on sample data; that is, the second information entropy can be obtained based on sampled data after anonymization processing. The specific data used to obtain the first and second information entropies is not explicitly limited here and can be selected according to the actual situation.

[0085] As a specific implementation method, in the process of obtaining the first information entropy H(x), this embodiment of the disclosure can obtain multiple features D = {x1, x2, x3, ..., x...} corresponding to the full amount of original data. i , ..., x n}, where the total number of data entries in the full original data set is N; x i Let i be the i-th feature of the data. Based on this, obtain the equivalence classes for each feature, where data with the same feature value in each feature can belong to the same equivalence class. For example, a data entry with the age of 18 can be one equivalence class, and a data entry with the age of 40 can be another. Similarly, a data entry with the anonymized ID number 21012****** can be one equivalence class, and a data entry with the anonymized ID number 21013****** can be another.

[0086] Based on the above partitioning method, the equivalence classes of the i-th feature can be: in, It can be the j-th equivalence class of the i-th feature, where m is the number of equivalence classes.

[0087] As an optional approach, after obtaining the equivalence classes of each feature, embodiments of this disclosure can determine the proportion between the number of data entries in each equivalence class and the total number of data entries in the entire original data set. That is, they can count the number of each equivalence class and calculate the ratio between the number of each equivalence class and the total number of data entries in the entire original data set. These ratios represent the proportion of feature equivalence classes to the total number of data entries in the dataset, and this proportion can be expressed as: in, For the data entry of the j-th equivalence class of the i-th feature, The proportion of data entries belonging to the j-th equivalence class of the i-th feature to the total number of data entries.

[0088] Furthermore, classification or prediction algorithms require calculating the degree of influence of features on classification or predicted labels. Therefore, embodiments of this disclosure can first obtain the feature weight of each feature and combine the feature weight w i and data entries for each equivalence class The proportion between the total number of data entries N in the full original data set and the total number of data entries N. To obtain the first information entropy H(x).

[0089] Specifically, in this embodiment, the algorithm evaluation index under the baseline model can be obtained first. That is, in the case of the full amount of original data, the classification prediction model index is used to perform F1 scoring. The specific calculation formula is as follows: Where P is the accuracy of the prediction result, R is the recall rate, and F is the score. Based on this, x is then sequentially... iFeatures are removed from the dataset to form a new dataset, and the prediction model is continued based on the new dataset to obtain new evaluation metrics. The final list of metrics F is obtained. total =[F1,F2,F3,…,F i ,...,F n-1 ,F n The list can be the scores after removing each feature.

[0090] Based on this, the embodiments of this disclosure can calculate the rate of change of the score ΔF, that is, ΔF=[ΔF1,ΔF2,ΔF3,…,ΔF i ,…,ΔF n ]. Wherein, ΔF i =FF i ΔF i Under normal circumstances, it is a positive number. If it is a negative number, it can be 0 or the minimum value can be selected from the ΔF list and replaced with the minimum value.

[0091] It should be noted that the F1 score, the accuracy P, or other model evaluation metrics can be used as the evaluation metric, such as the AUC curve.

[0092] As an alternative approach, after obtaining the rate of change in scores ΔF, embodiments of this disclosure can normalize ΔF to obtain feature weights W, where W = [w1, w2, ..., w i ,…,w n The normalization formula can be...

[0093] It should be noted that the feature weights proposed here are specific implementation methods for the weights. The choice of prediction algorithm and model evaluation metric can be arbitrary, mainly depending on the purpose of the data. Therefore, there are no explicit restrictions on the specific prediction algorithm or model evaluation metric to be selected; they can be chosen according to the actual situation.

[0094] Based on this, the embodiments of this disclosure can incorporate the feature weight w i And the ratio between the number of data entries in each equivalence class and the total number of data entries in the original dataset. To obtain the first information entropy H(x), the first information entropy can be the information entropy of the feature, and its specific calculation formula can be:

[0095]

[0096] After simplification, the formula is as follows: Wherein, H(x) i ) is a feature xi The amount of information carried. Next, embodiments of this disclosure can calculate the sum of the information entropy of all features. The sum of these information entropies can be used as the first information entropy H(x).

[0097] In this embodiment of the disclosure, the second information entropy The process of obtaining the first information entropy H(x) is similar to that of obtaining the first information entropy H(x), and will not be described in detail here. The difference lies in the number of equivalence class entries; that is, the number of equivalence classes usually decreases after desensitization. Finally, the information change rate in this embodiment of the disclosure...

[0098] Because the desensitized data needs to undergo desensitization processing such as generalization or masking, It represents the proportion of anonymized data to the original data. In other words, the information change rate ΔH(x) represents the amount of data information lost after anonymization; the larger the information change rate, the more original data information is retained.

[0099] As a specific implementation method, when the information change rate ΔH(x) = 1, it indicates that the de-identified data has the same amount of information as the original data, meaning that the de-identification process did not result in information loss. Therefore, this embodiment of the disclosure can use the information change rate ΔH(x) as a criterion for judging the amount of valuable feature data retained after de-identification. The larger the information change rate ΔH(x), the more valuable feature data is retained. Conversely, the smaller the information change rate ΔH(x), the less valuable feature data is retained.

[0100] In step S440, the first indicator is obtained.

[0101] As an alternative approach, during the process of obtaining sensitive data audit indicators, embodiments of this disclosure may also obtain a first indicator, wherein the first indicator can be used to represent the feature diversity of the data after the anonymization process is performed.

[0102] Specifically, embodiments of this disclosure can obtain the weight w of each feature corresponding to the full amount of original data. i And obtain the number m of the first equivalence classes for each feature. i Based on this, according to the weight w of each feature... i The number of first equivalence classes m i The first metric is obtained by calculating the total number of data entries N in the full set of raw data. The specific calculation formula can be... Among them, w i Let m be the weight of the i-th feature. i Let N be the number of equivalence classes for the i-th feature, and N be the total number of data entries.

[0103] As an example, when all feature values ​​of the i-th feature are generalized, it can be considered as an equivalence class, i.e., m i =1, when all features are generalized to an equivalence class, the first index can be expressed as

[0104] As another example, when all eigenvalues ​​are not simultaneously m i =N, the first index can be expressed as

[0105] As can be seen, in this embodiment of the disclosure, the range of values ​​for the first index T can be... The magnitude of the first indicator represents the categorical diversity of each feature. A smaller value indicates higher feature similarity, signifying a higher degree of anonymization, and thus a lower probability of identifying specific data through the feature value. Conversely, a larger first indicator indicates lower feature similarity, signifying a lower degree of anonymization, and thus a higher probability of identifying specific data through the feature value. In summary, this embodiment of the disclosure uses the first indicator T to represent data sensitivity; lower data sensitivity indicates better privacy protection of the dataset.

[0106] In step S450, sensitive data audit indicators are obtained based on the information change rate and the first indicator.

[0107] In some implementations, after obtaining the information change rate and the first indicator, embodiments of this disclosure can obtain the sensitive data review indicator S based on the information change rate ΔH(x) and the first indicator T. For example, the sensitive data review indicator... It is evident that the larger the S value of the sensitive data review index, the less the data's value characteristics are lost, and the higher the degree of desensitization.

[0108] In other embodiments, during the process of obtaining sensitive data review indicators based on the information change rate and the first indicator, embodiments of this disclosure may also obtain a second indicator, wherein the second indicator is used to represent the data diversity of the entire original data. Based on this, the sensitive data review indicators are obtained by combining the first indicator, the second indicator, and the information change rate.

[0109] Specifically, in this embodiment of the disclosure, a set of quasi-identifiers can be extracted from the full set of original data, and the number of second equivalence classes corresponding to the full set of original data can be obtained based on the set of quasi-identifiers. On this basis, a second indicator can be obtained based on the number of second equivalence classes and the total number of data entries in the full set of original data. Finally, a sensitive data review indicator can be obtained based on the information change rate, the first indicator, and the second indicator.

[0110] The quasi-identifier set can be a set of features that can predict sensitive attributes, such as age, gender, address, and postal code. In this embodiment, certain attributes can be manually or automatically designated as quasi-identifiers. Here, several data entries with the same quasi-identifier features can be considered as an equivalence class, i.e., a second equivalence class.

[0111] As a specific implementation method, if the number of equivalence classes in the dataset is r and the total number of records in the dataset is N, then the formula for obtaining the second indicator can be expressed as follows:

[0112] In this embodiment, the smaller the second index, the higher the feature value similarity, indicating a higher degree of desensitization. Conversely, the larger the second index, the lower the feature value similarity, indicating a lower degree of desensitization. The domain of the second index is defined similarly to the K-anonymity algorithm. Specifically, when there are fewer equivalence classes, the same equivalence class contains more identical data, and the data is less sensitive. For example, when r=1, the second index is at its minimum. When there are more equivalence classes, the fewer identical data are contained in the same equivalence class, and the more sensitive the data is. For example, when r=N, the second indicator is at most 1.

[0113] The formula for obtaining the sensitive data review indicator S by combining the first indicator T, the second indicator D, and the information change rate ΔH(x) can be: Similarly, the larger the S value of the sensitive data review indicator, the less the data's value characteristics are lost and the higher the degree of desensitization.

[0114] In step S460, the sample data is reviewed according to the sensitive data review indicators to obtain open data.

[0115] In some implementations, sensitive data review indicators can be used as evaluation indicators for de-identification algorithms or sample data. That is, based on sensitive data review indicators, this disclosure embodiment can evaluate and select multiple de-identification algorithms or multiple de-identified data.

[0116] As an example, in the k-anonymity algorithm, suppose we have a dataset D = {td, x1, x2, x3, ..., x}. i , ..., x n The dataset D can be a database table. Here, td can be a non-sensitive attribute, {x1, x2, x3, ..., x}. i ,…,x n} is the quasi-identifier, s is the sensitive attribute, and td is any other non-critical attribute besides the quasi-identifier and the sensitive attribute.

[0117] After obtaining the dataset D, embodiments of this disclosure can divide the dataset into equivalence classes C = {C1, C2, C3, ..., C...} based on quasi-identifiers. j ,…,Cm}, |C j | ≥ k, 1 ≤ j ≤ m, where, for each element in the equivalence class C j the values on each quasi-identifier x i are the same, and the number of data in the C j classes that meet the requirements can be greater than k. In the embodiments of the present disclosure, the value of k can be dynamically set to determine whether the data meets the requirements after data desensitization.

[0118] Specifically, assuming that the number of data in the current data set is N, the value range of k is [1, N]. In the embodiments of the present disclosure, all eligible k values can be selected in sequence for N selections of desensitized data, that is, one k value can correspond to one selection of desensitized data. On this basis, the corresponding sensitive data audit index S is calculated for the N selections of desensitized data respectively, and then the maximum value is selected from the multiple sensitive data audit indexes, and the k value of the data set corresponding to the algorithm of this maximum value is used as the k value for data compliance discrimination. In other words, in the embodiments of the present disclosure, the k value of the data set corresponding to the algorithm with the largest S value can be selected as the k value for data compliance discrimination.

[0119] After obtaining the k value, in the embodiments of the present disclosure, the anonymization algorithm with this k value can be used to audit the open data. If |C j | < k, it means that the sample data is unqualified, and at this time, the desensitization process needs to be carried out again. Optionally, if |C j | > k, it means that the sample data is qualified, that is, the data audit is passed. At this time, the data that passes the audit can be selected, and the maximum information change rate can be calculated based on these data That is, the data set with the smallest information loss can be used as the open data set.

[0120] It should be noted that in the open data audit and selection method, the embodiments of the present disclosure mention the specific method of data audit. And this method is only one way of using the open data evaluation index, and various methods of data sensitivity detection and screening can adopt this sensitive data audit index.

[0121] In summary, for the problem of retaining the value characteristics of the data after desensitization, the embodiments of the present disclosure calculate the information loss before and after data desensitization, that is, obtain the information change rate, and for the problem of the degree of data desensitization, calculate the feature diversity (the first index) and the data set diversity (the second index). Finally, a sensitive data audit index based on the data feature value and sensitivity is obtained, and this sensitive data audit index can be used to measure the quality of the data. In addition, the embodiments of the present disclosure also propose corresponding data audit and opening methods, that is, the final high-quality open data can be obtained according to the sensitive data audit index.

[0122] This embodiment of the disclosure uses sensitive data audit indicators to ensure a sufficiently high level of de-identification while minimizing the loss of data's value characteristics. Specifically, it acquires all raw data and stores it in a data sandbox set up by the blockchain. De-identification processing is performed on the entire raw data to obtain sample data. Based on this, sensitive data audit indicators are obtained from the entire raw data. These indicators are used to evaluate the degree of de-identification of the sample data. Finally, the sample data is audited according to the sensitive data audit indicators to obtain open data. This approach simultaneously considers the data's characteristic value and sensitivity. Furthermore, this embodiment of the disclosure combines information change rate, a first indicator, and a second indicator to effectively obtain more accurate sensitive data audit indicators.

[0123] Figure 6 This is a data processing apparatus 600 illustrated according to an exemplary embodiment, such as... Figure 6 As shown, the data processing device 600 may include a data acquisition module 610, a desensitization module 620, an indicator acquisition module 630, and an auditing module 640.

[0124] The data acquisition module 610 is configured to acquire all raw data and store the raw data in the data sandbox set up by the blockchain;

[0125] The desensitization module 620 is configured to perform desensitization processing on the full set of original data to obtain sample data;

[0126] The indicator acquisition module 630 is configured to acquire sensitive data review indicators based on the full set of original data. The sensitive data review indicators are used to evaluate the degree of desensitization of the sample data.

[0127] The audit module 640 is configured to audit the sample data according to the sensitive data audit indicators to obtain open data.

[0128] In some implementations, the indicator acquisition module 630 may include:

[0129] The information change rate acquisition submodule is configured to acquire the information change rate based on the full amount of original data, where the information change rate is the amount of information loss after performing the desensitization process.

[0130] The first indicator acquisition submodule is configured to acquire a first indicator, which is used to represent the feature diversity of the data after performing the de-identification process.

[0131] The audit indicator acquisition submodule is configured to acquire the sensitive data audit indicators based on the information change rate and the first indicator.

[0132] In some implementations, the first indicator acquisition submodule is further configured to acquire the weight of each feature corresponding to the full set of original data, and the number of first equivalence classes for each feature; and acquire the first indicator based on the weight of each feature, the number of first equivalence classes, and the total number of data entries in the full set of original data.

[0133] In some implementations, the indicator acquisition module 630 may further include:

[0134] The equivalence class count acquisition submodule is configured to extract the set of quasi-identifiers from the full set of original data, and obtain the number of second equivalence classes corresponding to the full set of original data based on the set of quasi-identifiers.

[0135] The second indicator acquisition submodule is configured to acquire a second indicator based on the number of the second equivalence classes and the total number of data entries in the full original data. The second indicator is used to represent the data diversity of the full original data.

[0136] The audit indicator acquisition submodule is also configured to acquire sensitive data audit indicators based on the information change rate, the first indicator, and the second indicator.

[0137] In some implementations, the information change rate acquisition submodule is further configured to: acquire a first information entropy of the full original data, the first information entropy being the amount of information carried by all features corresponding to the full original data; acquire a second information entropy of the full original data after de-identification processing, the second information entropy being the amount of information carried by all features corresponding to the full original data after de-identification processing; and determine the information change rate based on the first information entropy and the second information entropy.

[0138] In some embodiments, the data processing apparatus 600 may further include:

[0139] The verification module is configured to verify the identity and usage permissions of the data user when a data user is detected to have a data usage request, and send an approval instruction to the data user after the verification is successful.

[0140] In some embodiments, the data processing apparatus 600 may further include:

[0141] The on-chain module is configured to upload the identities of each member in the data sandbox to the blockchain. The members include data providers, data auditors, and data users.

[0142] This disclosed embodiment ensures a sufficiently high level of data anonymization while minimizing the loss of data value characteristics through sensitive data review indicators. Specifically, it acquires all raw data and stores it in a data sandbox set up by the blockchain. Anonymization processing is then performed on the raw data to obtain sample data. Based on this, sensitive data review indicators are obtained from the raw data. These indicators are used to evaluate the anonymization level of the sample data. Finally, the sample data is reviewed according to the sensitive data review indicators to obtain open data. This approach simultaneously considers both data value and sensitivity.

[0143] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0144] Figure 7 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. Figure 7 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0145] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the data processing method described above. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 703 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0146] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the data processing method described above.

[0147] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the data processing method described above. For example, the computer-readable storage medium may be the memory 702 including program instructions described above, which may be executed by the processor 701 of the electronic device 700 to complete the data processing method described above.

[0148] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the data processing method described above when executed by the programmable device.

[0149] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0150] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0151] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A data processing method, characterized in that, The method includes: Obtain all raw data and store it in the data sandbox set up by the blockchain; The original data was anonymized to obtain sample data. Sensitive data review indicators are obtained based on the full set of original data. These sensitive data review indicators are used to evaluate the degree of anonymization of the sample data. The sample data is reviewed according to the aforementioned sensitive data review indicators to obtain open data; The process of obtaining sensitive data review indicators based on the full set of original data includes: The information change rate is obtained based on the full amount of original data, and the information change rate is the amount of information lost after performing the desensitization process. Obtain a first indicator, which is used to represent the feature diversity of the data after performing the de-identification process; The sensitive data review indicator is obtained based on the information change rate and the first indicator; The acquisition of the first indicator includes: Obtain the weight of each feature corresponding to the full set of original data, and obtain the number of equivalence classes for each feature; The first indicator is obtained based on the weight of each feature, the number of equivalence classes of each feature, and the total number of data entries in the full original data. The process of obtaining sensitive data review indicators based on the information change rate and the first indicator includes: Extract the set of quasi-identifiers from the full set of original data, and obtain the number of equivalence classes corresponding to the full set of quasi-identifiers. A second indicator is obtained based on the number of equivalence classes corresponding to the full original data and the total number of data entries in the full original data. The second indicator is used to represent the data diversity of the full original data. Sensitive data review indicators are obtained based on the information change rate, the first indicator, and the second indicator.

2. The method according to claim 1, characterized in that, The step of obtaining the information change rate based on the full set of original data includes: Obtain the first information entropy of the full set of original data, where the first information entropy is the amount of information carried by all features corresponding to the full set of original data; Obtain the second information entropy of the full original data after de-identification processing. The second information entropy is the amount of information carried by all features corresponding to the full original data after de-identification processing. The rate of information change is determined based on the first and second information entropies.

3. The method according to claim 1 or 2, characterized in that, The method further includes: When a data user is detected to have a data usage requirement, the identity and usage permissions of the data user are verified, and an approval instruction is sent to the data user after the verification is successful.

4. The method according to claim 1 or 2, characterized in that, The method further includes: The identities of each member in the data sandbox are recorded on the blockchain. These members include data providers, data auditors, and data users.

5. A data processing apparatus, characterized in that, include: The data acquisition module is configured to acquire all raw data and store the raw data in a data sandbox set up by the blockchain; The desensitization module is configured to perform desensitization processing on the full set of original data to obtain sample data; The indicator acquisition module is configured to acquire sensitive data review indicators based on the full set of original data. The sensitive data review indicators are used to evaluate the degree of desensitization of the sample data. The review module is configured to review the sample data based on the sensitive data review indicators to obtain open data; The indicator acquisition module includes: The information change rate acquisition submodule is configured to acquire the information change rate based on the full amount of original data, where the information change rate is the amount of information loss after performing the desensitization process. The first indicator acquisition submodule is configured to acquire a first indicator, which is used to represent the feature diversity of the data after performing the de-identification process. The audit indicator acquisition submodule is configured to acquire the sensitive data audit indicator based on the information change rate and the first indicator; The first indicator acquisition submodule is also configured as follows: Obtain the weight of each feature corresponding to the full set of original data, and obtain the number of equivalence classes for each feature; The first indicator is obtained based on the weight of each feature, the number of equivalence classes of each feature, and the total number of data entries in the full original data. The indicator acquisition module includes: The equivalence class count acquisition submodule is configured to extract the set of quasi-identifiers from the full set of original data, and obtain the number of equivalence classes corresponding to the full set of original data based on the set of quasi-identifiers. The second indicator acquisition submodule is configured to acquire a second indicator based on the number of equivalence classes corresponding to the full original data and the total number of data entries in the full original data. The second indicator is used to represent the data diversity of the full original data. The audit indicator acquisition submodule is also configured to acquire sensitive data audit indicators based on the information change rate, the first indicator, and the second indicator.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-4.

7. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-4.

Images