A data processing method, device, apparatus, and storage medium

Abstract

The application relates to the technical field of computers, in particular to a data processing method and device, equipment and a storage medium. The method comprises the following steps: obtaining to-be-processed data and a predefined knowledge base associated with the to-be-processed data; determining metadata of the to-be-processed data; determining a data index range containing at least one preset attribute from the metadata based on the predefined knowledge base; dividing the data index range to obtain a data storage set corresponding to the at least one preset attribute; generating data index information according to the data storage set; and determining output data corresponding to the to-be-processed data according to the data index information. The method balances the relationship between data privacy and data utility by performing privacy calculation based on the metadata of the data and determining the output data corresponding to the data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] Humanity has entered the data age, where data permeates every industry and business function, becoming a crucial factor of production. In reality, many organizations regularly release data, such as medical, transportation, and government data. This data contains a vast amount of personal privacy information, and its leakage could lead to incalculable losses. In the field of data release, to prevent the complete public disclosure of private data, data-releasing organizations typically employ privacy protection technologies to hide users' sensitive attributes. Whether processed data will still leak privacy, the extent of privacy leaks, and the impact on data usability are key factors influencing data release. Failure to effectively measure privacy and data utility will result in a dilemma of having data but being hesitant to release it. However, not releasing data leads to low levels of data resource openness and sharing, and hinders the effective exploitation of data value. Therefore, quantifying the trade-off between data privacy and data utility is a challenge that needs to be addressed. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and storage medium that performs privacy calculations based on the data's metadata to determine the corresponding output data, thus balancing the relationship between data privacy and data utility.

[0004] In a first aspect, embodiments of this application disclose a data processing method, the method comprising:

[0005] Acquire the data to be processed and the predefined knowledge base associated with the data;

[0006] Determine the metadata of the data to be processed;

[0007] Based on a predefined knowledge base, determine the range of data indexes containing at least one preset attribute from the metadata;

[0008] Divide the data index range to obtain a data storage set corresponding to at least one preset attribute;

[0009] Generate data index information based on the data storage set;

[0010] Based on the data index information, determine the output data corresponding to the data to be processed.

[0011] Furthermore, the data index range is divided to obtain a data storage set corresponding to at least one preset attribute, including:

[0012] Determine the numerical range corresponding to at least one preset attribute;

[0013] According to predefined rules, the numerical regions corresponding to at least one preset attribute are divided into at least one storage dataset;

[0014] Based on the stored dataset, generate a data storage set corresponding to at least one preset attribute.

[0015] Furthermore, data index information is generated based on the data storage set, including:

[0016] Identify at least one target storage dataset within the storage dataset;

[0017] Generate data index information based on at least one target stored dataset.

[0018] Furthermore, based on the data index information, the output data corresponding to the data to be processed is determined, including:

[0019] Based on the data index information, perform data statistics on the data to be processed to obtain the data statistics results;

[0020] Based on the statistical results, determine the output data corresponding to the data to be processed.

[0021] Furthermore, based on the data statistics, the output data corresponding to the data to be processed is determined, including:

[0022] Obtain predefined privacy thresholds;

[0023] Anonymity index is determined based on privacy thresholds and statistical results;

[0024] If the anonymity index is less than the preset value, the entity data corresponding to the statistical results will be replaced with masked data to obtain the replaced data.

[0025] The replaced data is confirmed to be the output data corresponding to the data to be processed.

[0026] Furthermore, to determine that the replaced data is the output data corresponding to the data to be processed, the method also includes:

[0027] Based on the replaced data, determine the updated data index information;

[0028] The updated data index information is used to perform data statistics on the output data to obtain the updated data statistics results;

[0029] Based on the updated data statistics, determine the updated output data corresponding to the output data.

[0030] Furthermore, before acquiring the data to be processed and the predefined knowledge base associated with it, the method also includes:

[0031] It acquires edge data published by multiple data sources, and there are correlations between the edge data published by different data sources;

[0032] The edge data published by multiple data sources is aggregated into data to be processed.

[0033] Secondly, embodiments of this application disclose a data processing apparatus, the apparatus comprising:

[0034] The acquisition module is used to acquire the data to be processed and the predefined knowledge base associated with the data to be processed;

[0035] The metadata determination module is used to determine the metadata of the data to be processed.

[0036] The data index range module is used to determine the data index range containing at least one preset attribute from metadata based on a predefined knowledge base;

[0037] The data index range partitioning module is used to partition the data index range to obtain a data storage set corresponding to at least one preset attribute.

[0038] The data index information generation module is used to generate data index information based on the data storage set;

[0039] The output data determination module is used to determine the output data corresponding to the data to be processed based on the data index information.

[0040] In some optional implementations, the data index range partitioning module includes:

[0041] The numerical region determination unit is used to determine the numerical region corresponding to at least one preset attribute.

[0042] Numerical region partitioning unit, used to divide the numerical region corresponding to at least one preset attribute into at least one storage dataset according to predefined rules;

[0043] The data storage set generation unit is used to generate a data storage set corresponding to at least one preset attribute based on the stored dataset.

[0044] In some optional implementations, the data index information generation module includes:

[0045] A target storage dataset determination unit is used to determine at least one target storage dataset from the storage dataset;

[0046] The data index information generation unit is used to generate data index information based on at least one target storage dataset.

[0047] In some optional implementations, the output data determination module includes:

[0048] The data statistics result determination unit is used to perform data statistics on the data to be processed based on the data index information and obtain the data statistics results.

[0049] The output data determination unit is used to determine the output data corresponding to the data to be processed based on the data statistical results.

[0050] In some optional implementations, the output data determination unit includes:

[0051] The privacy threshold acquisition subunit is used to acquire predefined privacy thresholds;

[0052] Anonymity index determination unit, used to determine the anonymity index based on privacy thresholds and statistical results;

[0053] The data replacement subunit is used to replace the entity data corresponding to the statistical results with masked data when the anonymity index is less than a preset value, so as to obtain the replaced data.

[0054] The output data determination sub-unit is used to determine that the replaced data is the output data corresponding to the data to be processed.

[0055] In some alternative embodiments, the device further includes:

[0056] The data index information update module is used to determine the updated data index information based on the replaced data;

[0057] The update data statistics result determination module is used to perform data statistics on the output data based on the update data index information to obtain the update data statistics results.

[0058] The update output data determination module is used to determine the updated output data corresponding to the output data based on the statistical results of the update data.

[0059] In some alternative embodiments, the device further includes:

[0060] The edge data acquisition module is used to acquire edge data published by multiple data sources, and there are correlations between the edge data published by different data sources;

[0061] The data aggregation module is used to aggregate edge data published by multiple data sources into data to be processed.

[0062] Thirdly, embodiments of this application disclose an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, and the processor loads and executes the data processing method described above.

[0063] Fourthly, embodiments of this application disclose a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the data processing method described above.

[0064] The technical solution provided in this application has the following technical effects:

[0065] This data processing method determines the index range of the data by processing its metadata, and then processes the data based on the index information to determine the output data. This scheme uses differentiated privacy to generate relevant statistical data without relying on specific detailed data, and presents the data in a non-privacy manner, ensuring that the results of a given query are not affected by the presence or absence of a single record. Furthermore, differentiated privacy reduces the attractiveness of the data to potential attackers and helps prevent them from connecting personal data from multiple platforms. Attached Figure Description

[0066] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of an application environment for a data processing method provided in an embodiment of this application;

[0068] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0069] Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0070] Figure 4 This is a hardware structure block diagram of a server for a data processing method provided in an embodiment of this application. Detailed Implementation

[0071] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0072] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0073] To make the objectives, technical solutions, and advantages disclosed in the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application.

[0074] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0075] In this era of globalization, organizations may need to release relevant data to business partners to remain competitive. This puts personal privacy at risk. Meanwhile, people's reliance on smartphones and tablets today is immense. These devices contain vast amounts of personal information due to their ability to perform numerous operations. People access their bank accounts, make purchases, and pay bills, making such devices potential targets for cybercriminals. Behavioral factors also increase privacy risks. People don't always know who is tracking them and how the tracked information is being used. To overcome privacy risks, traditionally, attributes that clearly identify individuals, such as names, Social Security numbers, and driver's license numbers, can be removed or suppressed using random values. However, these suppressed personal attributes can be combined and identified when certain values ​​or combinations thereof in the data might be associated with external knowledge. Therefore, privacy threats must be considered when data from distributed sources is linked.

[0076] In view of this, embodiments of this application provide a data processing method that uses data published by multiple data sources to form data to be processed, performs privacy calculations on the data to be processed, and determines the output data to ensure that the output data does not leak any privacy attributes.

[0077] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment for a data processing method provided in an embodiment of this application, such as... Figure 1 As shown, the application environment may include a data processing server.

[0078] In this embodiment, the data processing server can connect to one or more data sources to obtain data published by these data sources. The data processing server can perform privacy-preserving computations on the data published by the data sources to obtain corresponding output data. Optionally, the data processing server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0079] The following describes a specific embodiment of one method of this application. Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server products, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the data processing method may include:

[0080] S201: Obtain the data to be processed and the predefined knowledge base associated with the data to be processed.

[0081] In this embodiment, the data to be processed may include data published by a single data source or edge data published by multiple data sources. Edge data refers to data published by a single data source that poses no risk of privacy leakage, but data involving privacy can be obtained by aggregating this data using data aggregation technology and then mining the aggregated data. As an optional implementation, the data to be processed includes edge data published by multiple data sources. The data processing server obtains edge data published by multiple data sources, and there are correlations between the edge data published by different data sources. Then, the edge data published by multiple data sources is aggregated into the data to be processed. The correlation between edge data can be that there are one or more data corresponding to the same attribute in different edge data.

[0082] In this embodiment, the predefined knowledge base serves as a standard for data anonymization, used to identify data attributes in the data to be processed that require anonymization. The predefined knowledge base can be determined based on prior knowledge in a specific domain. For example, in the medical field, data such as name, age, and postal code in medical data involve personally identifiable information and are data attributes that require anonymization. Therefore, the predefined knowledge base can be used to identify these data attributes.

[0083] S203: Determine the metadata of the data to be processed.

[0084] In this embodiment, metadata refers to data that describes the data. After obtaining the data to be processed, information can be extracted from the data to be processed to obtain the metadata of the data to be processed.

[0085] S205: Based on a predefined knowledge base, determine the range of data indexes containing at least one preset attribute from the metadata.

[0086] In this embodiment, one or more preset attributes are identified from metadata based on a predefined knowledge base, and a data search space, i.e., the data index range, is constructed based on these preset attributes. For example, preset attributes may include tax payments for the current fiscal year, heart rate, location information, name, address, social media accounts, bank account numbers, passport information, healthcare-related information, credit and debit card numbers, driver's license and provincial ID card information, medical insurance information, student information, etc. Data containing such preset attributes, if disseminated in its raw form, may lead to data privacy breaches. The ways in which such preset attributes are described in metadata include, but are not limited to, one or more of binary characters, categorical descriptions, numbers, and descriptive text.

[0087] It should be noted that preset attributes refer to data attributes that may involve personally identifiable information or privacy data. Privacy data includes not only individual privacy data in the general sense, but also other data that the data publisher does not wish to be accessed by others.

[0088] S207: Divide the data index range to obtain a data storage set corresponding to at least one preset attribute.

[0089] In this embodiment, the data index range can be divided into one or more data storage sets, each data storage set including one or more stored datasets. Optionally, each data storage set corresponds to one or more preset attributes.

[0090] As an optional implementation, dividing the data index range to obtain a data storage set corresponding to at least one preset attribute may include: a data processing server determining the numerical region corresponding to each of the at least one preset attribute, then dividing the numerical region corresponding to each of the at least one preset attribute into at least one storage dataset according to predefined rules, and finally generating a data storage set corresponding to the at least one preset attribute based on the storage dataset. The numerical region can be a predefined numerical range or one of predefined upper and lower limits. Optionally, the numerical range or upper and lower limits are determined based on the preset attribute. That is, the data processing server determines the numerical region corresponding to each preset attribute, divides the numerical region to obtain one or more storage datasets, and then uses the storage dataset corresponding to each preset attribute as a data storage set. The division of the numerical region can be an average division or a division according to the data distribution density, etc. In some embodiments, the preset attribute and the data storage set do not necessarily have to correspond strictly one-to-one; multiple preset attributes can correspond to one data storage set.

[0091] As an example, suppose there is a dataset to be processed containing three data attributes: age, gender, and regional administrative code. When the default attribute is age, with a numerical range of 0-60, this range can be divided into six stored datasets: [0-10], [10-20], [20-30], [30-40], [40-50], and [50-60]. These six stored datasets constitute the age data storage set. As another example, when the attribute is gender, its numerical range can be 0 and 1 (assuming 0 represents female and 1 represents male). This range can then be divided into two stored datasets: 0 and 1. These two stored datasets constitute the gender data storage set. As another example, when the preset attribute is a postal code, and its numerical range is 0-999999, this numerical range can be divided into five storage datasets: [0-199999], [200000-399999], [400000-599999], [600000-799999], and [800000-999999]. These five storage datasets constitute the postal code data storage set.

[0092] In this embodiment, each data entity can be represented using a data index format. For example, in the above example, the age data storage set includes six stored datasets, which can be represented as r1age, r2age, r3age, r4age, r5age, and r6age, respectively. Similarly, the gender data storage set includes two stored datasets, which can be represented as r1gender and r2gender, where r1gender represents female and r2gender represents male. The postal code data storage set includes five stored datasets, which can be represented as r1zipcode, r2zipcode, r3zipcode, r4zipcode, and r5zipcode, respectively. For a data entity whose age is between 20 and 30 years old, whose gender is female, and whose postal code is between 600000 and 799999, its data index format can be represented as 3, 1, and 4. Representing data entities using an index format allows us to determine the stored dataset in which each data entity belongs, and also makes it easy to determine the number of data entities in each stored dataset.

[0093] S209: Generate data index information based on the data storage set.

[0094] In this embodiment, data index information can be generated based on the data storage set. The data index information is used to determine which storage dataset a data entity belongs to and how many data entities are included in a storage dataset. Optionally, the data index information can be an adversary model. The adversary model is used to mine the data to be processed, thereby determining which data entities are at risk of leakage. As an optional implementation, the data processing server determines at least one target storage dataset in the storage dataset and then generates data index information based on at least one target storage dataset. Specifically, the data processing server randomly selects one data storage set from all data storage sets as the target data storage set, then randomly selects one or more storage datasets from that target data storage set as the target storage dataset, and then uses the target storage dataset as the data index information to index the data to be processed, thereby determining the number of data entities in the target storage dataset. As an example, for the storage dataset in the above example, the data index information can be an age of [20-30], or a gender of male, or a postal code of [600000-799999], etc. Of course, the data storage server can also randomly select two or more data storage sets as target data storage sets from all data storage sets, and then randomly select one or more storage datasets from each of these two or more target data storage sets as target storage datasets, and then use these target storage datasets as data index information. For two or more target data storage sets, one or more storage datasets can be randomly selected from each target data storage set as target storage datasets. As an example, for the storage datasets in the above example, the data index information could be an age of [20-30], a gender of male, and a postal code of [600000-799999], etc.

[0095] S211: Determine the output data corresponding to the data to be processed based on the data index information.

[0096] In this embodiment of the application, the risk of potential data entity leakage is determined based on data index information, and the potentially leaked data is processed to obtain the output data corresponding to the data to be processed.

[0097] As an optional implementation, the data processing server performs data statistics on the data to be processed based on the data index information, obtains the statistical results, and then determines the output data corresponding to the data to be processed based on the statistical results. When determining the output data corresponding to the data to be processed, the data processing server needs to obtain a predefined privacy threshold, and then determine an anonymity index based on the privacy threshold and the statistical results. If the anonymity index is less than a preset value, the entity data corresponding to the statistical results is replaced with masked data to obtain the replaced data. The data processing server then determines the replaced data as the output data corresponding to the data to be processed. Specifically, the data information determined by the data processing server based on the data index information is the statistical information of the data entities, that is, the number of entities falling within the data index information. If the number of data entities indexed by the data index information is large, the risk of leakage for each data entity is smaller. Conversely, if the number of data entities indexed by the data index information is large, the risk of leakage for each data entity is greater. The risk of data entity leakage can be characterized using an anonymity index, calculated as: a = C / V, where a is the anonymity index, C is the number of data entities, and V is the privacy threshold. Since different data attributes have different anonymity requirements, the privacy threshold can be set according to different preset attributes or combinations of different preset attributes. After calculating the anonymity index of the data entities indexed by the data index information, it is determined whether the anonymity index is less than a preset value. Optionally, the preset value can be 1 or other values, which can be set according to actual needs. When the anonymity index is less than 1, it indicates that the data entities indexed by the data index information are at risk of leakage, so these data need to be anonymized. Optionally, the anonymization of the data entities indexed by the data index information can be performed by replacing these data entities with predefined characters. Specifically, the data entities indexed by the data index information can be treated as a separate data processing set, and then a masked data set can be generated using predefined characters or strings. The masked data set is then used to replace the data processing set, thereby obtaining the replaced data. The data processing server determines that the replaced data is the output data corresponding to the data to be processed. Optionally, the predefined characters include, but are not limited to, one or more combinations of numbers, letters, symbols, text, or illustrative characters. In some implementations, predefined characters can be used to directly replace the data entities indexed by the data index information one by one, thereby obtaining the replaced data. The data processing server then determines that the replaced data is the output data corresponding to the data to be processed.

[0098] As an example, suppose the data index information is as follows: age between 20 and 30 years old, gender female, and postal code between 600000 and 799999. All data entities are plotted within a single data index range, encompassing three data storage sets: age, gender, and postal code. Then, the data entities are indexed within this search range based on the data index information. If the data index identifies multiple data entities, since each entity has an equal probability of appearing, the index cannot identify a specific person. If the index identifies only a few data entities, such as one, then that entity poses a significant risk of disclosure. In this case, a predefined character needs to be used to replace the data entity to protect its privacy.

[0099] It should be noted that anonymizing only the data entities indexed by a single data index is clearly insufficient to fully guarantee the privacy and security of the data to be processed. Therefore, multiple data indexes are needed to index the data to be processed, identify the data entities at risk of leakage, and then replace these data entities one by one. This ensures the privacy and security of the final output data corresponding to the data to be processed.

[0100] In this embodiment of the application, after determining that the replaced data is the output data corresponding to the data to be processed, the data processing server determines the updated data index information based on the replaced data, then performs data statistics on the output data based on the updated data index information to obtain the updated data statistics result, and finally determines the updated output data corresponding to the output data based on the updated data statistics result.

[0101] As an optional implementation, the data processing server can determine the updated data index information by generating multiple data indexes at once, then selecting one of these indexes for data analysis of the data to be processed, while using the remaining indexes as updated indexes to analyze the replaced data. In other words, the generated indexes are analyzed one by one for each data entity to be processed. If the anonymity index of a data entity indexed by a particular index is less than a preset value, masked data is used to replace these entities. Then, the next index is used to continue analyzing the replaced data until the number of data entities analyzed by all indexes meets the anonymization requirements.

[0102] As an alternative implementation, the data processing server determines the updated data index information by: for the replaced data corresponding to the data to be processed, the data processing server re-extracts metadata, then redetermines the data index range containing at least one preset attribute based on the re-extracted metadata, and divides the redetermined data index range to obtain a data storage set corresponding to at least one preset attribute. Then, data index information is generated for the redetermined data storage set as updated data index information, and data statistics are performed on the replaced data based on this updated data index information. Similarly, for the number of data entities indexed by the updated data index information, their anonymity index is calculated to determine whether these data entities meet the data anonymization requirements, i.e., whether the anonymity index of these data entities is less than a preset value. If it is less, it indicates that these data have a risk of privacy leakage and need to be anonymized, i.e., data replacement, to obtain the replaced data. The data processing server then determines that the replaced data is the output data corresponding to the data to be processed. This process continues until the final output data corresponding to the data to be processed, where the number of data entities indexed by any data index information meets the data anonymization requirements.

[0103] In this embodiment, after obtaining the output data corresponding to the data to be processed, the data processing server can further perform purification processing on the anonymized data. Optionally, purification techniques for the anonymized data include, but are not limited to, privacy-level masking techniques, privacy-based data analysis techniques, and clustering techniques. Furthermore, data mining techniques can also be used to process the output data corresponding to the data to be processed, so that after the processed data is divided into data storage sets, each data storage set has a balanced entity count, thus obtaining a balanced data set. Optionally, after obtaining the balanced data set, the data processing server can also calculate a certainty index based on the midpoint of the balanced data set and the privacy data, and calculate the changes in preset attributes corresponding to the two sets based on the amount of change between the data storage set before balancing and the balanced data set.

[0104] In this embodiment, the output data corresponding to the data to be processed obtained by the data processing server can be further processed to meet the requirements of one or more data buyers. Furthermore, a data evaluation report can be generated for the output data to assess its privacy, security, and utility.

[0105] The data processing method described in this application allows for the deployment of related services at both central and edge nodes. After data access, statistical data is generated through statistical analysis rather than detailed data, and specific data is presented in a non-privacy manner to ensure that the results of a given query are not affected by the presence or absence of a single record. Furthermore, differential privacy reduces the attractiveness of the data to potential attackers and helps prevent them from connecting personal data from multiple platforms.

[0106] This application also provides a data processing apparatus. Figure 3 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application, as shown below. Figure 3 As shown, the device includes:

[0107] The acquisition module 301 is used to acquire the data to be processed and the predefined knowledge base associated with the data to be processed;

[0108] Metadata determination module 303 is used to determine the metadata of the data to be processed;

[0109] The data index range module 305 is used to determine the data index range containing at least one preset attribute from the metadata based on a predefined knowledge base;

[0110] The data index range division module 307 is used to divide the data index range to obtain a data storage set corresponding to at least one preset attribute.

[0111] The data index information generation module 309 is used to generate data index information based on the data storage set;

[0112] The output data determination module 311 is used to determine the output data corresponding to the data to be processed based on the data index information.

[0113] In some optional implementations, the data index range partitioning module includes:

[0114] The numerical region determination unit is used to determine the numerical region corresponding to at least one preset attribute.

[0115] Numerical region partitioning unit, used to divide the numerical region corresponding to at least one preset attribute into at least one storage dataset according to predefined rules;

[0116] The data storage set generation unit is used to generate a data storage set corresponding to at least one preset attribute based on the stored dataset.

[0117] In some optional implementations, the data index information generation module includes:

[0118] A target storage dataset determination unit is used to determine at least one target storage dataset from the storage dataset;

[0119] The data index information generation unit is used to generate data index information based on at least one target storage dataset.

[0120] In some optional implementations, the output data determination module includes:

[0121] The data statistics result determination unit is used to perform data statistics on the data to be processed based on the data index information and obtain the data statistics results.

[0122] The output data determination unit is used to determine the output data corresponding to the data to be processed based on the data statistical results.

[0123] In some optional implementations, the output data determination unit includes:

[0124] The privacy threshold acquisition subunit is used to acquire predefined privacy thresholds;

[0125] Anonymity index determination unit, used to determine the anonymity index based on privacy thresholds and statistical results;

[0126] The data replacement subunit is used to replace the entity data corresponding to the statistical results with masked data when the anonymity index is less than a preset value, so as to obtain the replaced data.

[0127] The output data determination sub-unit is used to determine that the replaced data is the output data corresponding to the data to be processed.

[0128] In some alternative embodiments, the device further includes:

[0129] The data index information update module is used to determine the updated data index information based on the replaced data;

[0130] The update data statistics result determination module is used to perform data statistics on the output data based on the update data index information to obtain the update data statistics results.

[0131] The update output data determination module is used to determine the updated output data corresponding to the output data based on the statistical results of the update data.

[0132] In some alternative embodiments, the device further includes:

[0133] The edge data acquisition module is used to acquire edge data published by multiple data sources, and there are correlations between the edge data published by different data sources;

[0134] The data aggregation module is used to aggregate edge data published by multiple data sources into data to be processed.

[0135] The data processing apparatus and data processing method embodiments described in this application are based on the same application concept. For specific implementation details of the data processing apparatus, please refer to all the above-described implementation methods of the data processing method.

[0136] This application also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The processor loads the at least one instruction or at least one program and executes the data processing method described above.

[0137] The data processing method embodiments provided in this application can be executed on a mobile terminal, computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 4 This is a hardware structure block diagram of a server for a data processing method provided in an embodiment of this application. For example... Figure 4 As shown, the server 400 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 410 (CPUs 410 may include, but are not limited to, microprocessors (MCUs) or field-programmable gate arrays (FPGAs), etc.), a memory 430 for storing data, and one or more storage media 420 (e.g., one or more mass storage devices) for storing application programs 423 or data 422. The memory 430 and storage media 420 may be temporary or persistent storage. The program stored in the storage media 420 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 410 may be configured to communicate with the storage media 420 and execute the series of instruction operations stored in the storage media 420 on the server 400. Server 400 may also include one or more power supplies 460, one or more wired or wireless network interfaces 450, one or more input / output interfaces 440, and / or one or more operating systems 421, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0138] The input / output interface 440 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 400. In one example, the input / output interface 440 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 440 may be a radio frequency (RF) module for wireless communication with the Internet.

[0139] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 400 may also include... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.

[0140] This application also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the data processing method described above.

[0141] In this embodiment, the computer storage medium can be located in at least one of multiple network servers in a computer network. Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0142] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0144] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0145] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: The process involves acquiring edge data published by multiple data sources, with correlations existing between the edge data published by different data sources; aggregating the edge data published by the multiple data sources into data to be processed; the edge data refers to data published by a single data source that poses no risk of privacy leakage, but data involving privacy can be obtained through mining the aggregated data; the correlation between the edge data refers to the existence of one or more data with the same attribute in different edge data sets; Acquire the data to be processed and a predefined knowledge base associated with the data to be processed; Determine the metadata of the data to be processed; Based on the predefined knowledge base, determine the data index range containing at least one preset attribute from the metadata; The data index range is divided to obtain a data storage set corresponding to the at least one preset attribute; Multiple data index information are generated based on the data storage set; Based on any one of the multiple data index information, determine the output data corresponding to the data to be processed; Based on the output data corresponding to the data to be processed, update data index information is determined, wherein the update data index information is the remaining data index information among the plurality of data index information except for any one of the data index information; data statistics are performed on the output data based on the update data index information to obtain update data statistics results; based on the update data statistics results, update output data corresponding to the output data is determined.

2. The method according to claim 1, characterized in that, The step of dividing the data index range to obtain the data storage set corresponding to the at least one preset attribute includes: Determine the numerical range corresponding to each of the at least one preset attribute; According to predefined rules, the numerical regions corresponding to each of the at least one preset attribute are divided into at least one storage dataset; Based on the stored dataset, generate the data storage set corresponding to the at least one preset attribute.

3. The method according to claim 2, characterized in that, The step of generating data index information based on the data storage set includes: At least one target storage dataset is identified in the storage dataset; The data index information is generated based on the at least one target storage dataset.

4. The method according to claim 1, characterized in that, The step of determining the output data corresponding to the data to be processed based on the data index information includes: Based on the data index information, perform data statistics on the data to be processed to obtain data statistics results; Based on the statistical results of the data, the output data corresponding to the data to be processed is determined.

5. The method according to claim 4, characterized in that, The step of determining the output data corresponding to the data to be processed based on the data statistics results includes: Obtain predefined privacy thresholds; An anonymity index is determined based on the privacy threshold and the statistical results; If the anonymity index is less than a preset value, the entity data corresponding to the statistical results is replaced with masked data to obtain the replaced data; The replaced data is determined to be the output data corresponding to the data to be processed.

6. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire edge data published by multiple data sources, and there are correlations between the edge data published by different data sources; to aggregate the edge data published by the multiple data sources into data to be processed; the edge data refers to data published by a single data source that has no risk of privacy leakage, but data involving privacy can be obtained by mining the aggregated data; the correlation between the edge data refers to the existence of one or more data with the same attribute in different edge data; and to acquire the data to be processed and a predefined knowledge base associated with the data to be processed. Metadata determination module, used to determine the metadata of the data to be processed; The data index range module is used to determine, based on the predefined knowledge base, a data index range containing at least one preset attribute from the metadata; A data index range division module is used to divide the data index range to obtain a data storage set corresponding to the at least one preset attribute; A data index information generation module is used to generate multiple data index information based on the data storage set; The output data determination module is configured to: determine the output data corresponding to the data to be processed based on any one of the plurality of data index information; determine updated data index information based on the output data corresponding to the data to be processed, wherein the updated data index information is the remaining data index information among the plurality of data index information excluding any one of the data index information; perform data statistics on the output data based on the updated data index information to obtain updated data statistics results; and determine the updated output data corresponding to the output data based on the updated data statistics results.

7. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded by the processor and executed as the data processing method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1-5.

Images