Data processing method and device, electronic equipment, chip and medium
By grading sensitive levels, noise addition and clustering of user data, the problem of user data privacy protection is solved, efficient data protection and privacy protection is achieved, and resource requirements and maintenance costs are reduced.
Patent Information
- Application Number
- CN202411611502.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2025-03-07
Smart Images

Figure CN119577795A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of privacy protection technology, and in particular to a data processing method, device, electronic device, chip, and medium. Background Art
[0002] When users use network services through terminals, the privacy and security issues involved lie in the risk of data leakage. If the personal information provided by users is accessed by unauthorized third parties, it may cause a series of security problems. If data security is not fully considered when designing applications and services, or there are loopholes in actual operation, user information will be easily attacked by hackers or stolen through malicious software, resulting in the leakage of user privacy data. Summary of the Invention
[0003] The present disclosure provides a data processing method, device, electronic device, chip and medium to protect user data.
[0004] The first aspect embodiment of the present disclosure proposes a data processing method, which includes: obtaining an initial data set of multiple nodes, the initial data set including user data of multiple users, and the user data including initial data of multiple preset dimensions; performing noise processing on the initial data according to the sensitivity level of the initial data to obtain a first data set, the first data set including the first data of multiple users; integrating the first data in the first data set according to the preset dimensions to obtain an integrated data matrix of different preset dimensions; clustering the integrated data matrices of different preset dimensions through hierarchical clustering and Ward criterion to obtain a cluster mapping relationship.
[0005] In some embodiments of the present disclosure, performing noise processing on the initial data according to the sensitivity level of the initial data includes: determining the sensitivity level of the initial data based on the correspondence between the dimension and the sensitivity level; determining the noise level of the Laplace noise corresponding to the initial data according to the sensitivity level of the initial data; generating Laplace noise based on the noise level, and adding the Laplace noise to the initial data.
[0006] In some embodiments of the present disclosure, determining the noise level of the Laplace noise corresponding to the initial data according to the sensitivity level of the initial data includes: determining the noise level corresponding to the initial data with a high sensitivity level as a first value; determining the noise level corresponding to the initial data with a medium sensitivity level as a second value, the first value being greater than the second value; and determining the noise level corresponding to the initial data with a low sensitivity level as a third value, the third value being less than the second value.
[0007] In some embodiments of the present disclosure, integrating the first data in the first data set according to a preset dimension includes: performing data cleaning and standardization on the first data in the first data set to obtain a second data set; performing dimension separation on the data in the second data set according to the preset dimension; and performing data encoding on the second data of different preset dimensions according to preset rules to obtain integrated data matrices of different preset dimensions, where the integrated data matrices of different preset dimensions correspond to integrated data vectors of multiple users.
[0008] In some embodiments of the present disclosure, clustering processing is performed on integrated data matrices of different preset dimensions through hierarchical clustering and Ward criterion, including: taking the integrated data vector corresponding to each user as the initial cluster center, performing hierarchical clustering to obtain an initial cluster set corresponding to each initial cluster center; taking every two initial cluster centers as cluster pairs, and merging the cluster pairs based on the Ward criterion to obtain a cluster mapping relationship.
[0009] In some embodiments of the present disclosure, merging cluster pairs based on the Ward criterion includes: determining the Euclidean distance of each cluster pair; merging the cluster pairs based on the Euclidean distance and a preset number of clusters until the preset number of clusters is reached, and obtaining a cluster mapping relationship, where the cluster mapping relationship is an integrated data vector corresponding to the identifier of each cluster and the user in the corresponding cluster.
[0010] The second aspect embodiment of the present disclosure proposes a data processing device, which includes: an acquisition module for acquiring an initial data set of multiple nodes, the initial data set including user data of multiple users, and the user data including initial data of multiple preset dimensions; a noise addition module for performing noise addition processing on the initial data according to the sensitivity level of the initial data to obtain a first data set, the first data set including the first data of multiple users; an integration module for integrating the first data in the first data set according to the preset dimension to obtain an integrated data matrix of different preset dimensions; a clustering module for performing clustering processing on the integrated data matrices of different preset dimensions through hierarchical clustering and Ward criterion to obtain a cluster mapping relationship.
[0011] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: one or more processors and a storage device communicatively connected to the one or more processors, wherein, when the one or more programs are executed by the one or more processors, the one or more processors execute the method described in the first aspect embodiment of the present disclosure.
[0012] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.
[0013] The fifth embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the embodiment of the first aspect of the present disclosure.
[0014] In summary, according to the data processing method proposed in the present disclosure, an initial data set of multiple nodes is obtained, the initial data set includes user data of multiple users, and the user data includes initial data of multiple preset dimensions; according to the sensitivity level of the initial data, the initial data is noise-processed to obtain a first data set, and the first data set includes the first data of multiple users; according to the preset dimensions, the first data in the first data set is integrated to obtain an integrated data matrix of different preset dimensions; through hierarchical clustering and Ward criterion, the integrated data matrices of different preset dimensions are clustered to obtain a cluster mapping relationship. The data processing method disclosed in the present disclosure can take into account the sensitivity of different data dimensions, determine the noise level of noise according to different sensitivities, perform noise processing on the data, and realize privacy protection of user data. At the same time, through integration and clustering processing, the inherent structure of the data is revealed to further protect user data.
[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.
[0017] Figure 1 A flowchart of a data processing method proposed in an embodiment of the present disclosure;
[0018] Figure 2 A flow chart of a method for performing noise processing on initial data proposed in an embodiment of the present disclosure;
[0019] Figure 3 A flow chart of a method for integrating a first data set proposed in an embodiment of the present disclosure;
[0020] Figure 4 A flow chart of the clustering processing method proposed in an embodiment of the present disclosure;
[0021] Figure 5 A flow chart of a method for merging cluster pairs proposed in an embodiment of the present disclosure;
[0022] Figure 6 is a flow chart of the data processing method;
[0023] Figure 7 A schematic structural diagram of a data processing device provided in an embodiment of the present disclosure;
[0024] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] The following describes the embodiments of the present disclosure in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions.
[0026] When users access services, they are often required to provide personal information such as name, address, phone number, and bank account information. Access to this information by unauthorized third parties can lead to a range of security issues, including identity theft, fraud, and even financial loss. Many applications and services may not adequately consider data security during design or may contain vulnerabilities in their operations, making user information vulnerable to hackers or theft via malware. Furthermore, some service providers may fail to adhere to best data protection practices, such as data encryption, secure data storage, and access control, increasing the risk of data breaches. Currently, data is encrypted primarily through complex encryption algorithms, which are regularly updated to protect user privacy. However, complex encryption algorithms usually require more computing resources, which may cause applications to run slowly, especially on mobile devices with limited resources. This performance impact may lead to a decline in user experience, especially in services that need to process large amounts of data in real time. Secondly, encryption algorithms need to be updated regularly to defend against emerging threats, which will result in high technology update costs. In addition, the real-time complexity of encryption algorithms may lead to configuration errors or improper implementation, thereby introducing new security vulnerabilities. Although encryption technology can protect the security of data during transmission and storage, if the user's device itself fails to take appropriate security measures, it will also put the data at risk of being stolen.
[0027] To address the issues existing in related technologies, this paper proposes a data processing method that adds Laplace noise based on the sensitivity of the data. The noisy data is then aggregated and clustered according to different characteristics to create a unified data view, enabling efficient operation of subsequent data analysis and machine learning scenarios. Compared to related technologies, the method proposed in this paper can add noise based on data characteristics and utilize differential privacy to ensure that personal data remains anonymous even in the event of a data leak. Aggregation and clustering reveal the inherent structure of the data, further protecting user privacy.
[0028] The data processing method provided in this application is described in detail below with reference to the accompanying drawings.
[0029] Figure 1 Flowchart of a data processing method provided by an embodiment of the present disclosure. Figure 1 As shown, the data processing method includes steps 101-104.
[0030] Step 101: Acquire an initial data set of multiple nodes.
[0031] In an embodiment of the present disclosure, the initial data set includes user data of multiple users, and the user data includes initial data of multiple preset dimensions.
[0032] In an embodiment of the present disclosure, obtaining the initial data set of multiple nodes may be collecting user data of users from multiple client nodes, where the user data includes initial data of multiple preset dimensions.
[0033] In an embodiment of the present disclosure, obtaining the initial data set of multiple nodes may be obtaining the initial data set of multiple nodes in a specific range based on a set specific range. In other words, the initial data set of multiple nodes in the specific range is collected, wherein the specific range may be a specific area or a specific group, and each node is responsible for collecting data of a specific area or a specific group.
[0034] In the embodiments of the present disclosure, collecting the initial data set of multiple nodes according to the set specific range can achieve data correlation, that is, limiting the range helps to collect data related to specific scenarios or needs, avoid mixing irrelevant data, and reduce interference; at the same time, for different regions or groups, setting a specific range helps to more accurately evaluate the sensitivity of the data, thereby more reasonably setting the noise level; in addition, setting a specific range can reduce the computing resource requirements for data processing, especially in a distributed environment, and help allocate resources to key areas.
[0035] In an embodiment of the present disclosure, multiple preset dimensions can be set according to needs. The preset dimensions can be dimensions such as geographic location, device type, interaction method, browsing history, communication records, transaction information, etc., or other preset dimensions, which are not limited by the present disclosure.
[0036] In the embodiments of the present disclosure, the geographic location refers to the user's real geographic location, such as GPS coordinates. Each user data generally includes a geographic location, and the geographic location can change dynamically, especially for mobile device users.
[0037] In the embodiments of the present disclosure, the device type refers to the type of device used by the user, such as a smartphone, tablet, laptop, etc. The same user may use multiple devices at the same time, so the device type data can be single or multiple, for example, the same user corresponds to different device types at different time periods.
[0038] In the embodiments of the present disclosure, the interaction mode refers to the user's personal behavior patterns and preferences, such as the way of using the device (clicking, sliding, typing, etc.). The interaction mode may be dynamic and change with time and application context. Therefore, a user may have different interaction mode data in different contexts.
[0039] In the embodiments of the present disclosure, browsing history refers to a user's web browsing records, communication records refer to a user's call records, message records, etc., and transaction information refers to a user's purchase or financial transaction records.
[0040] For example, based on a set range, user data on each server node within the range is collected. The user data includes data on three main characteristics: geographic location, device type, and interaction method.
[0041] Step 102: performing noise processing on the initial data according to the sensitivity level of the initial data to obtain a first data set.
[0042] In an embodiment of the present disclosure, the first data set includes first data of multiple users.
[0043] In an embodiment of the present disclosure, the initial data is subjected to noise processing according to the sensitivity level of the initial data. The sensitivity level of the initial data may be determined according to the preset dimension to which the initial data belongs and the sensitivity level of the preset dimension, thereby performing noise processing on the initial data of different sensitivity levels.
[0044] In an embodiment of the present disclosure, the sensitivity level of the initial data is determined according to the sensitivity level of the preset dimension. Evaluation rules may be used to determine the sensitivity levels of different preset dimensions, thereby obtaining the sensitivity level of the initial data.
[0045] In an embodiment of the present disclosure, the initial data is subjected to noise processing according to the sensitivity level of the initial data. The initial data may be subjected to a multi-dimensional comprehensive evaluation using a trained evaluation model according to the type of the initial data and the characteristics of the region or group in which it is located to determine the sensitivity level of the initial data, wherein the evaluation model may be trained using training data.
[0046] In an embodiment of the present disclosure, the noise addition process may be to determine the type of noise to be added and the parameters of the noise according to the sensitivity level of the initial data, and to determine the method of noise addition according to the dimension to which the initial data belongs, thereby adding noise to the initial data.
[0047] In the embodiments of the present disclosure, the noise added corresponding to the noise adding process may be Laplace noise, Gaussian noise, uniformly distributed noise, etc., which is not limited in the present disclosure.
[0048] In an embodiment of the present disclosure, the first data of each user in the first data set are all initial data after noise addition, and each first data corresponds to a preset dimension.
[0049] In the above embodiment, performing noise processing on the initial data according to the sensitivity level of the initial data can achieve differentiated noise addition, thereby achieving a better balance between privacy protection and data availability.
[0050] Step 103 : Integrate the first data in the first data set according to preset dimensions to obtain integrated data matrices of different preset dimensions.
[0051] In an embodiment of the present disclosure, the first data in the first data set is integrated according to a preset dimension, and the first data in the first data set may be classified and aggregated according to the preset dimension to obtain an integrated data matrix of multiple preset dimensions.
[0052] In an embodiment of the present disclosure, integrating the first data in the first data set can organize different types of data according to dimensions, providing a consistent and structured data view for subsequent data processing steps, and further, preparing input data for machine learning or data mining algorithms.
[0053] For example, user data of each node after adding noise data is integrated according to different preset dimensions to obtain integrated data, and the integrated data includes data matrices of different preset dimensions.
[0054] In the above embodiment, the first data of different preset dimensions are integrated into a matrix, thereby obtaining a multi-dimensional structure of the data.
[0055] Step 104 : clustering the integrated data matrices of different preset dimensions by using hierarchical clustering and Ward criterion to obtain cluster mapping relationships.
[0056] In the embodiments of the present disclosure, hierarchical clustering is a data clustering algorithm that analyzes the similarities and differences of a dataset by constructing a cluster tree. This involves decomposing the dataset hierarchically according to a certain method until certain conditions are met. Hierarchical clustering includes agglomerative and divisive types. Agglomerative hierarchical clustering is a bottom-up strategy that includes initialization and merging, while divisive hierarchical clustering uses a top-down strategy that includes initialization and splitting. Different hierarchical clustering methods may be used in different embodiments, and this disclosure does not limit this.
[0057] In the embodiments of the present disclosure, the Ward criterion, also known as the Ward equivalence principle or the sum of squared deviations method, is a clustering analysis method based on the minimum variance and minimum squared difference criteria, which can minimize the increase in total cohesive distance caused by merging, so that the clustering process tends to create clusters of similar size, thereby preventing a few large clusters from dominating the entire data set.
[0058] In an embodiment of the present disclosure, hierarchical clustering and Ward criterion are used to cluster the integrated data matrices of different preset dimensions. The integrated data of different users in the integrated data matrices of different preset dimensions can be clustered and merged to obtain a clustering result, that is, a cluster mapping relationship.
[0059] In the embodiment of the present disclosure, the cluster mapping relationship is saved for subsequent data analysis and result interpretation to ensure the repeatability of the analysis.
[0060] For example, a clustering algorithm is used to perform clustering processing on the integrated data, and cluster mapping relationship information is saved.
[0061] In summary, according to the data processing method proposed in the present disclosure, initial data of different preset dimensions can be obtained according to the set specific range, and the initial data can be denoised according to the sensitivity level of the initial data to achieve differential privacy protection of user data within a specific range, and the noisy data can be integrated and clustered to further ensure data privacy.
[0062] The data processing method proposed in the present disclosure reduces the demand for computing resources, can process large amounts of data in real time, and compared with the encryption algorithm method, reduces the need for frequent updates of the encryption algorithm, reduces maintenance costs, and also reduces the possibility of security vulnerabilities introduced due to configuration errors or improper implementation. Even if the security measures of the user device are insufficient, user data can still be protected by implementing the data processing method on the server side.
[0063] Figure 2 This is a flow chart of the method for performing noise processing on initial data proposed in this disclosure, based on Figure 1 The embodiment shown, Figure 2 Step 102 is further defined. Figure 2 As shown, the following steps are included:
[0064] Step 201: Determine the sensitivity level of the initial data based on the correspondence between dimensions and sensitivity levels.
[0065] In an embodiment of the present disclosure, the correspondence between dimensions and sensitivity levels may be a correspondence obtained through expert review, historical data analysis, or privacy impact assessment. For example, the correspondence between different preset dimensions and sensitivity levels may be determined based on historical analysis and expert review.
[0066] In an embodiment of the present disclosure, the correspondence between different preset dimensions and sensitivity levels may be determined by experts, privacy protection personnel, or a data analysis team based on historical data analysis, privacy impact assessment, and the like.
[0067] In the embodiments of the present disclosure, the correspondence between dimensions and sensitivity levels can be: for geographic location data, since geographic location data is usually more sensitive and requires a larger noise level to avoid accurate inference of user location, the sensitivity level is high sensitivity; for device type data, since device type data is sensitive but relatively stable and does not directly disclose personal location or behavior, the sensitivity level can be medium sensitivity; for interaction method data, the interaction method involves the user's behavior pattern and may reveal the user's personal habits or preferences, so the sensitivity level is high sensitivity.
[0068] In an embodiment of the present disclosure, the correspondence between dimensions and sensitivity levels may also include: browsing history, the user's web browsing history, which may expose the user's interests and behavior patterns, and the sensitivity level is high sensitivity; communication records, the user's call records, message records, etc., which have a high sensitivity level; transaction information, the user's purchase or financial transaction records, which may expose the financial situation, and the sensitivity level is high sensitivity. The correspondence may also include the sensitivity levels of other dimensions, which are not limited by the present disclosure.
[0069] For example, the initial sensitivity assessment is completed by expert review, historical data analysis or privacy impact assessment. The sensitivity levels corresponding to data characteristics (such as geographic location, device type, and interaction method) are pre-set, and the potential risks of each characteristic to user privacy are evaluated through experience and domain knowledge, and rules for different data characteristics and corresponding sensitivity levels are obtained.
[0070] In an embodiment of the present disclosure, based on the correspondence between dimensions and sensitivity levels, the sensitivity level of the initial data can be determined according to the sensitivity levels of the above-mentioned different characteristics. For initial data whose data is geographic location, the sensitivity level is high sensitivity; for initial data whose data is device type, the sensitivity level is medium sensitivity; and for initial data whose data is interaction mode, the sensitivity level is high sensitivity.
[0071] Illustratively, the system automatically classifies each data feature as high, medium, or low sensitivity based on the derived rules.
[0072] Step 202: Determine the noise level of the Laplace noise corresponding to the initial data according to the sensitivity level of the initial data.
[0073] In an embodiment of the present disclosure, determining the noise level of the Laplace noise corresponding to the initial data according to the sensitivity level of the initial data may be determining the noise level corresponding to the initial data having a high sensitivity level as a first value.
[0074] In an embodiment of the present disclosure, determining the noise level of the Laplace noise corresponding to the initial data according to the sensitivity level of the initial data can be to determine the noise level corresponding to the initial data with a medium sensitivity level as a second value, and the first value is greater than the second value.
[0075] In an embodiment of the present disclosure, determining the noise level of the Laplace noise corresponding to the initial data according to the sensitivity level of the initial data can be to determine the noise level corresponding to the initial data with a low sensitivity level as a third value, which is smaller than the second value.
[0076] In the embodiments of the present disclosure, the noise used in the noise addition process is Laplace noise, which is a two-tailed symmetric distribution, meaning that it is distributed equally on both sides of the mean. This symmetry is very important in protecting privacy because it ensures that the noise added to the data will not be biased in a certain direction, thereby reducing the risk of privacy leakage. At the same time, the Laplace distribution has a peak at the center position (0), which means that most noise values will be relatively close to the real data points, and only a few extreme noises will be generated. This ensures that most data still maintains relatively accurate characteristics after being noised, which is suitable for differential privacy scenarios, protecting privacy without seriously affecting the availability of data.
[0077] In the embodiments disclosed herein, Laplace noise is used for noise processing. While the normal distribution (i.e., Gaussian noise) is more common, it can generate larger noise values and a longer tail, which can easily produce extreme values and significantly deviate from the true data. The Laplace distribution, with its sharp peaks and rapidly decaying tails, can reduce the generation of extreme noise and control data bias. Compared to uniformly distributed noise, which randomly generates noise values within a certain interval and can result in excessive randomness, undermining the actual meaning of the data, the Laplace distribution provides a more reasonable distribution of noise values, resulting in more balanced privacy protection.
[0078] In an embodiment of the present disclosure, the noise magnitude is a parameter corresponding to the noise. Based on the sensitivity level of the initial data, the standard deviation of the Laplace noise (scale parameter, expressed as β) is set for data of different sensitivity levels. In this method, the value of β is proportional to the sensitivity level of the data and can be set using the following principles: for high-sensitivity initial data, such as geographic location and interaction method, a larger β value is selected, such as β = 1.0 or higher, to provide stronger privacy protection; for medium-sensitivity initial data, such as device type, a medium β value is selected, such as β = 0.5; for low-sensitivity initial data, a low β value is selected, such as = 0.1.
[0079] Step 203: Generate Laplace noise based on the noise level, and add the Laplace noise to the corresponding initial data.
[0080] In an embodiment of the present disclosure, Laplace noise is generated based on the noise level, which may be a first value, a second value, or a third value of the noise level determined according to different initial data.
[0081] In an embodiment of the present disclosure, a formula for generating Laplace noise is: Noise=Laplace(0,β), where Laplace(0,β) represents a Laplace distribution centered at 0 and with β as a scale parameter.
[0082] In an embodiment of the present disclosure, adding to the corresponding initial data may be adding the generated Laplace noise to the corresponding initial data.
[0083] In an embodiment of the present disclosure, the generated Laplace noise may be added to the corresponding initial data by determining a noise addition method based on a preset dimension of the initial data. In other words, the noise addition method may be different for data of different preset dimensions, for example, different noise addition methods may be used for data of geographic location, device type, or interaction method.
[0084] In an embodiment of the present disclosure, for the initial data of the geographic location, the initial data of the geographic location is represented by two coordinate values of longitude and latitude, and Laplace noise is added to the two coordinates of longitude and latitude respectively, which means that each location coordinate (longitude, latitude) will be added with an independently generated noise value, thereby disrupting the actual geographic location.
[0085] For example, if the user's geographic location is (100.1234, 50.5678), the system will add Laplace noise to the two values 100.1234 and 50.5678 respectively, so that the new coordinates (100.1234+noise 1, 50.5678+noise 2) become the user's location.
[0086] In the embodiments of the present disclosure, for the initial data of the device type, the device type is usually categorical data, indicating the type of device, such as a smartphone, tablet computer, notebook, etc. Since the device type is not numerical data, noise cannot be added directly. The noise here will be added to the encoding of the device type, or privacy protection will be achieved indirectly by disturbing the device type data. By encoding the device type, for example, encoding a smartphone as 1 and a tablet computer as 2, the generated noise is added to these codes to change the value of the device type. Since the device type data is categorical data, noise may cause a certain degree of disturbance in the device type between adjacent categories.
[0087] For example, the device type is coded as 1 (smartphone) and the noise value is 0.4. The result after adding noise can be 1 (smartphone) or 2 (tablet) with a slight disturbance, thereby disturbing the user's device information.
[0088] In the embodiments of the present disclosure, for the initial data of the interaction mode, the interaction mode involves the user's behavior pattern, such as clicking, sliding, inputting, etc. This type of data can be discrete data (such as the number of clicks, operation type) or continuous data (such as sliding distance, input time). If it is discrete data, noise can be added directly to the digital value. For example, if the number of clicks is 5, after adding noise, it may be programmed as 6 or 4; if it is categorical data, noise can be achieved through categorical perturbation. Similar to the device type, different interaction modes will be encoded, and noise perturbation will be performed on these encoded data.
[0089] For example, the user interaction mode is "click", which is encoded as 1 and the noise value is 0.2. After adding noise disturbance, it is encoded as 2, which becomes "slide".
[0090] In the above embodiment, through the correspondence between the system's preset dimensions and sensitivity levels, the sensitivity level of the initial data under different preset dimensions can be determined, and Laplace noise of different noise levels can be generated for different sensitivity levels. The generated Laplace noise is added to the corresponding initial data according to the noise addition method of different preset dimensions, thereby realizing differential privacy protection of user data, improving the security of user data, avoiding the high computing resource requirements of using complex encryption algorithms, reducing maintenance costs, simplifying data protection measures, and avoiding the possibility of security vulnerabilities introduced due to configuration errors or improper implementation.
[0091] Figure 3 A flow chart of the method for integrating the first data set proposed in the present disclosure, based on Figure 1-Figure 2 The embodiment shown, as Figure 3 As shown, Figure 3 right Figure 1 Step 103 in the embodiment is further defined, and includes the following steps:
[0092] Step 301: Perform data cleaning and standardization on first data in a first data set to obtain a second data set.
[0093] In an embodiment of the present disclosure, data cleaning and standardization processing of the first data in the first data set may be data cleaning and standardization or normalization of the noisy user data, removing incomplete or erroneous records, such as processing missing values, outliers, etc. Standardization or normalization of the data can ensure that the data is comparable at different scales to ensure data consistency.
[0094] In an embodiment of the present disclosure, the second data set obtained by performing data cleaning and standardization processing on the first data includes a plurality of second data, and the second data is the data after the initial data is noised.
[0095] For example, noisy user data is collected from nodes distributed across various locations. Each node is responsible for data from a specific region or user group, and data preprocessing is performed, where the data preprocessing includes data cleaning and data standardization.
[0096] Step 302: Separate the data in the second data set into different dimensions according to a preset dimension.
[0097] In an embodiment of the present disclosure, performing dimension separation on the data in the second data set according to a preset dimension may be separating the second data of different preset dimensions from the second data set.
[0098] In an embodiment of the present disclosure, dimensional separation of data in the second data set can be performed by taking the data set obtained after data cleaning and standardization as the total data, separating the geographic location data from the total data, separating the device type data from the total data, and separating the interaction mode data from the total data.
[0099] In an embodiment of the present disclosure, when the preset dimensions include at least one of geographic location, device type, interaction mode, browsing history, communication records, transaction information, etc., the data is separated from the second data set according to different dimensions to obtain data sets of different preset dimensions.
[0100] Step 303 : performing data encoding on the second data of different preset dimensions according to preset rules to obtain integrated data matrices of different preset dimensions.
[0101] In an embodiment of the present disclosure, integrated data matrices of different preset dimensions correspond to integrated data vectors of multiple users.
[0102] In the embodiment of the present disclosure, the preset rules may be processing methods corresponding to different preset dimensions of the pre-set device.
[0103] In an embodiment of the present disclosure, the preset rules may be that for geographic location data, the coordinates of the geographic location are converted into grid-based codes, and the corresponding rows of the output geographic location matrix represent users, and the columns represent geographic location codes at different times; for device type data, the device types are classified and encoded, and the corresponding rows of the output device type matrix represent users, and the columns represent device type codes used by users at different time points; for interaction mode data, the interaction modes are classified and encoded, and the corresponding rows of the output interaction mode matrix represent users, and the columns represent interaction mode codes of users at different times or in different application scenarios.
[0104] In an embodiment of the present disclosure, for the second data of different preset dimensions, data encoding is performed according to preset rules. The second data may be the second data of geographic location. Each second data is encoded or converted to construct a geographic location matrix, in which rows represent individual users and columns represent the geographic location codes of users at different times.
[0105] For example, user A's location is (100.1234, 50.5678), which is the longitude and latitude values after noise. The longitude and latitude coordinates (100.1234, 50.5678) are mapped to a gridded area code, such as area code A12. This can reduce the accuracy of the geographic location and enhance privacy protection. The resulting geographic location matrix has rows representing user A and the corresponding columns are A12.
[0106] In an embodiment of the present disclosure, for the second data of different preset dimensions, data encoding is performed according to preset rules. The second data of the device type can be classified and encoded, and a device type matrix is constructed according to the time series (or other related variables) of user and device usage, where the rows represent individual users and the columns represent the device type codes at different times.
[0107] For example, the device used by user B is a smartphone. According to the rule that smartphones are coded as 1, tablets are coded as 2, and laptops are coded as 3, user B's device type is coded as 1. The row in the resulting device type matrix represents user B, and the corresponding column is 1.
[0108] In an embodiment of the present disclosure, for the second data of different preset dimensions, data encoding is performed according to preset rules. The second data of the interaction mode is classified and encoded, and an interaction mode matrix is constructed, in which rows represent individual users and columns represent the interaction mode encoding of users at different times or in different application scenarios.
[0109] For example, user C's interaction methods are click and slide. According to the rule that click is coded as 1, slide is coded as 2, and input is coded as 3, user C's interaction methods are coded as 1 and 2. In the resulting interaction method matrix, the rows represent user C, and the columns are 1 and 2.
[0110] In the embodiments of the present disclosure, for the interaction mode matrix, its columns represent type codes for different times or different application scenarios, which means that the interaction mode can change not only with time, but also with the application scenario in which the user is located. For example, the user's interaction mode in a social media application may be sliding and motor, while the interaction mode in a work application may be typing and selecting.
[0111] In the embodiments of the present disclosure, the interaction mode matrix has two possible structures: one is a structure with a single time dimension, in which the columns represent the interaction mode encoding of users at different times. This matrix is similar to the geographic location matrix and the device type matrix; the other is a structure with a time + application scenario dimension, in which the columns represent not only time but also application scenarios. For example, the interaction mode in a certain time period can be subdivided according to the application scenario currently being used by the user.
[0112] For example, user D's interaction mode when using a social media application at 9 a.m. is clicking, coded as 1, while when using an office application at 3 p.m., the interaction mode is typing, coded as 3. In this case, the columns in the interaction mode matrix represent not only different times but also different application scenarios.
[0113] In an embodiment of the present disclosure, a geographic location matrix, a device type matrix, and an interaction mode matrix are obtained through data cleaning and standardization processing, dimensional separation, and data encoding. The integrated data matrices of different preset dimensions correspond to integrated data vectors of multiple users, wherein the integrated data vector of each user includes at least one of the integrated data under the geographic location dimension, the integrated data under the device type dimension, and the integrated data under the interaction mode dimension corresponding to the user.
[0114] In the above embodiment, by performing differentiated noise processing before data integration, even if the matrix data is leaked, it is difficult to accurately trace it back to a specific individual, thereby protecting user privacy. Furthermore, by cleaning and standardizing the noise-added data, separating the dimensions according to preset dimensions, and encoding the data of different dimensions according to preset rules, the resulting integrated data matrix can be used for data analysis and machine learning algorithms to help analyze user behavior patterns, predict user trends, or perform other statistical analyses.
[0115] Figure 4 A flow chart of the clustering processing method proposed in this disclosure, based on Figure 1-Figure 3 The embodiment shown, as Figure 4 As shown, Figure 4 right Figure 1 Step 104 in the embodiment is further defined, and includes the following steps:
[0116] Step 401: Take the integrated data vector corresponding to each user as the initial cluster center, perform hierarchical clustering, and obtain an initial cluster set corresponding to each initial cluster center.
[0117] In the embodiment of the present disclosure, taking the integrated data vector corresponding to each user as the initial cluster center may mean taking each integrated data vector as an independent cluster.
[0118] In an embodiment of the present disclosure, hierarchical clustering may be performed using an agglomerative hierarchical clustering algorithm. During initialization, the integrated data vector of each user is treated as an independent cluster to obtain an initial cluster set for each independent cluster.
[0119] For example, the integrated data includes a geographic location matrix, a device type matrix, and an interaction mode matrix. An agglomerative hierarchical clustering algorithm is used. During initialization, each data point is considered as an independent cluster, and the output is an initial cluster set with each data point as an independent cluster.
[0120] In step 402, every two initial cluster centers are taken as cluster pairs, and the cluster pairs are merged based on the Ward criterion to obtain a cluster mapping relationship.
[0121] In the embodiment of the present disclosure, taking every two initial cluster centers as a cluster pair may mean that any two initial cluster centers are formed into a cluster pair, and each initial cluster center may form a cluster pair with all other initial cluster centers.
[0122] In an embodiment of the present disclosure, based on the Ward criterion, merging cluster pairs can start by treating the integrated data vector of each user as a separate cluster, and gradually merging the closest cluster pairs until all integrated data vectors are unified into one cluster or the user-defined number of clusters is reached.
[0123] In an embodiment of the present disclosure, taking every two initial cluster centers as a cluster pair can be when the input is multiple initial cluster sets and the output is a distance matrix, wherein the horizontal axis corresponds to an initial cluster center, the vertical axis corresponds to an initial cluster center, and the intersection of the horizontal and vertical axes is the distance between the two initial cluster centers, that is, the distance of the cluster pair.
[0124] In an embodiment of the present disclosure, based on the Ward criterion, cluster pairs may be merged according to the principle of minimizing the increase in internal variance, and the cluster pairs with the smallest increase in total internal variance may be merged.
[0125] In the above embodiment, clustering is performed on the integrated data to reveal the inherent structure of the data, and the privacy of the user data during the retrieval process is further protected through the cluster mapping relationship.
[0126] Figure 5 This is a flow chart of the method for merging cluster pairs proposed in the present disclosure, based on Figures 1-4 The embodiment shown, as Figure 5 As shown, Figure 5 right Figure 4 Step 402 in the embodiment is further defined, and includes the following steps:
[0127] Step 501: Determine the Euclidean distance of each cluster pair.
[0128] In the embodiment of the present disclosure, determining the Euclidean distance of each cluster pair may be calculating the distance between each pair of initial cluster centers.
[0129]
[0130] The following formula can be used to calculate the distance:
[0131] Among them, χ i and y i are the coordinates of the corresponding points of the two initial cluster centers, that is, the integrated data vector corresponding to the initial cluster centers.
[0132] In an embodiment of the present disclosure, determining the Euclidean distance of a cluster pair may be obtaining a distance matrix between the cluster pairs.
[0133] Step 502: Based on the Euclidean distance and the preset number of clusters, cluster pairs are merged until the preset number of clusters is reached to obtain a cluster mapping relationship.
[0134] In an embodiment of the present disclosure, the cluster mapping relationship is an integrated data vector corresponding to the identifier of each cluster and the users in the corresponding cluster.
[0135] In an embodiment of the present disclosure, merging cluster pairs may be merging initial cluster sets corresponding to the cluster pairs.
[0136] In an embodiment of the present disclosure, cluster pairs are merged based on Euclidean distance and a preset cluster cluster tree. The distance matrix outputted in step 501 is used, and the Ward criterion is used to select cluster pairs that minimize the increase in internal variance for merging to obtain a new cluster set after merging. The new cluster set after merging can be merged into one cluster or reach a set number of clusters.
[0137] In an embodiment of the present disclosure, the number of clusters may be preset by constructing a dendrogram, and selecting a height of the dendrogram, ie, a truncation point, as needed to determine the final number of clusters.
[0138] In an embodiment of the present disclosure, based on the Euclidean distance and the preset number of clusters, merging cluster pairs can be to visualize the clustering process as a tree diagram to show the relationship and distance between clusters. By analyzing the tree diagram, an appropriate cutoff point is selected to determine the final number of clusters. It can be based on business needs, data distribution characteristics or other heuristic methods to ultimately obtain the number of clusters and the mapping of each integrated data vector to its cluster.
[0139] In the embodiments of the present disclosure, the principle of determining the truncation point can be determined based on the following factors: significant changes in the merging distance. Each bifurcation point in the dendrogram represents the merging of clusters. The merging distance will show a significant jump or change at a certain height (i.e., merging two clusters requires a greater cost). The truncation point is usually selected at the location of significant change because continuing to merge at this time will cause the variance within the cluster to increase significantly; business needs or the preset number of clusters. In some cases, the user may have business needs, such as the data must be divided into a specific number of clusters. In this case, the truncation point closest to the need can be found based on the dendrogram; data distribution characteristics: By observing the dendrogram, the distribution characteristics at different clustering levels can be seen. If at a certain level, the number of clusters is more consistent with the structure of the data distribution (such as balanced cluster size or obvious hierarchical structure), the truncation point can be selected here.
[0140] In an embodiment of the present disclosure, cluster pairs are merged to obtain a preset number of cluster clusters, each cluster cluster includes all integrated data vectors constituting the cluster, thereby obtaining a cluster mapping relationship, which is the integrated data vector corresponding to the identifier of each cluster cluster and the user in the corresponding cluster cluster.
[0141] In embodiments of the present disclosure, a cluster mapping includes a cluster identifier and a list of all integrated data vectors that constitute the cluster. The resulting cluster mapping is saved to support reproducible analysis and secure data retrieval. For example, each integrated data vector and its corresponding cluster ID is stored in a database or file system. Archiving the cluster mapping facilitates subsequent data analysis, monitoring, or verification.
[0142] In an embodiment of the present disclosure, when user data needs to be retrieved, after the user's complete identity verification is completed, the preset dimension to be queried is input, and the user's initial data is retrieved based on the cluster mapping relationship and the noise addition method.
[0143] For example, the user is authenticated, and the authentication can be completed in a variety of ways, including but not limited to passwords, biometrics, two-factor authentication, etc. After the user passes the authentication, the specific query conditions entered are obtained, such as geographic location, device type, and interaction method. Data is retrieved based on the cluster mapping relationship to find a data set that matches the query conditions entered by the user. After the corresponding data is extracted, the data is processed using the noise addition processing method disclosed above and returned to the user.
[0144] In the above embodiment, by merging cluster pairs based on the Euclidean distance between them and a preset number of clusters, and using the Ward criterion, the increase in internal variance of the resulting clusters can be minimized, preventing a few large clusters from dominating the entire dataset, and further protecting user data privacy. Furthermore, the resulting cluster mapping can be used for subsequent data analysis, monitoring, or verification, supporting repeatable analysis and secure data retrieval.
[0145] Figure 6 is a flow chart of the data processing method, such as Figure 6 As shown, the method includes:
[0146] S110, adding random noise to the collected user data on each server node for processing to obtain processed user data, where the random noise is noise related to the user data according to geographic location, device type, and interaction mode.
[0147] For example, in differential privacy, the Laplace distribution is adopted in this application due to its ideal properties in differential privacy, namely symmetry and peakedness.
[0148] The noise level in this application is not randomly selected but rather correlated with the sensitivity of the data. In this step, noise is added based on three key characteristics of the user data: geographic location, device type, and interaction method. Raw user data is collected from each client. The noise level is determined based on the sensitivity of each data type, and a corresponding noise value is generated. The calculated noise is added to the corresponding user data. The noisy data is sent to the server for further analysis or storage.
[0149] Specifically, 1. First, conduct a data sensitivity assessment: Each characteristic of user data (geographic location, device type, interaction method) undergoes a sensitivity assessment. This can be accomplished through expert review, historical data analysis, or privacy impact assessments. For example, geographic location is rated as highly sensitive because it may reveal the user's specific location information; device type is generally considered moderately sensitive because, while it provides information about the user's device, it does not directly reveal an individual's location or behavior; interaction method may be considered highly sensitive based on the user's behavioral patterns because it may reveal the user's personal habits or preferences.
[0150] 2. Determine the noise parameter: Based on the sensitivity of the data, set the standard deviation of the Laplace noise (scale parameter, denoted as β) for each data type. In differential privacy, the value of β is generally proportional to the sensitivity of the data. The following guidelines can be used to set this parameter: For highly sensitive data (such as geographic location and interaction mode), choose a larger β value (for example, β = 1.0 or higher) to provide stronger privacy protection; for sensitive data (such as device type), choose a moderate β value (for example, β = 0.5).
[0151] 3. Generate Laplace noise: Generate Laplace-distributed noise for each data point based on its type. The Laplace distribution is used because of its symmetry and peakedness, making it suitable for differential privacy. The noise generation formula is: Noise = Laplace(0, β), where Laplace(0, β) represents a Laplace distribution centered at 0 and with β as the scale parameter.
[0152] 4. Data processing and transmission:
[0153] The generated noise is added directly to each corresponding user data point. For example, if the original data is the user's location coordinates (longitude and latitude), the corresponding noise will be added to these two coordinate values. The noisy data is then securely sent to the server for storage or further analysis.
[0154] S120 , integrating the user data of each node after adding noise data according to three dimensions: geographic location, device type, and interaction mode, to obtain integrated data, where the integrated data includes a geographic location matrix, a device type matrix, and an interaction mode matrix.
[0155] For example, the primary purpose of data integration is to create a unified data view, enabling data analysis and machine learning algorithms to run effectively. By integrating data based on geography, device type, and interaction method, we can better understand the multidimensional structure of the data, supporting subsequent clustering, prediction, and other data mining tasks. Noised data is collected from each node. Each node may be responsible for collecting data for a specific region or user group.
[0156] Specifically, 1. Data preprocessing:
[0157] Data collection: Collect noisy user data from nodes distributed across the country. Each node is responsible for data from a specific region or user group.
[0158] Data cleaning: remove incomplete or erroneous records from the data, such as processing missing values, outliers, etc.
[0159] Data standardization: Standardize or normalize the data to ensure that the data is comparable at different scales to facilitate subsequent analysis.
[0160] 2. Dimension separation and matrix construction:
[0161] (1) Location Matrix:
[0162] Input: User's geolocation data (longitude and latitude) after adding noise.
[0163] Processing: Separate the geographic location data from the total data and encode or transform each user's location data, such as converting geographic coordinates into a grid-based encoding.
[0164] Output: Construct a matrix where the rows represent individual users and the columns represent the geographic location codes of users at different times.
[0165] (2) Device Type Matrix (Device_Type_Matrix):
[0166] Input: Noised user device type data (such as mobile phone, tablet, laptop, etc.).
[0167] Processing: Extract device type information from the total data, classify and code the device type.
[0168] Output: Construct a matrix based on the time series of user and device usage (or other related variables), with rows representing individual users and columns representing device type codes at different times.
[0169] (3) Interaction Mode Matrix:
[0170] Input: Noised user interaction data (such as clicks, slides, and inputs).
[0171] Processing: Extract interaction style data, categorize and code them by type.
[0172] Output: Construct a matrix where rows represent individual users and columns represent the encoding of users' interaction methods at different times or in different application scenarios.
[0173] 3. Application of Matrix:
[0174] Data Analysis and Machine Learning: These matrices will be used in data analysis and machine learning algorithms to help analyze user behavior patterns, predict user trends, or perform other statistical analysis.
[0175] Privacy protection: By adding noise before data integration, even if the matrix data is leaked, it is difficult to accurately trace it back to a specific individual, thereby protecting user privacy.
[0176] S130: clustering the integrated data using a hierarchical clustering algorithm, and saving cluster mapping relationship information.
[0177] S1301. Clustering begins using an agglomerative approach. Agglomerative hierarchical clustering is a bottom-up clustering method. The process begins by treating each data point as a separate cluster and then gradually merges the closest cluster pairs until all data points are unified into a single cluster or the user-defined number of clusters is reached.
[0178] Input: Aggregated data, including geolocation matrix, device type matrix, and interaction mode matrix.
[0179] Processing: An agglomerative hierarchical clustering algorithm is used, and each data point is considered as an independent cluster during initialization.
[0180] Output: An initial set of clusters where each data point is treated as an independent cluster.
[0181] S1302. Calculate cluster distance based on Euclidean distance.
[0182] Input: the initial cluster set generated in S1301.
[0183] Processing: Calculate the distance for each pair of clusters using the Euclidean distance formula:
[0184]
[0185] Among them, χ i and y i are the coordinates of the corresponding points in the two clusters.
[0186] Output: distance matrix between cluster pairs.
[0187] S1303. Clusters are merged based on the Ward linkage criterion. The Ward linkage criterion attempts to minimize the increase in total cohesive distance (i.e., variance) caused by the merger when merging clusters. This causes the clustering process to tend to create clusters of similar size, thereby preventing a few large clusters from dominating the entire dataset.
[0188] Specifically, input: cluster distance matrix of S1302.
[0189] Treatment: Use the Ward linkage criterion to select cluster pairs that minimize the increase in internal variance. This criterion works by merging clusters that minimize the increase in total internal variance.
[0190] Output: The new set of clusters after merging, merging the two closest clusters in each iteration.
[0191] S1304: Construct a dendrogram and select the number of clusters. After constructing the dendrogram, select the height of the dendrogram to be cut off as needed to determine the final number of clusters.
[0192] Specifically, input: data that is gradually merged during the clustering process.
[0193] Processing: Visualize the clustering process as a dendrogram to show the relationship and distance between clusters.
[0194] Decision: By analyzing the dendrogram, choose an appropriate cutoff point to determine the final number of clusters. This can be based on business requirements, data distribution characteristics, or other heuristics.
[0195] Output: The final number of clusters and the mapping of each data point to its cluster.
[0196] S1404. Save cluster mapping information. Saving the mapping relationship between each data point and its corresponding cluster is crucial. This not only facilitates subsequent data analysis and result interpretation, but is also a key step in ensuring analysis reproducibility.
[0197] Specifically, input: the cluster mapping relationship determined in S1304.
[0198] Processing: Store each data point and its corresponding cluster number in a database or file system. This includes the cluster identifier and a list of all the data points that make up that cluster.
[0199] Output: Archive files or database records of cluster mapping relationships, which can be used for subsequent data analysis, monitoring or verification work.
[0200] S140, after the user who needs to call information completes identity authentication, enters the geographic location to be queried, device type and interaction method, and calls user data based on cluster mapping relationship information and noise encryption method.
[0201] For example, first ensure strict user authentication. Authentication can be accomplished in a variety of ways, including but not limited to passwords, biometrics, two-factor authentication, etc.
[0202] After users are authenticated, they can enter specific query criteria, such as geographic location, device type, and interaction method. These criteria are used to locate the specific data sets to be retrieved, ensuring that users can only access the information they are authorized to view.
[0203] Based on cluster mapping information, the system retrieves data using the previously established cluster mapping to find a dataset that matches the user's input criteria. After extracting the relevant data, the system applies noise encryption to further protect user privacy. This step involves adding a certain degree of random noise before returning the data to the user. This noise method uses the Laplace method.
[0204] In summary, this method has the following beneficial effects compared to the prior art:
[0205] 1. Reduced resource consumption: Using differential privacy noise processing based on sensitivity levels reduces the demand for computing resources compared to complex encryption algorithms, optimizes application performance, and can be used in scenarios where large amounts of data are processed in real time.
[0206] 2. Simplified maintenance requirements: By performing differential privacy noise processing based on sensitivity levels, the need for frequent updates to encryption algorithms is reduced, lowering maintenance costs.
[0207] 3. Enhanced security: Simplified data protection measures reduce the possibility of security vulnerabilities introduced by misconfiguration or improper implementation. At the same time, even if the security measures of the user device are insufficient, user data can be protected by implementing the above data processing methods on the server side.
[0208] Figure 7 This is a schematic diagram of the structure of a data processing device provided by an embodiment of the present disclosure. The device 700 includes: an acquisition module 710, a noise addition module 720, an integration module 730, and a clustering module 740. The device 700 implements the data processing method through the above four modules.
[0209] In an embodiment of the present disclosure, the acquisition module 710 is used to acquire an initial data set of multiple nodes, where the initial data set includes user data of multiple users, and the user data includes initial data of multiple preset dimensions.
[0210] The noise adding module 720 is configured to perform noise adding processing on the initial data according to the sensitivity level of the initial data to obtain a first data set, where the first data set includes first data of multiple users.
[0211] The integration module 730 is configured to integrate the first data in the first data set according to a preset dimension to obtain an integrated data matrix of different preset dimensions.
[0212] The clustering module 740 is used to perform clustering processing on the integrated data matrices of different preset dimensions through hierarchical clustering and Ward criterion to obtain cluster mapping relationships.
[0213] In some embodiments, the noise addition module 720 is also used to: determine the sensitivity level of the initial data based on the correspondence between the dimension and the sensitivity level; determine the noise level of the Laplace noise corresponding to the initial data according to the sensitivity level of the initial data; generate Laplace noise based on the noise level, and add the Laplace noise to the corresponding initial data.
[0214] In some embodiments, the noise addition module 720 is also used to: determine the noise level corresponding to the initial data with a high sensitivity level as a first value; determine the noise level corresponding to the initial data with a medium sensitivity level as a second value, and the first value is greater than the second value; determine the noise level corresponding to the initial data with a low sensitivity level as a third value, and the third value is less than the second value.
[0215] In some embodiments, the integration module 730 is further used to: perform data cleaning and standardization on the first data in the first data set to obtain a second data set; perform dimension separation on the data in the second data set according to preset dimensions; perform data encoding on the second data of different preset dimensions according to preset rules to obtain integrated data matrices of different preset dimensions, where the integrated data matrices of different preset dimensions correspond to integrated data vectors of multiple users.
[0216] In some embodiments, the clustering module 730 is also used to: use the integrated data vector corresponding to each user as the initial cluster center, perform hierarchical clustering, and obtain an initial cluster set corresponding to each initial cluster center; use every two initial cluster centers as cluster pairs, and merge the cluster pairs based on the Ward criterion to obtain a cluster mapping relationship.
[0217] In some embodiments, the clustering module 730 is further used to: determine the Euclidean distance of each cluster pair; merge the cluster pairs based on the Euclidean distance and the preset number of cluster clusters until the preset number of cluster clusters is reached, and obtain a cluster mapping relationship, where the cluster mapping relationship is an integrated data vector corresponding to the identifier of each cluster cluster and the user in the corresponding cluster cluster.
[0218] In summary, through the data processing device proposed in the present disclosure, the initial data can be denoised according to the sensitivity level of the initial data to achieve differential privacy protection, and the noisy data can be integrated and clustered to further protect user data.
[0219] In the embodiments provided above, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the various functions of the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the aforementioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. One of the aforementioned functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.
[0220] Figure 8 is a block diagram of an electronic device 800 for implementing the above data processing method according to an exemplary embodiment.
[0221] Reference Figure 8 The electronic device 800 may include a communication interface 801 capable of interacting with other devices; a processor 802 connected to the communication interface 801 to interact with other devices and configured to execute the methods provided by one or more of the above technical solutions when running a computer program; and a memory 803 on which the computer program is stored. Specifically, the specific processing process of the processor 802 may refer to the data processing method described in the above embodiments of the present disclosure.
[0222] Of course, in actual application, the various components in the electronic device 800 are coupled together through the bus system 804. It is understood that the bus system 804 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 8 Various buses are labeled as bus system 804 .
[0223] The memory 803 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 800. Examples of such data include: any computer program used to operate on the electronic device 800.
[0224] The methods disclosed in the above embodiments of the present application can be applied to or implemented by the processor 802. The processor 802 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 802 or by instructions in the form of software. The above processor 802 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 802 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 803. The processor 802 reads the information in the memory 803 and completes the steps of the above method in combination with its hardware.
[0225] In an exemplary embodiment, the electronic device 800 can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.
[0226] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the data processing method described in the above embodiments of the present disclosure.
[0227] An embodiment of the present disclosure further provides a computer program product, including a computer program, which executes the data processing method described in the above embodiments of the present disclosure when a processor is used.
[0228] An embodiment of the present disclosure also proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the data processing method described in the above embodiments of the present disclosure.
[0229] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.
[0230] Throughout this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" indicate that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative uses of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
[0231] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.
[0232] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (control method), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CD ROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.
[0233] It should be understood that various parts of the embodiments of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0234] Those skilled in the art will understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.
[0235] Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in either hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as standalone products, they may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.
[0236] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.
Claims
1. A data processing method, characterized in that: The method comprises: Acquire an initial data set of multiple nodes, wherein the initial data set includes user data of multiple users, and the user data includes initial data of multiple preset dimensions; According to the sensitivity level of the initial data, the initial data is subjected to noise processing to obtain a first data set, wherein the first data set includes first data of a plurality of users; Integrating the first data in the first data set according to a preset dimension to obtain an integrated data matrix of different preset dimensions; The integrated data matrices of different preset dimensions are clustered by hierarchical clustering and Ward criterion to obtain a cluster mapping relationship.
2. The method according to claim 1, characterized in that The performing noise addition processing on the initial data according to the sensitivity level of the initial data comprises: Determining the sensitivity level of the initial data based on the correspondence between the dimension and the sensitivity level; Determining a noise level of the Laplace noise corresponding to the initial data according to a sensitivity level of the initial data; Laplace noise is generated based on the noise level, and the Laplace noise is added to the corresponding original data.
3. The method according to claim 2, characterized in that Determining the noise level of the Laplace noise corresponding to the initial data according to the sensitivity level of the initial data includes: Determine the noise level corresponding to the initial data with a high sensitivity level as a first value; Determine the noise level corresponding to the initial data with medium sensitivity as a second value, wherein the first value is greater than the second value; The noise level corresponding to the initial data with low sensitivity is determined as a third value, and the third value is smaller than the second value.
4. The method according to claim 3, characterized in that The integrating the first data in the first data set according to the preset dimension includes: Performing data cleaning and standardization processing on the first data in the first data set to obtain a second data set; According to the preset dimension, the data in the second data set is dimensionally separated; For the second data of different preset dimensions, data encoding is performed according to preset rules to obtain integrated data matrices of the different preset dimensions, and the integrated data matrices of the different preset dimensions correspond to integrated data vectors of multiple users.
5. The method according to claim 4, characterized in that The clustering process of the integrated data matrices of different preset dimensions by using hierarchical clustering and Ward criterion includes: Taking the integrated data vector corresponding to each user as the initial cluster center, performing hierarchical clustering to obtain an initial cluster set corresponding to each initial cluster center; Every two initial cluster centers are taken as cluster pairs, and based on the Ward criterion, the cluster pairs are merged to obtain the cluster mapping relationship.
6. The method according to claim 5, characterized in that The merging of the cluster pairs based on the Ward criterion includes: Determine the Euclidean distance for each pair of clusters; Based on the Euclidean distance and a preset number of clusters, the cluster pairs are merged until the preset number of clusters is reached to obtain the cluster mapping relationship, where the cluster mapping relationship is an integrated data vector corresponding to the identifier of each cluster and the user in the corresponding cluster.
7. A data processing device, characterized in that: include: An acquisition module, used to acquire an initial data set of multiple nodes, wherein the initial data set includes user data of multiple users, and the user data includes initial data of multiple preset dimensions; a noise adding module, configured to perform noise adding processing on the initial data according to the sensitivity level of the initial data to obtain a first data set, wherein the first data set includes first data of multiple users; An integration module, used to integrate the first data in the first data set according to a preset dimension to obtain an integrated data matrix of different preset dimensions; The clustering module is used to perform clustering processing on the integrated data matrices of different preset dimensions through hierarchical clustering and Ward criterion to obtain a cluster mapping relationship.
8. An electronic device, characterized in that: include: one or more processors; A storage device communicatively connected to the one or more processors, on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
10. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal includes a computer instruction stored in the memory, and when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 6.
Citation Information
Patent Citations
Vehicle privacy information protection method and device, computer equipment and storage medium
CN117332437A
Data classification and hierarchical clustering
WO2008154029A1
