Data analysis methods, systems and servers

Abstract

This disclosure provides a data analysis method, system, and server, relating to big data analysis technology. The method includes: a master node receiving a data analysis request sent by a client and obtaining the original dataset to be analyzed based on the request; a mapper node processing the original dataset to obtain a first set of key-value pairs; a counting node determining a first total number of attribute categories for all key-value pairs based on multiple first key-value pair sets; a mapper node determining a second set of key-value pairs based on the first set of key-value pairs and the first total number, and sending the second set of key-value pairs to each reducer node; and a reducer node determining a target set of key-value pairs based on the first total number and multiple second set of key-value pairs, and feeding the target set of key-value pairs back to the client. This solution can determine the total number of attribute categories included in the data analysis task by processing the first set of key-value pairs obtained from the original dataset, ensuring the normal operation of the data analysis task.

Description

Technical Field

[0001] This disclosure relates to big data analytics technology, and more particularly to a data analytics method, system, and server. Background Technology

[0002] Currently, the MapReduce architecture, as a parallel programming model for large-scale data processing, is widely used in the Hadoop cluster, a distributed system infrastructure. The MapReduce architecture includes mapper nodes and reducer nodes. During the execution of a MapReduce program, there is a threat of sensitive information being leaked through the data volume information transmitted between nodes, which can seriously affect the confidentiality of information in the big data platform. Therefore, a data exchange scheme between MapReduce nodes can be used to ensure that the amount of data transmitted between each node is the same, achieving de-association and shuffling between the data volume information transmitted between computing nodes and the original dataset, thereby preventing the leakage of sensitive information between nodes.

[0003] In existing technologies, the process of deassociating and shuffling data transmitted between computing nodes with the original dataset requires manual pre-determination of the total number of attribute categories included in the data analysis task that will process the original dataset.

[0004] However, the total number of attribute categories is not easy to obtain manually, which affects the normal operation of data analysis tasks. Summary of the Invention

[0005] This disclosure provides a data analysis method, system, and server to address the problem in the prior art that the total number of attribute categories in data analysis tasks that process raw datasets is not easily obtained manually, thus affecting the normal operation of the data analysis tasks.

[0006] According to a first aspect of this disclosure, a data analysis method is provided, applied to a system based on a MapReduce architecture, the system including a master node, multiple mapper nodes, a counter node, and multiple reducer nodes, the method comprising:

[0007] The master node receives a data analysis request sent by the client and obtains the original dataset to be analyzed according to the data analysis request; the mapper node processes the original dataset to obtain a first key-value pair set; the first key-value pair set includes at least one key-value pair; the key-value pair includes a key value and a value corresponding to the key value; the key value is used to characterize the attribute category of the key-value pair;

[0008] The counting node determines the first total number of attribute categories for all key-value pairs in the multiple sets of first key-value pairs based on the multiple sets of first key-value pairs.

[0009] The mapper node determines a second key-value pair set based on the first key-value pair set and the first total number; and sends the second key-value pair set to each of the reducer nodes; the total number of key-value pairs in the second key-value pair set is equal to the first total number.

[0010] The reducer node determines a target key-value pair set based on the first total number and multiple sets of the second key-value pairs, wherein the total number of key-value pairs in the target key-value pair set is equal to the first total number, and feeds back the target key-value pair set to the client.

[0011] According to a second aspect of this disclosure, a data analysis system based on a MapReduce architecture is provided, the system comprising a master node, multiple mapper nodes, a counter node, and multiple reducer nodes, including:

[0012] The master node is used to obtain data analysis requests sent by the client and obtain the original dataset to be analyzed based on the data analysis requests;

[0013] The mapper node is used to process the original dataset to obtain a first key-value pair set; the first key-value pair set includes at least one key-value pair; the key-value pair includes a key value and a value corresponding to the key value; the key value is used to characterize the attribute category of the key-value pair.

[0014] The counting node is used to determine a first total number of attribute categories for all key-value pairs in the multiple first key-value pair sets based on the multiple first key-value pair sets.

[0015] The mapper node is further configured to determine a second key-value pair set based on the first key-value pair set and the first total number; and send the second key-value pair set to each of the reducer nodes respectively; the total number of key-value pairs in the second key-value pair set is equal to the first total number;

[0016] The reducer node is configured to determine a target key-value pair set based on the first total number and multiple second key-value pair sets, wherein the total number of key-value pairs in the target key-value pair set is equal to the first total number, and to feed back the target key-value pair set to the client.

[0017] According to a third aspect of this disclosure, a server is provided, including a memory and a processor; wherein,

[0018] The memory is used to store computer programs;

[0019] The processor is configured to read the computer program stored in the memory and execute the data analysis method as described in the first aspect according to the computer program in the memory.

[0020] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the data analysis method as described in the first aspect is implemented.

[0021] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the data analysis method as described in the first aspect.

[0022] The data analysis method, system, and server disclosed herein include: a master node acquiring a data analysis request sent by a client and acquiring a raw dataset to be analyzed based on the data analysis request; a mapper node processing the raw dataset to obtain a first key-value pair set; the first key-value pair set includes at least one key-value pair; each key-value pair includes a key and a corresponding value; the key is used to characterize the attribute category of the key-value pair; a counting node determining a first total number of attribute categories for all key-value pairs in the multiple first key-value pair sets based on the multiple first key-value pair sets; the mapper node determining a second key-value pair set based on the first key-value pair set and the first total number; and sending the second key-value pair set to each reducer node; the total number of key-value pairs in the second key-value pair set equals the first total number; and the reducer node determining a target key-value pair set based on the first total number and the multiple second key-value pair sets, the total number of key-value pairs in the target key-value pair set equals the first total number, and feeding back the target key-value pair set to the client. The data analysis methods, systems, and servers provided in this solution can determine the total number of attribute categories included in the data analysis task that processes the original dataset by processing the first set of key-value pairs through mapper nodes, thus ensuring the normal operation of the data analysis task. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a data analysis method as an exemplary embodiment of this disclosure;

[0025] Figure 2 A flowchart illustrating a data analysis method as another exemplary embodiment of this disclosure;

[0026] Figure 3 This is a schematic diagram illustrating a Bloom filter cluster architecture as an exemplary embodiment of this disclosure;

[0027] Figure 4 This is a schematic diagram of a trusted data shuffling framework for inter-node covert channels, as illustrated in an exemplary embodiment of this disclosure.

[0028] Figure 5 This is a schematic diagram illustrating a data analysis process as an exemplary embodiment of the present disclosure;

[0029] Figure 6 A structural diagram of a data analysis system based on the MapReduce architecture, as illustrated in an exemplary embodiment of this disclosure;

[0030] Figure 7 This is a structural diagram of a server shown as an exemplary embodiment of the present disclosure. Detailed Implementation

[0031] Currently, the MapReduce architecture, as a parallel programming model for large-scale data processing, is widely used in the Hadoop cluster, a distributed system infrastructure. Hadoop includes the MapReduce architecture and the Hadoop Distributed File System (HDFS); the MapReduce architecture represents the computing power of Hadoop, while HDFS represents its storage capacity. The MapReduce architecture includes mapper nodes and reducer nodes. Mapper nodes process the raw dataset for analysis into key-value pairs, which are then used in subsequent data analysis. During data transfer from mapper nodes to reducer nodes, and during the process of reducer nodes transferring data to the distributed file system or to clients, attackers can easily obtain information about the amount of data transferred between nodes from the logs. This data volume information can be used to infer sensitive information from the original dataset, severely compromising the confidentiality of information within the big data platform. Therefore, a data exchange scheme between MapReduce nodes can be used to ensure that the amount of data transferred between nodes is the same. This allows for the de-association and shuffling of the data volume information transferred between computing nodes with the original dataset, thereby preventing the leakage of sensitive information between nodes. To ensure that the amount of data transmitted between nodes is the same, the total number of attribute categories included in the corresponding data analysis task must first be determined. This total number of attribute categories then determines the amount of data transmitted between nodes. In existing technologies, the total number of attribute categories included in the data analysis task that processes the original dataset needs to be manually predetermined.

[0032] However, the total number of attribute categories is not easy to obtain manually, which affects the normal operation of data analysis tasks.

[0033] To address the aforementioned technical issues, the solution provided in this disclosure allows for the determination of the total number of attribute categories included in the data analysis task that processes the original dataset by using mapper nodes to process the first set of key-value pairs, thereby ensuring the normal operation of the data analysis task.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0035] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0036] Figure 1 This is a flowchart illustrating a data analysis method in an exemplary embodiment of the present disclosure. The data analysis method provided in this embodiment is applied to a system based on a MapReduce architecture, the system including a master node, multiple mapper nodes, a counting node, and multiple reducer nodes.

[0037] like Figure 1 As shown, the data analysis method provided in this embodiment includes:

[0038] Step 101: The master node obtains the data analysis request sent by the client and obtains the original dataset to be analyzed based on the data analysis request.

[0039] The execution entity of the method provided in this disclosure can be a system based on the MapReduce architecture.

[0040] Specifically, the system based on the MapReduce architecture can be a Hadoop cluster. A Hadoop cluster can include the MapReduce architecture and HDFS. The MapReduce architecture can include a master node, multiple mapper nodes, a counter node, and multiple reducer nodes.

[0041] The master node is able to receive data analysis requests sent by the client.

[0042] Specifically, the data analysis request can include the location information of the original dataset to be analyzed in HDFS. The master node can obtain the original dataset to be analyzed from HDFS based on this location information.

[0043] Furthermore, before the client sends a data analysis request, the original dataset to be analyzed can be stored in HDFS.

[0044] Furthermore, the data analysis request may also include the type of data analysis task that you want to analyze and process the original dataset. For example, the data analysis task type may be to separately count the number of men and women in the original dataset, or to count the age in the original dataset in stages.

[0045] The original dataset can be encrypted data.

[0046] Furthermore, after the master node obtains the original dataset to be analyzed, it can divide the original dataset into multiple blocks and send the divided original datasets to the mapper nodes for processing.

[0047] Step 102: The mapper node processes the original dataset to obtain a first key-value pair set; the first key-value pair set includes at least one key-value pair; the key-value pair includes a key value and the value corresponding to the key value; the key value is used to represent the attribute category of the key-value pair.

[0048] Specifically, the MapReduce architecture includes multiple mapper nodes. For example, if the MapReduce architecture includes M mapper nodes, the data in the original dataset can be divided into M equal parts, and each mapper node processes one part of the first data. When each mapper node receives a part of the first data, it can first decrypt the first data, and then process the decrypted data into key-value pairs, thus obtaining the first key-value pair set.

[0049] Specifically, each mapper node, after processing the corresponding first data, can obtain a first set of key-value pairs.

[0050] Specifically, the first key-value pair set includes at least one key-value pair, each key-value pair including a key value and a value corresponding to the key value, wherein the key value is used to characterize the attribute category of the key-value pair.

[0051] For example, if the original dataset contains information such as the names and ages of multiple people, and you want to perform age-based statistics on the original dataset, you can divide the age into multiple segments, each with a unique key-value pair. For instance, after processing the original dataset through a mapper node, the resulting set of key-value pairs can be represented as: key-value pair (1, 10), key-value pair (2, 30), and key-value pair (3, 20). Then, key-value pair (1, 10) represents 10 people aged 0-10; 1 represents the key value, also indicating that the attribute category of this key-value pair is the 0-10 age group, and 10 represents the value of the key-value pair corresponding to key value 1. Key-value pair (2, 30) represents 30 people aged 11-30; 2 represents the key value, also indicating that the attribute category of this key-value pair is the 11-30 age group, and 30 represents the value of the key-value pair corresponding to key value 2. The key-value pair (3, 20) can represent that there are 20 people whose age group is over 30 years old; 3 represents the key value, which is also used to characterize the attribute category of the key-value pair as the age group over 30 years old, and 20 represents the value of the key-value pair corresponding to the key value 3.

[0052] Step 103: The counting node determines the first total number of attribute categories for all key-value pairs in the multiple first key-value pair sets based on the multiple first key-value pair sets.

[0053] Specifically, the counting node can determine the first total number of attribute categories for all key-value pairs in all first key-value pair sets obtained from all mapper nodes.

[0054] For example, a Bloom filter set in the counting node can be used to count the attribute categories of all key-value pairs in all key-value pair sets to obtain the first total. Specifically, a Bloom filter can be used to filter the keys in all key-value pairs, removing duplicate keys and counting the unique keys, thereby obtaining the first total of attribute categories for all key-value pairs in all key-value pair sets.

[0055] Specifically, this first total can be used to characterize the total number of attribute categories contained in the data analysis tasks included in the data analysis request.

[0056] Step 104: The mapper node determines the second key-value pair set based on the first key-value pair set and the first total number; and sends the second key-value pair set to each reducer node; the total number of key-value pairs in the second key-value pair set is equal to the first total number.

[0057] Specifically, the mapper node can be used to determine the second key-value pair set based on the first key-value pair set and the first total number; the total number of key-value pairs in the second key-value pair set is equal to the first total number.

[0058] For example, if the original dataset contains information such as the names and ages of multiple people, and you want to perform age-based statistics on that dataset, you can divide the ages into three segments: 0-10 years old, 11-30 years old, and over 30 years old; each segment has a unique key-value pair. For example, the key-value pair for 0-10 years old is 1; the key-value pair for 11-30 years old is 2; and the key-value pair for over 30 years old is 3. In this example, the first total would be 3.

[0059] A given mapper node produces a first set of key-value pairs including key-value pair (1, 5) and key-value pair (2, 10). Therefore, this mapper node has a total of 2 key-value pairs, one less than the initial total of 3. Thus, a dummy key-value pair can be added to the first set of key-value pairs of this mapper node to obtain a second set of key-value pairs, bringing the total number of key-value pairs in the second set to 3, which is the initial total.

[0060] Next, each mapper node can send its corresponding set of second key-value pairs to each reducer node. For example, if the MapReduce architecture includes two mapper nodes (a first mapper node and a second mapper node) and two reducer nodes (a first reducer node and a second reducer node), the first mapper node can send its corresponding set of second key-value pairs to both the first and second reducer nodes; the second mapper node can send its corresponding set of second key-value pairs to both the first and second reducer nodes.

[0061] Step 105: The reducer node determines the target key-value pair set based on the first total number and multiple second key-value pair sets. The total number of key-value pairs in the target key-value pair set is equal to the first total number, and the target key-value pair set is fed back to the client.

[0062] Specifically, the attribute categories of the data processed by each reducer node can be pre-defined. Then, each reducer node processes all received second key-value pair sets according to the attribute categories it can process. Specifically, it first deletes key-value pairs that do not conform to the attribute categories it can process, as well as spurious key-value pairs; then it merges the remaining key-value pairs according to the attribute categories to obtain the third key-value pair set.

[0063] Next, based on the third set of key-value pairs and the first total number, the target set of key-value pairs is determined.

[0064] For example, a certain reducer node might produce a third set of key-value pairs containing key-value pairs (1, 5) and (2, 10). Since the first total is 3, this reducer node has a total of 2 key-value pairs, one less than the first total of 3. Therefore, a dummy key-value pair can be added to the third set of key-value pairs in this reducer node to obtain the target set of key-value pairs, bringing the total number of key-value pairs in the target set to 3, which is the first total.

[0065] Specifically, the set of all target key-value pairs obtained by all reducer nodes can be stored in HDFS first, and then fed back to the client.

[0066] The data analysis method disclosed herein includes: a master node acquiring a data analysis request sent by a client and acquiring a raw dataset to be analyzed based on the data analysis request; a mapper node processing the raw dataset to obtain a first key-value pair set; the first key-value pair set includes at least one key-value pair; each key-value pair includes a key and a corresponding value; the key is used to characterize the attribute category of the key-value pair; a counting node determining a first total number of attribute categories for all key-value pairs in the multiple first key-value pair sets based on the multiple first key-value pair sets; the mapper node determining a second key-value pair set based on the first key-value pair set and the first total number; and sending the second key-value pair set to each reducer node; the total number of key-value pairs in the second key-value pair set equals the first total number; and the reducer node determining a target key-value pair set based on the first total number and the multiple second key-value pair sets, the total number of key-value pairs in the target key-value pair set equals the first total number, and feeding back the target key-value pair set to the client. In the method disclosed herein, the first set of key-value pairs obtained by processing the original dataset through the mapper node can be used to determine the total number of attribute categories included in the data analysis task that processes the original dataset, thus ensuring the normal operation of the data analysis task.

[0067] Figure 2 This is a flowchart illustrating a data analysis method as another exemplary embodiment of the present disclosure. The data analysis method provided in this embodiment is applied to a system based on a MapReduce architecture, the system including a master node, multiple mapper nodes, a counting node, and multiple reducer nodes.

[0068] like Figure 2 As shown, the data analysis method provided in this embodiment includes:

[0069] Step 201: The master node obtains the data analysis request sent by the client and obtains the original dataset to be analyzed based on the data analysis request.

[0070] Specifically, the principle and implementation of step 201 are similar to those of step 101, and will not be repeated here.

[0071] Step 202: The mapper node processes the original dataset to obtain a first key-value pair set; the first key-value pair set includes at least one key-value pair; the key-value pair includes a key value and the value corresponding to the key value; the key value is used to represent the attribute category of the key-value pair.

[0072] Specifically, the principle and implementation of step 202 are similar to those of step 102, and will not be repeated here.

[0073] Step 203: The counting node determines the input dataset based on the key values ​​of all key-value pairs included in the multiple first key-value pair sets, and inputs the input dataset into the preset Bloom filter cluster.

[0074] The preset Bloom filter cluster is a pre-configured cluster of Bloom filters based on actual conditions. This Bloom filter cluster includes multiple Bloom filters.

[0075] Specifically, the counting node can take the key values ​​of all key-value pairs included in the first key-value pair set obtained from all mapper nodes to form the input dataset. This input dataset is then fed into a pre-defined Bloom filter cluster.

[0076] Step 204: The counting node processes the input dataset using a pre-defined Bloom filter cluster to obtain the first total number of attribute categories for all key-value pairs in multiple first key-value pair sets.

[0077] Because the key in a key-value pair represents the attribute category of the key-value pair, and the key corresponding to each attribute category is unique, a pre-defined Bloom filter cluster can be used to deduplicate the data in the input dataset and count the unique data. This allows us to obtain the first total number of attribute categories for all key-value pairs included in the first key-value pair set obtained using all mapper nodes.

[0078] In one possible implementation, the pre-defined Bloom filter cluster includes at least one Bloom filter group; the number of Bloom filters in each Bloom filter group does not exceed a first threshold; and the total amount of data processed by each Bloom filter does not exceed a second threshold.

[0079] The first threshold is a threshold that is preset according to the actual situation.

[0080] The second threshold can be determined based on the parameters of each Bloom filter group. The second threshold of Bloom filters in the same Bloom filter group can be the same, while the second threshold of Bloom filters in different Bloom filter groups can be different.

[0081] Specifically, the pre-configured Bloom filter cluster may include at least one Bloom filter group, the number of Bloom filters in each Bloom filter group does not exceed a first threshold, and the total amount of data processed by each Bloom filter in each Bloom filter group does not exceed a second threshold.

[0082] Initialize a Bloom filter set for the input dataset, and a Bloom filter within the Bloom filter set; process the input dataset using the Bloom filter.

[0083] Specifically, you can first initialize an initial Bloom filter set for the input dataset, and then initialize a Bloom filter within that initial Bloom filter set, and use that Bloom filter to process the input dataset.

[0084] Specifically, a Bloom filter can deduplicate data in an input dataset and count the unique data.

[0085] Specifically, the parameters of the Bloom filters in the Bloom filter cluster can be preset.

[0086] In one possible implementation, the parameters of the current Bloom filter group are as follows:

[0087] G i =(P i ξ i m i v i p, M i )

[0088] ξ i =ξ0×r i =1-(1-P) i ) λ

[0089] m i =m0×s i

[0090] v i =log2(1 / P) i )

[0091]

[0092] Among them, G i P represents the i-th Bloom filter group; i is a natural number; i G represents i The false positive rate of each Bloom filter; ξ i G represents i Grouping misclassification rate; m i G represents i The size of each Bloom filter segment; v i G represents i The number of hash functions in the corresponding hash function set; p represents G i The expected fill factor for each Bloom filter is given by p = 0.5; M i ξ represents the second threshold; ξ0 represents the preset grouping misclassification rate; r represents the preset reduction factor parameter, r is less than 1; λ represents the first threshold; m0 represents the preset Bloom filter segment size; s represents the preset expansion factor parameter, s is greater than or equal to 1.

[0093] Specifically, G i P represents the i-th Bloom filter group; i is a natural number; i G represents i The false positive rate of each Bloom filter; ξ i G represents i Grouping misclassification rate; m i G represents i The size of each Bloom filter segment; v i G represents i The number of hash functions in the corresponding hash function set can also be represented by G. i The number of segments in each Bloom filter; p represents G i The expected fill factor of each Bloom filter is determined by the fact that the Bloom filter can accommodate the largest number of elements when the expected fill factor is 0.5. Therefore, in this embodiment, p = 0.5 can be used. i This represents the second threshold, namely G. i The threshold for the amount of data processed by each Bloom filter in G can also be represented as G. i The size of each Bloom filter; ξ0 represents the preset grouping misclassification rate; r represents the preset reduction factor parameter, r is less than 1. In order to ensure the convergence of the misclassification rate of the overall architecture of the preset Bloom filter cluster, this reduction factor r less than 1 is introduced; λ represents the first threshold, that is, the number of Bloom filters in each Bloom filter group; m0 represents the preset Bloom filter fragment size; s represents the preset expansion factor parameter, s is greater than or equal to 1. In order to enable each Bloom filter in the newly expanded Bloom filter group to accommodate more elements, an expansion factor s greater than or equal to 1 is introduced.

[0094] Furthermore, the false positive rate P0 of each Bloom filter in the initial Bloom filter group G0 can be preset.

[0095] In one possible implementation, all Bloom filters in the same Bloom filter group share the same set of hash functions; different Bloom filter groups correspond to different sets of hash functions.

[0096] Specifically, all Bloom filters within the same Bloom filter group have the same parameters. Furthermore, all Bloom filters within the same Bloom filter group share the same set of hash functions; different Bloom filter groups correspond to different sets of hash functions.

[0097] Furthermore, the convergence of the total false positive rate of the preset Bloom filter cluster can be determined through formula derivation, as shown below:

[0098]

[0099] Where P represents the total false positive rate of the preset Bloom filter cluster; L represents the number of Bloom filter groups in the preset Bloom filter cluster; ξ0 represents the preset grouping false positive rate; i is a natural number; and r represents the preset reduction factor parameter, where r is less than 1.

[0100] When the total amount of data in the input dataset approaches infinity, it can be deduced that the total false positive rate of the preset Bloom filter cluster has convergence, that is, it satisfies the following relationship:

[0101]

[0102] Where P represents the total false positive rate of the preset Bloom filter cluster; ξ0 represents the preset grouping false positive rate; r represents the preset reduction factor parameter, r is less than 1; λ represents the first threshold, i.e. the number of Bloom filters in each Bloom filter group; P0 represents the false positive rate of each Bloom filter in the preset initial filter group.

[0103] Specifically, such as Figure 3 The diagram shown is a schematic of a pre-configured Bloom filter cluster architecture. The initial Bloom filter group can be represented as G0.

[0104] Specifically, the following steps can be repeated until all data in the input dataset has been processed.

[0105] If the total amount of data processed by the current Bloom filter reaches the second threshold, then the next Bloom filter is generated.

[0106] Specifically, the total amount of data processed by the current Bloom filter can be detected. When it is determined that the initial total amount of data processed by the current Bloom filter has reached the second threshold, and it is determined that the number of Bloom filters in the current Bloom filter group has not reached the first threshold, then the next Bloom filter is generated in the current Bloom filter group, and the next Bloom filter is used to continue processing the unprocessed data in the input dataset.

[0107] If it is determined that the number of Bloom filters in the current Bloom filter group has reached the first threshold, then the next Bloom filter group is generated, and a Bloom filter is generated in the next Bloom filter group.

[0108] Specifically, if it is determined that the total amount of data processed by the current Bloom filter has reached the second threshold, and it is determined that the number of Bloom filters in the current Bloom filter group has reached the first threshold, then the next Bloom filter group is generated, and a Bloom filter is generated in the next Bloom filter group. The newly generated Bloom filter is then used to continue processing the unprocessed data in the input dataset.

[0109] Specifically, after the preset Bloom filter cluster has processed all the data in the input dataset, the resulting count value is the first total number of attribute categories of all key-value pairs included in the first key-value pair set obtained using all mapper nodes.

[0110] The mainstream Count Bloom Filter (CBF) can also be used to process the input dataset to obtain the first total.

[0111] However, CBF does not use sharding, has a large storage space, poor robustness, and large error; it uses a large number of hash functions, resulting in long computation time and poor performance; and it can only process a fixed amount of data, which cannot meet the needs of constantly changing input datasets.

[0112] Compared to the current mainstream CBF, the preset Bloom filter cluster provided in this embodiment can dynamically expand the number of Bloom filter groups and Bloom filters in the preset Bloom filter cluster according to the total amount of data in the input dataset, which can meet the needs of the constantly changing total amount of data in the input dataset; moreover, the Bloom filters in the preset Bloom filter cluster use sharding, which has small storage space, high robustness, and small error; the number of hash functions used is small, which results in short computation time and better performance.

[0113] Step 205: The mapper node determines the second key-value pair set based on the first key-value pair set and the first total number; and sends the second key-value pair set to each reducer node; the total number of key-value pairs in the second key-value pair set is equal to the first total number.

[0114] Specifically, the principle and implementation of step 205 are similar to those of step 104, and will not be repeated here.

[0115] Step 206: The reducer node determines the target key-value pair set based on the first total number and multiple second key-value pair sets. The total number of key-value pairs in the target key-value pair set is equal to the first total number, and the target key-value pair set is fed back to the client.

[0116] Specifically, the principle and implementation of step 206 are similar to those of step 105, and will not be repeated here.

[0117] Figure 4 This is a schematic diagram of a trusted data shuffling framework for inter-node covert channels, as illustrated in an exemplary embodiment of this disclosure.

[0118] This embodiment provides a data analysis method based on the MapReduce architecture. This method is based on a secure hidden channel association hiding model, which performs the same distributed computing task for two original datasets D0 and D1. This secure hidden channel association hiding model can effectively protect the confidentiality of data exchange between MapReduce nodes. The secure hidden channel association hiding model satisfies the following conditions: |D0|=|D1| is equal to the original dataset size if and only if the original datasets are the same size, i.e., |D0|=|D1| is satisfied, and:

[0119] 1) Different key-value pairs in the dataset execution task have the same attribute category;

[0120] 2) Distributed computing tasks support the pre-reduction Combine function, which means that the input key-value pairs and output key-value pairs of the reduce function have the same type, and the reduce function produces at least one key-value pair.

[0121] like Figure 4 As shown, based on the above definition of the secure hidden channel association concealment model, this embodiment provides a trusted data shuffling framework for inter-node hidden channels (wherein, the inter-node hidden channel refers to data transmission between mapper nodes and reducer nodes) to protect sensitive user information in distributed computing tasks. The data transmitted between nodes is securely protected by semantic security encryption technology, and the specific process is as follows:

[0122] Step 1: The client uploads the encrypted raw dataset to the cloud server and creates distributed computing tasks for data processing and analysis based on business needs;

[0123] Step 2: The cloud-based distributed computing task distributes the original dataset to different mapper nodes to execute specific functions (such as the map function);

[0124] Step 3: Each compute node performs a pre-reduction operation on the key-value pairs output by the map function, that is, pre-merges key-value pairs with the same attribute.

[0125] Step 4: Each mapper node communicates with the intermediate counting server and uses the pre-set Bloom filter cluster in the server to count the total number of key-value pair attribute categories in different mapper nodes while ensuring a low error rate. The total number of attribute categories is then returned to each mapper node through a reliable communication method.

[0126] Step 5: For the intermediate channel transmission process (referring to the data transmission process between mapper nodes and reducer nodes), a node-to-node hidden channel data shuffling scheme based on a preset Bloom filter cluster is used to protect the privacy of sensitive data in the channel (i.e., the data in the intermediate channel is shuffled in the intermediate channel), so that the inter-node channel traffic in the same computing task corresponding to different datasets is indistinguishable to malicious attackers.

[0127] Step 6: Transfer the shuffled key-value pair data to the corresponding reducer node to execute the reduce function logic processing, and remove invalid data according to the function definition rules;

[0128] Step 7: Encrypt and store the distributed computing results in HDFS.

[0129] Figure 5 This is a schematic diagram illustrating a data analysis process based on the MapReduce architecture, which is an exemplary embodiment of this disclosure.

[0130] like Figure 5 As shown, the execution of distributed computing tasks in the aforementioned inter-node hidden-channel trusted data shredding framework consists of a standard Mapping phase (which includes multiple map tasks) and a Reducing phase (which includes multiple reduce tasks). In addition, a Counting phase is added between these two phases. In this phase, a method based on a pre-defined Bloom filter cluster is designed to count the number of attribute categories with different key values ​​in all map tasks (each mapper node includes one map task). Throughout all these phases, data and programs within the computing nodes are executed in a secure environment, preventing malicious attackers from eavesdropping or tampering with the distributed tasks. Furthermore, data transmission between trusted nodes is protected using semantic security encryption technology, ensuring secure transmission to the next node or storage in a distributed file system.

[0131] The execution process of this scheme mainly includes three stages: Mapping, Counting, and Reducing.

[0132] Mapping Phase: In this phase, the encrypted raw dataset is distributed proportionally to each map task (the input raw dataset is first divided into blocks, for example, the raw dataset can be divided into data blocks 0-5). In each map task, the data exists in the form of key-value pairs. First, when each map task receives an encrypted key-value pair, it decrypts the key-value pair and passes it as a parameter to the map function. The map function performs logical processing on the key-value pairs according to the specific distributed computing task. Second, the processed key-value pairs are sent to the pre-reduction combine function to execute the reduce function aggregation logic, that is, key-value pairs with the same attribute category will be merged into a key-value pair set. As a result, each map task will generate a key-value pair set, where the key and value of each key-value pair are unique and independent. We can represent the set of key-value pairs generated by the map task as Tm. Then, the map task stores the key (i.e., the value) of each key-value pair in the key-value pair set Tm into a set, which we call the aggregated key-value set and denoted as Km (where, for example, ...). Figure 5 As shown, Km can include k1, k2, k3), and finally, the encrypted aggregated key-value set Km is sent to the Counting stage and waits for the response from that stage. Assuming there are M map tasks in the distributed computing task, the Counting stage will receive M aggregated key-value sets.

[0133] Counting Phase: In this phase, firstly, each aggregated key-value set Km is decrypted, and each element is stored in a pre-defined Bloom filter cluster to calculate the total number of attribute categories for different attribute key-value pairs across all map tasks. We use the symbol K_distinct to represent the total number of attribute categories for different attribute key-value pairs across all map tasks (this total number can be called the first total, such as...). Figure 5 As shown in the example, the first total is 3), and this value is then encrypted and transmitted to each map task. For each map task in the Mapping phase, when it receives the total number K_distinct of attribute categories for encrypted key-value pairs, it decrypts them and generates K' fake key-value pairs so that the number of attribute categories for different attribute key-value pairs in all map tasks reaches this statistical value, i.e., satisfying the relation K' = K_distinct - |Tm|, where each fake key-value pair contains a key-value pair (k_fake, v_fake). Finally, the K_distinct number of encrypted key-value pairs are distributed to each reduce task to perform the reduction logic processing. Figure 5In the distributed computing task shown, in the Counting stage, three aggregated key-value sets will be received. The elements in each aggregated key-value set will be used as input data to execute a method based on a preset Bloom filter cluster, and the calculated K_distinct value is 3. Finally, this value will be encrypted and distributed to each map task.

[0134] Reducing stage: In this stage, first, the reduce tasks (each reducer node includes a reduce task, such as Figure 5 in the example shown, there are two reduce tasks, respectively represented as r1 and r2) will decrypt the multiple encrypted key-value pairs t received. The key-value pair can be expressed as (t.key, t.value). Subsequently, authentication will be performed on the key-value in the key-value pair. If the key-value is the key-value in a false key-value pair or the key-value pair should not be processed in this node, that is, satisfying the relationship t.key = k_fake or P(t.key) ≠ r, the reduce computing task will remove this key-value pair. For the remaining key-value pairs, the computing task will use them as the input key-value pairs of the reduce function to perform reduction logic processing. Finally, each reduce task will generate an aggregated key-value pair set Tr, where the key-value of each key-value pair is unique and independent. If the number of elements in the aggregated key-value pair set is less than the total number of attribute categories of different attribute key-value pairs in all map tasks, that is, |Tr| < Kdistinct, the reduce task r will generate Fr false key-value pairs, where Fr = K_distinct - |Tr|, and the false key-value pair can be (k_fake, v-fake). Finally, the |Tr| key-value pairs and the Fr false key-value pairs will be encrypted and saved in the distributed file system. As Figure 5 shown, in this distributed computing task, for the reduce task r1, three false key-value pairs and two key-value pairs from the map tasks are removed. Since r1 generates two key-value pairs through the reduce function, this task will add a false key-value pair to the result set and encrypt and store it in HDFS together with the aggregated key-value pair set.

[0135] Figure 6 This is the structure diagram of the data analysis system based on the MapReduce architecture shown in an exemplary embodiment of the present disclosure. The data analysis system based on the MapReduce architecture provided in this embodiment includes a master node, multiple mapper nodes, counting nodes, and multiple reducer nodes.

[0136] As Figure 6 shown, the data analysis system based on the MapReduce architecture provided by the present disclosure includes:

[0137] The master node 610 is used to obtain data analysis requests sent by the client and obtain the original dataset to be analyzed according to the data analysis requests;

[0138] Mapper node 620 is used to process the original dataset to obtain the first key-value pair set; the first key-value pair set includes at least one key-value pair; the key-value pair includes a key and a value corresponding to the key; the key is used to represent the attribute category of the key-value pair;

[0139] Counting node 630 is used to determine the first total number of attribute categories for all key-value pairs in the multiple first key-value pair sets, based on the multiple first key-value pair sets.

[0140] The mapper node 620 is also used to determine the second key-value pair set based on the first key-value pair set and the first total number; and to send the second key-value pair set to each reducer node respectively; the total number of key-value pairs in the second key-value pair set is equal to the first total number;

[0141] Reducer node 640 is used to determine the target key-value pair set based on the first total and multiple second key-value pair sets. The total number of key-value pairs in the target key-value pair set is equal to the first total, and the target key-value pair set is fed back to the client.

[0142] The counting node 630 is specifically used to determine the input dataset based on the key values ​​of all key-value pairs included in multiple first key-value pair sets, and input the input dataset into a preset Bloom filter cluster;

[0143] The input dataset is processed using a pre-defined cluster of Bloom filters to obtain the first total number of attribute categories for all key-value pairs in multiple first key-value pair sets.

[0144] The counting node 630 is specifically used for a preset Bloom filter cluster including at least one Bloom filter group; the number of Bloom filters in each Bloom filter group does not exceed a first threshold; the total amount of data processed by each Bloom filter does not exceed a second threshold; a Bloom filter group and a Bloom filter in the Bloom filter group are initialized for the input dataset; the input dataset is processed using the Bloom filter.

[0145] If it is determined that the total amount of data processed by the current Bloom filter has reached the second threshold, then the next Bloom filter is generated.

[0146] If it is determined that the number of Bloom filters in the current Bloom filter group has reached the first threshold, then the next Bloom filter group is generated, and a Bloom filter is generated in the next Bloom filter group.

[0147] Repeat the step of generating the next Bloom filter if the current Bloom filter's data processing volume reaches the second threshold, until the input dataset is completely processed.

[0148] In one possible implementation, the parameters of the current Bloom filter group are as follows:

[0149] G i =(P i ξ i m i v i p, M i )

[0150] ξ i =ξ0×r i =1-(1-P) i ) λ

[0151] m i =m0×s i

[0152] v i =log2(1 / P) i )

[0153]

[0154] Among them, G i P represents the i-th Bloom filter group; i is a natural number; i G represents i The false positive rate of each Bloom filter; ξ i G represents i Grouping misclassification rate; m i G represents i The size of each Bloom filter segment; v i G represents i The number of hash functions in the corresponding hash function set; p represents G i The expected fill factor for each Bloom filter is given by p = 0.5; M i ξ represents the second threshold; ξ0 represents the preset grouping misclassification rate; r represents the preset reduction factor parameter, r is less than 1; λ represents the first threshold; m0 represents the preset Bloom filter segment size; s represents the preset expansion factor parameter, s is greater than or equal to 1.

[0155] In one possible implementation, all Bloom filters in the same Bloom filter group share the same set of hash functions; different Bloom filter groups correspond to different sets of hash functions.

[0156] Figure 7 This is a structural diagram of a server shown as an exemplary embodiment of the present disclosure.

[0157] like Figure 7 As shown, the server provided in this embodiment includes:

[0158] Memory 701;

[0159] Processor 702; and

[0160] Computer programs;

[0161] The computer program is stored in memory 701 and configured to be executed by processor 702 to implement any of the data analysis methods described above.

[0162] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement any of the data analysis methods described above.

[0163] This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described data analysis methods.

[0164] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data analysis method, characterized in that, The method, applied to a MapReduce-based architecture system including a master node, multiple mapper nodes, a counter node, and multiple reducer nodes, includes: The master node obtains the data analysis request sent by the client and obtains the original dataset to be analyzed based on the data analysis request; For each mapper node: the mapper node processes the original dataset to obtain a first key-value pair set; the first key-value pair set includes at least one key-value pair; the key-value pair includes a key value and a value corresponding to the key value; the key value is used to characterize the attribute category of the key-value pair; The counting node determines the input dataset based on the set of all first key-value pairs obtained from all mapper nodes, and inputs the input dataset into a preset Bloom filter cluster. The preset Bloom filter cluster includes at least one Bloom filter group, and the number of Bloom filters in each Bloom filter group does not exceed a first threshold; the total data processing volume of each Bloom filter in each Bloom filter group does not exceed a second threshold; all Bloom filters in the same Bloom filter group share the same set of hash functions; different Bloom filter groups correspond to different sets of hash functions; and the input dataset is processed using the Bloom filters. If it is determined that the initial total processing volume of the current Bloom filter reaches the second threshold, and the number of Bloom filters in the current Bloom filter group has not reached the first threshold, then a next Bloom filter is generated in the current Bloom filter group, and the next Bloom filter is used to continue processing the unprocessed data in the input dataset; and if it is determined that the total processing volume of the current Bloom filter reaches the second threshold, and the number of Bloom filters in the current Bloom filter group reaches the first threshold, then a next Bloom filter group is generated, and a Bloom filter is generated in the next Bloom filter group, and the newly generated Bloom filter is used to continue processing the unprocessed data in the input dataset. Until all data in the input dataset has been processed, the first total number of attribute categories for all key-value pairs in the first key-value pair set is obtained; For any mapper node, if the total number of key-value pairs in the mapper node is less than the first total number, then add a fake key-value pair to obtain a second key-value pair set, and the total number of key-value pairs in the second key-value pair set is equal to the first total number; if the total number of key-value pairs in the mapper node is equal to the first total number, then the mapper node obtains the second key-value pair set from the first key-value pair set. For each mapper node: the mapper node sends the corresponding set of the second key-value pairs to each reducer node; The reducer node determines a target key-value pair set based on the first total number and multiple sets of the second key-value pairs, wherein the total number of key-value pairs in the target key-value pair set is equal to the first total number, and feeds back the target key-value pair set to the client.

2. The method according to claim 1, characterized in that, Before processing the input dataset using the Bloom filter, the method further includes: Initialize a Bloom filter group and a Bloom filter in the input dataset.

3. The method according to claim 2, characterized in that, The parameters of the current Bloom filter group are as follows: in, This represents the i-th Bloom filter group; i is a natural number. express The false positive rate of each Bloom filter; express The grouping misclassification rate; express The size of each Bloom filter segment in the filter; express The number of hash functions in the corresponding hash function set; p represents The expected fill rate of each Bloom filter is set to p=0.5; express Second threshold; This indicates the preset grouping misjudgment rate; This represents the preset reduction factor parameter, where r is less than 1; This represents the first threshold; This indicates the preset Bloom filter slice size; This represents the preset expansion factor parameter. Greater than or equal to 1.

4. A data analysis system based on MapReduce architecture, characterized in that, The system includes a master node, multiple mapper nodes, a counter node, and multiple reducer nodes, including: The master node is used to obtain data analysis requests sent by the client and obtain the original dataset to be analyzed based on the data analysis requests; The mapper node is used to process the original dataset to obtain a first key-value pair set; the first key-value pair set includes at least one key-value pair; the key-value pair includes a key value and a value corresponding to the key value; the key value is used to characterize the attribute category of the key-value pair. The counting node is used to determine the input dataset by analyzing all the first key-value pairs obtained by all mapper nodes and inputting the input dataset into a preset Bloom filter cluster. The preset Bloom filter cluster includes at least one Bloom filter group, and the number of Bloom filters in each Bloom filter group does not exceed a first threshold. The total data processing volume of each Bloom filter in each Bloom filter group does not exceed a second threshold. All Bloom filters in the same Bloom filter group share the same set of hash functions. Different Bloom filter groups correspond to different sets of hash functions. The input dataset is processed using the Bloom filters. This process is repeated until the initial total processing volume of the current Bloom filter reaches the second threshold, and when... If the number of Bloom filters in the previous Bloom filter group has not reached the first threshold, then the next Bloom filter is generated in the current Bloom filter group, and the next Bloom filter is used to continue processing the unprocessed data in the input dataset; and if it is determined that the total amount of data processed by the current Bloom filter has reached the second threshold, and the number of Bloom filters in the current Bloom filter group has reached the first threshold, then the next Bloom filter group is generated, and a Bloom filter is generated in the next Bloom filter group, and the newly generated Bloom filter is used to continue processing the unprocessed data in the input dataset; until all the data in the input dataset has been processed, the first total number of attribute categories of all key-value pairs in all the first key-value pair sets is obtained; The mapper node is further configured to, if the total number of key-value pairs in the mapper node is less than the first total number, add dummy key-value pairs to obtain a second key-value pair set, wherein the total number of key-value pairs in the second key-value pair set is equal to the first total number; if the total number of key-value pairs in the mapper node is equal to the first total number, the mapper node obtains the first key-value pair set as the second key-value pair set; and send the corresponding second key-value pair set to each of the reducer nodes respectively. The reducer node is configured to determine a target key-value pair set based on the first total number and multiple second key-value pair sets, wherein the total number of key-value pairs in the target key-value pair set is equal to the first total number, and to feed back the target key-value pair set to the client.

5. A server, characterized in that, Includes memory and processor; among which, The memory is used to store computer programs; The processor is configured to read a computer program stored in the memory and execute the method described in any one of claims 1-3 according to the computer program in the memory.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-3.

Images