Frequent data mining method and related equipment based on hierarchical federated system

Abstract

The embodiment of the present application discloses a frequent data mining method and related equipment based on a hierarchical federated system, which is used to reduce the risk of single point failure, improve the utility and scalability of the federated system, and improve the fault tolerance and accuracy of frequent data mining. The method of the embodiment of the present application includes: receiving candidate frequent data sent by each auxiliary service end; the candidate frequent data is determined by the corresponding auxiliary server according to local disturbance data, and the local disturbance data includes disturbance data sent by the corresponding user end; the importance index of each candidate frequent data is calculated according to a preset importance formula; according to the importance index of each candidate frequent data, k target frequent data is determined from multiple candidate frequent data, and k is the expected number of target frequent data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data mining, and in particular to a frequent data mining method and related equipment based on a hierarchical federated system. Background Art

[0002] Federated Analytics (FA) allows the federated server to analyze data information from multiple users without the original data of the participants leaving the local server. The user needs to process the original data and send it to the federated server. In order to protect the user's data privacy in a semi-trusted execution environment, the user can perturb the data through the LDP algorithm. Under the premise of protecting the user's data privacy, the federated server performs statistical analysis on the perturbed data from the user and obtains valuable information, such as the top k frequent data.

[0003] In existing technical solutions, a federated server usually collects and analyzes disturbance data from the user end. However, this data mining method has the risk of single point failure. In addition, the communication and interaction between a federated server and a large number of user ends will generate a large system delay, affecting the efficiency of frequent data mining. Summary of the invention

[0004] The embodiment of the present application provides a frequent data mining method and related equipment based on a hierarchical federated system, which are used to reduce the risk of single point failure.

[0005] A first aspect of an embodiment of the present application provides a frequent data mining method based on a hierarchical federated system, comprising:

[0006] Receiving candidate frequent data sent by each auxiliary server; the candidate frequent data is determined by the corresponding auxiliary server according to local disturbance data, and the local disturbance data includes disturbance data sent by the corresponding user terminal;

[0007] Calculate the importance index of each candidate frequent data according to the preset importance formula;

[0008] According to the importance index of each candidate frequent data, k target frequent data are determined from the multiple candidate frequent data, where k is the expected number of target frequent data.

[0009] In a specific implementation, calculating the importance index of each candidate frequent data according to a preset importance formula includes:

[0010] An importance index of each candidate frequent data is calculated according to the frequency density of each candidate frequent data among the plurality of candidate frequent data and the expected number.

[0011] In a specific implementation, determining the importance index of each candidate frequent data according to the frequency density of each candidate frequent data among the multiple candidate frequent data and the expected number includes:

[0012] The importance index of each candidate frequent data is calculated according to the following formula:

[0013] α(c)=tanh(0.2k+f(c))

[0014] Wherein, α(c) is the importance index of the c-th candidate frequent data, and f(c) is the frequency density of the c-th candidate frequent data among the multiple candidate frequent data.

[0015] In a specific implementation, determining k target frequent data from a plurality of candidate frequent data according to the importance index of each candidate frequent data includes:

[0016] Arrange each candidate frequent data in descending order according to the corresponding importance index to obtain a candidate frequent data sequence;

[0017] Determining each valid candidate frequent data in the candidate frequent data sequence as the candidate frequent data to be processed in turn from first to last, until the number of target frequent data is equal to the expected number, wherein the valid candidate frequent data is any candidate frequent data in the plurality of candidate frequent data except invalid data;

[0018] Calculating a similarity index between the candidate frequent data to be processed and a subsequent candidate frequent data, the next candidate frequent data of the candidate frequent data to be processed in the candidate frequent data sequence being the subsequent candidate frequent data;

[0019] If the similarity index is less than a preset similarity threshold, the subsequent candidate frequent data is determined as invalid data, and the candidate frequent data to be processed is determined as target frequent data.

[0020] In a specific implementation, the similarity index is a Hamming distance, and the method further includes:

[0021] Determining the bit length of the candidate frequent data to be processed;

[0022] The product of the bit length of the candidate frequent data to be processed and the preset similarity ratio is used as the preset similarity threshold.

[0023] In a specific implementation, determining k target frequent data from a plurality of candidate frequent data according to the importance index of each candidate frequent data includes:

[0024] The first k candidate frequent data with the highest corresponding importance index among multiple candidate frequent data are determined as target frequent data.

[0025] In a specific implementation, the importance index includes an importance index based on frequency or an importance index based on frequency density.

[0026] A second aspect of an embodiment of the present application provides a computer device, including:

[0027] A receiving unit, configured to receive candidate frequent data sent by each auxiliary server; the candidate frequent data is determined by the corresponding auxiliary server according to local disturbance data, and the local disturbance data includes disturbance data sent by the corresponding user terminal;

[0028] A calculation unit, used to calculate the importance index of each candidate frequent data according to a preset importance formula;

[0029] The determining unit is used to determine k target frequent data from multiple candidate frequent data according to the importance index of each candidate frequent data, where k is the expected number of target frequent data.

[0030] In a specific implementation manner, the calculation unit is specifically configured to calculate the importance index of each candidate frequent data according to the frequency density of each candidate frequent data among the multiple candidate frequent data and the expected number.

[0031] In a specific implementation, the calculation unit is specifically used to calculate the importance index of each candidate frequent data according to the following formula:

[0032] α(c)=tanh(0.2k+f(c))

[0033] Wherein, α(c) is the importance index of the c-th candidate frequent data, and f(c) is the frequency density of the c-th candidate frequent data among the multiple candidate frequent data.

[0034] In a specific implementation, the determining unit is specifically used to arrange each candidate frequent data in descending order according to the corresponding importance index to obtain a candidate frequent data sequence;

[0035] Determining each valid candidate frequent data in the candidate frequent data sequence as the candidate frequent data to be processed in turn from first to last, until the number of target frequent data is equal to the expected number, wherein the valid candidate frequent data is any candidate frequent data in the plurality of candidate frequent data except invalid data;

[0036] Calculating a similarity index between the candidate frequent data to be processed and a subsequent candidate frequent data, the next candidate frequent data of the candidate frequent data to be processed in the candidate frequent data sequence being the subsequent candidate frequent data;

[0037] If the similarity index is less than a preset similarity threshold, the subsequent candidate frequent data is determined as invalid data, and the candidate frequent data to be processed is determined as target frequent data.

[0038] In a specific implementation, the similarity index is the Hamming distance.

[0039] The determining unit is further used to determine the bit length of the candidate frequent data to be processed;

[0040] The determining unit is further configured to use the product of the bit length of the candidate frequent data to be processed and a preset similarity ratio as the preset similarity threshold.

[0041] In a specific implementation, the determining unit is specifically configured to determine top k candidate frequent data having the highest corresponding importance index among the plurality of candidate frequent data as target frequent data.

[0042] In a specific implementation, the importance index includes an importance index based on frequency or an importance index based on frequency density.

[0043] A third aspect of an embodiment of the present application provides a computer device, including:

[0044] CPU, memory and input / output interface;

[0045] The memory is a short-term storage memory or a persistent storage memory;

[0046] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect.

[0047] A fourth aspect of the embodiments of the present application provides a computer program product comprising instructions, and when the computer program product is run on a computer, the computer is caused to execute the method described in the first aspect.

[0048] A fifth aspect of an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect.

[0049] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: a hierarchical federation system is adopted, in which the federation server communicates only with a small number of auxiliary servers, and each auxiliary server communicates with a large number of corresponding user terminals. Compared with the traditional federation system architecture, that is, a method in which one federation server communicates with each user terminal, the risk of single point failure can be effectively reduced. At the same time, this hierarchical design greatly reduces the communication pressure of the federation server and improves the utility and scalability of the federation analysis system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A system architecture diagram of the frequent data mining method disclosed in the embodiment of the present application;

[0051] Figure 2 A flowchart of a frequent data mining method disclosed in an embodiment of the present application;

[0052] Figure 3 Another schematic diagram of the frequent data mining process disclosed in the embodiment of the present application;

[0053] Figure 4 A schematic diagram of the Hamming similarity measurement disclosed in the embodiment of the present application;

[0054] Figure 5 A schematic diagram of the structure of a computer device disclosed in an embodiment of the present application;

[0055] Figure 6 Another structural schematic diagram of the computer device disclosed in the embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] The embodiment of the present application provides a frequent data mining method and related equipment based on a hierarchical federated system, which are used to reduce the risk of single point failure.

[0058] See also Figure 1In order to better implement the frequent data mining method of the embodiment of the present application, the embodiment of the present application provides a hierarchical federation system for reducing the risk of single point failure. The hierarchical federation system includes at least one auxiliary server 101, multiple user terminals 102 and a federal server 103, wherein each auxiliary server 101 and multiple clients corresponding to the auxiliary server 101 constitute a group, and each group is managed by a unique auxiliary server 101 in the group and the auxiliary server 101 communicates and interacts with each user terminal 102 in the group. When frequent data mining is required, the user terminal 102 will use a local differential privacy algorithm to perturb the local original data, and send the perturbed data to the auxiliary server 101 of the group. Again, the auxiliary server 101 performs statistical analysis based on the perturbed data collected from each user terminal 102, obtains candidate frequent data of the group (i.e., candidate frequent data set), and sends the candidate frequent data to the federal server 103. Finally, the federal server 103 can further distinguish the interference data by importance index and similarity measurement based on the candidate frequent data (ie, candidate frequent data set Ω1) from each auxiliary server to obtain the top k target frequent data (ie, candidate frequent data set Ω2).

[0059] Through the hierarchical federated analysis system, while ensuring the privacy of user local data, the communication pressure of the federated server is reduced, and the utility and scalability of the federated analysis system are improved. In addition, the present invention proposes an aggregated federated analysis method using importance index and Hamming similarity measurement for frequent data mining, thereby improving the fault tolerance of the federated server analysis algorithm and the accuracy of the analysis results.

[0060] Based on the above hierarchical federation system, see Figure 2 The embodiment of the present application provides a frequent data mining method, which is performed by the aforementioned auxiliary service end 101, the user end 102 and / or the federal server 103, and includes the following steps:

[0061] 201. Receive candidate frequent data sent by each auxiliary server. The candidate frequent data is determined by the corresponding auxiliary server according to local disturbance data, and the local disturbance data includes disturbance data sent by the corresponding user terminal.

[0062] When frequent data mining is required, each user terminal will send data perturbed by the local differential privacy algorithm, i.e., perturbed data, to its corresponding auxiliary server, that is, the auxiliary server in the same group. Subsequently, each auxiliary server will perform statistical analysis on the perturbed data received from each user terminal to obtain candidate frequent data in the corresponding group. For details, any frequent data mining scheme based on perturbed data can be referred to here, and is not limited here. Finally, the federal server can receive the candidate frequent data obtained after statistical analysis by each auxiliary server, and determine the target frequent data from them.

[0063] 202. Calculate the importance index of each candidate frequent data according to a preset importance formula.

[0064] In order to select more representative data that can better represent the data features of the multiple candidate frequent data from the multiple candidate frequent data obtained in step 201 as the target frequent data, the embodiment of the present application uses an importance index to evaluate the importance of each candidate frequent data among the multiple candidate frequent data.

[0065] Specifically, the importance index here includes but is not limited to an importance index based on frequency or an importance index based on frequency density.

[0066] In some specific implementations, if the importance index is an importance index based on frequency density, the importance index of each candidate frequent data can be calculated according to the frequency density and expected number of each candidate frequent data in multiple candidate frequent data. The expected number is the number (or number) of target frequent data expected to be obtained by the federated server.

[0067] For details, please refer to the formula

[0068] α(c)=tanh(0.2k+f(c))

[0069] Wherein, α(c) is the importance index of the cth candidate frequent data, and f(c) is the frequency density of the cth candidate frequent data in the multiple candidate frequent data. Wherein, 0.2 in the formula 0 can also be other decimals set as required, such as 0.4, which is not specifically limited here.

[0070] If the importance index is a frequency-based importance index, then the importance index of each candidate frequent data may be calculated according to the frequency of each candidate frequent data among multiple candidate frequent data.

[0071] For details, please refer to the formula

[0072] α(c)=tanh(f(c))

[0073] Wherein, α(c) is the importance index of the c-th candidate frequent data, and f(c) is the frequency of the c-th candidate frequent data among the multiple candidate frequent data.

[0074] 203. According to the importance index of each candidate frequent data, k target frequent data are determined from multiple candidate frequent data, where k is the expected number of target frequent data.

[0075] After calculating the importance index of each candidate frequent data in step 202, k target frequent data can be determined from multiple candidate frequent data based on the importance index. In some specific implementations, the first k candidate frequent data with higher corresponding importance indexes among multiple candidate frequent data can be simply used as target frequent data.

[0076] In some other implementations, in order to ensure the representativeness of the selected target frequent data as much as possible, this step can also be implemented in the following way: arrange each candidate frequent data in descending order according to the corresponding importance index to obtain a candidate frequent data sequence; determine each valid candidate frequent data as the candidate frequent data to be processed in turn from first to last until the number of target frequent data is equal to the expected number, and the valid candidate frequent data is any candidate frequent data except invalid data among the multiple candidate frequent data; calculate the similarity index between the candidate frequent data to be processed and the next candidate frequent data, and the next candidate frequent data of the candidate frequent data to be processed in the candidate frequent data sequence is the next candidate frequent data; if the similarity index is less than the preset similarity threshold, the next candidate frequent data is determined as invalid data, and the candidate frequent data to be processed is determined as the target frequent data.

[0077] Specifically, after constructing a candidate frequent data sequence arranged in descending order according to the corresponding importance index, each valid candidate frequent data in the sequence is taken as the candidate frequent data to be processed in turn from the first to the last (that is, from the high to the low importance index). The similarity index between the candidate frequent data to be processed and its next candidate frequent data, that is, the latter candidate frequent data, is calculated. In order to reflect the overall data characteristics of multiple candidate frequent data as comprehensively as possible when the number of target frequent data is limited, if the similarity index is less than the preset similarity threshold, the one with the higher importance index is retained, that is, the candidate frequent data to be processed is determined as the target frequent data, and the latter candidate frequent data is determined as invalid data (because the latter candidate frequent data is relatively similar to the candidate frequent data to be processed, and under overall consideration, it is hoped that each target frequent data determined can represent the data characteristics of a class / medium data, and the importance index of the latter candidate frequent data is lower, so it is invalid).

[0078] The aforementioned similarity index may be any similarity index, such as Euclidean distance, Hamming distance, etc., which is not specifically limited in this embodiment.

[0079] In order to better illustrate the concept of the next candidate frequent data, an example is given below: If the candidate frequent data sequence is {a, b, c, d}, then when the candidate frequent data to be processed is a, then the corresponding next candidate frequent data is b.

[0080] Furthermore, the aforementioned preset similarity threshold may be a preset specific value, or the product of the bit length of the candidate frequent data to be processed and a preset similarity ratio (a preset specific value) may be used as the preset similarity threshold, which is not specifically limited in this embodiment.

[0081] In the embodiments of the present application, first, a hierarchical federated system is adopted, which can effectively reduce the risk of single point failure compared with the traditional federal system architecture, that is, a method in which a federal server communicates with each user terminal. Second, through the hierarchical architecture design, the federal server only needs to communicate and interact with a small number of auxiliary servers, and each auxiliary server communicates and interacts with a large number of user terminals, which greatly reduces the communication pressure of the federal server and improves the utility and scalability of the federal analysis system. Third, based on the frequent data mining method proposed using importance indicators and Hamming similarity metrics, the federal server can aggregate the prediction results of the auxiliary servers (i.e., candidate frequent data) and distinguish interference data (i.e., invalid data), thereby improving the fault tolerance of the federal analysis algorithm and the accuracy of the analysis results.

[0082] The above describes various embodiments of frequent data mining in this application, please refer to Figure 3 , the frequent data mining method of the embodiment of the present application is described below in some specific scenarios.

[0083] The federated server is responsible for aggregating and analyzing the auxiliary server S τ The candidate frequent data Ω sent τ , and obtain the final prediction result, that is, the first k target frequent data. There are a total of τ groups, that is, τ auxiliary servers.

[0084] 1. The federated server aggregates the frequent data prediction results from the auxiliary servers.

[0085] 1a. Define an importance index α(c) = tanh(0.2k+f(c)), where k is the number of target frequent data expected to be obtained; c refers to the cth candidate frequent data, that is, c∈Ω τ ; f(c) is c in Ω τ The frequency density in , f(c)∈(0,1].

[0086] 1b. Calculate and count the importance index of all candidate frequent data, that is, ∑ Ωτ α(c).

[0087] 1c. The federated server sorts the candidate frequent data in descending order according to their importance indexes and obtains Ω1 (i.e., the candidate frequent data sequence).

[0088] 2. Ω1 obtained by the federated server analysis:

[0089] 2a. Create Ω2 to store the target frequent data and initialize the validity of all candidate frequent data to true.

[0090] 2b. Traverse the candidate frequent data in Ω1, c i =Ω1[i].

[0091] 2c. For effective c i (i.e. candidate frequent data to be processed) and candidate frequent data with lower importance (i.e. the next candidate frequent data) c j (j=i+1,…), calculate the Hamming distance d(c i ,c j ), that is, the number of different bits in the bit string corresponding to the two candidate frequent data, see Figure 4 Schematic diagram of Hamming similarity calculation. Set the Hamming similarity measurement threshold (i.e., preset similarity threshold) δ = 0.5*len(c i ), where len(c i ) is c i In the business scenario of the embodiment of the present application, if the number of bits in each corresponding bit string of two candidate frequent data is the same, the expected value of the ratio of the Hamming distance to the maximum length of the bit string (i.e. ) is greater than 0.5. Therefore, in order to identify the differences of all items, in the business scenario of the embodiment of the present application, 0.5 can be used as the Hamming similarity measurement threshold.

[0092] (1) If d(c i ,c j )<δ, then c j can be considered as c i The interference term is c j The importance index value of c is lower, so c j The validity is changed to false.

[0093] (2) c i Add to Ω2 until the number of target frequent data in Ω2 is k.

[0094] 3. The target frequent data in Ω2 is to predict the top k frequent data among multiple user terminals.

[0095] It should be noted that when the disturbance data provided by the user end comes from the user operation data, the disturbance data, candidate frequent data, target frequent data and other related data involved in the embodiments of the present application are all obtained after the user's authorization. In addition, when the embodiments of the present application are applied to specific products or technologies, the data involved in the use need to obtain the user's permission or consent, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0096] See also Figure 5 , an embodiment of the present application provides a computer device, including:

[0097] The receiving unit 501 is used to receive candidate frequent data sent by each auxiliary server; the candidate frequent data is determined by the corresponding auxiliary server according to local disturbance data, and the local disturbance data includes disturbance data sent by the corresponding user terminal;

[0098] A calculation unit 502 is used to calculate the importance index of each candidate frequent data according to a preset importance formula;

[0099] The determining unit 503 is configured to determine k target frequent data from the plurality of candidate frequent data according to the importance index of each candidate frequent data, where k is the expected number of the target frequent data.

[0100] In a specific implementation, the calculation unit 502 is specifically configured to calculate the importance index of each candidate frequent data according to the frequency density and expected number of each candidate frequent data among multiple candidate frequent data.

[0101] In a specific implementation, the calculation unit 502 is specifically configured to calculate the importance index of each candidate frequent data according to the following formula:

[0102] α(c)=tanh(0.2k+f(c))

[0103] Among them, α(c) is the importance index of the c-th candidate frequent data, and f(c) is the frequency density of the c-th candidate frequent data among multiple candidate frequent data.

[0104] In a specific implementation, the determining unit 503 is specifically configured to arrange each candidate frequent data in descending order according to the corresponding importance index to obtain a candidate frequent data sequence;

[0105] From the first to the last, each valid candidate frequent data in the candidate frequent data sequence is determined as the candidate frequent data to be processed in turn until the number of target frequent data is equal to the expected number, and the valid candidate frequent data is any candidate frequent data except the invalid data among the multiple candidate frequent data;

[0106] Calculate the similarity index between the candidate frequent data to be processed and the next candidate frequent data, and the next candidate frequent data of the candidate frequent data to be processed in the candidate frequent data sequence is the next candidate frequent data;

[0107] If the similarity index is less than the preset similarity threshold, the next candidate frequent data is determined as invalid data, and the candidate frequent data to be processed is determined as the target frequent data.

[0108] In a specific implementation, the similarity index is the Hamming distance.

[0109] The determination unit 503 is further used to determine the bit length of the candidate frequent data to be processed;

[0110] The determining unit 503 is further configured to use the product of the bit length of the candidate frequent data to be processed and the preset similarity ratio as a preset similarity threshold.

[0111] In a specific implementation, the determining unit 503 is specifically configured to determine the first k candidate frequent data with the highest corresponding importance index among the plurality of candidate frequent data as the target frequent data.

[0112] In a specific implementation, the importance index includes an importance index based on frequency or an importance index based on frequency density.

[0113] Figure 6 6 is a schematic diagram of a computer device structure provided in an embodiment of the present application. The computer device 600 may include one or more central processing units (CPU) 601 and a memory 605. The memory 605 stores one or more application programs or data.

[0114] The memory 605 may be a volatile storage or a persistent storage. The program stored in the memory 605 may include one or more modules, each of which may include a series of instruction operations in the computer device. Furthermore, the central processing unit 601 may be configured to communicate with the memory 605 and execute a series of instruction operations in the memory 605 on the computer device 600.

[0115] The computer device 600 may also include one or more power supplies 602, one or more wired or wireless network interfaces 603, one or more input and output interfaces 604, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0116] The CPU 601 can execute the aforementioned Figures 1 to 5 The operations performed by the computer device in the illustrated embodiment will not be described in detail here.

[0117] It should be noted that, although the steps in the flowcharts involved in the embodiments are drawn in sequence according to the instructions of the arrows, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0119] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0122] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

[0123] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the frequent data mining method based on the hierarchical federated system as described above.

Claims

1. A frequent data mining method based on a hierarchical federated system, characterized in that: The method comprises: Receiving candidate frequent data sent by each auxiliary server; the candidate frequent data is determined by the corresponding auxiliary server according to local disturbance data, and the local disturbance data includes disturbance data sent by the corresponding user terminal; Calculate the importance index of each candidate frequent data according to the preset importance formula; According to the importance index of each candidate frequent data, k target frequent data are determined from multiple candidate frequent data, where k is the expected number of target frequent data; The step of determining k target frequent data from a plurality of candidate frequent data according to the importance index of each candidate frequent data comprises: Arrange each candidate frequent data in descending order according to the corresponding importance index to obtain a candidate frequent data sequence; Determining each valid candidate frequent data in the candidate frequent data sequence as candidate frequent data to be processed in turn from first to last, until the number of target frequent data is equal to the expected number, wherein the valid candidate frequent data is any candidate frequent data except invalid data in the plurality of candidate frequent data; Calculating a similarity index between the candidate frequent data to be processed and a subsequent candidate frequent data, the next candidate frequent data of the candidate frequent data to be processed in the candidate frequent data sequence being the subsequent candidate frequent data; If the similarity index is less than a preset similarity threshold, the subsequent candidate frequent data is determined as invalid data, and the candidate frequent data to be processed is determined as target frequent data.

2. The method according to claim 1, characterized in that The step of calculating the importance index of each candidate frequent data according to a preset importance formula includes: An importance index of each candidate frequent data is calculated according to the frequency density of each candidate frequent data among the plurality of candidate frequent data and the expected number.

3. The method according to claim 2, characterized in that The determining, according to the frequency density of each candidate frequent data among the plurality of candidate frequent data and the expected number, an importance index of each candidate frequent data comprises: The importance index of each candidate frequent data is calculated according to the following formula: α(c)=tanh(0.2k+f(c)) Wherein, α(c) is the importance index of the c-th candidate frequent data, and f(c) is the frequency density of the c-th candidate frequent data among the multiple candidate frequent data.

4. The method according to claim 1, characterized in that The similarity index is the Hamming distance, and the method further includes: Determining the bit length of the candidate frequent data to be processed; The product of the bit length of the candidate frequent data to be processed and the preset similarity ratio is used as the preset similarity threshold.

5. The method according to claim 1, characterized in that The step of determining k target frequent data from a plurality of candidate frequent data according to the importance index of each candidate frequent data comprises: The first k candidate frequent data with the highest corresponding importance index among multiple candidate frequent data are determined as target frequent data.

6. The method according to any one of claims 1 to 5, characterized in that The importance index includes an importance index based on frequency or an importance index based on frequency density.

7. A computer device, characterized in that: include: A receiving unit, configured to receive candidate frequent data sent by each auxiliary server; The candidate frequent data is determined by the corresponding auxiliary server according to the local disturbance data, and the local disturbance data includes the disturbance data sent by the corresponding user terminal; A calculation unit, used to calculate the importance index of each candidate frequent data according to a preset importance formula; A determination unit is used to arrange each candidate frequent data in descending order according to the corresponding importance index to obtain a candidate frequent data sequence; and to determine each valid candidate frequent data in the candidate frequent data sequence as a candidate frequent data to be processed in turn from the first to the last, until the number of target frequent data is equal to the expected number, wherein the valid candidate frequent data is any candidate frequent data except invalid data among the multiple candidate frequent data; Calculating a similarity index between the candidate frequent data to be processed and a subsequent candidate frequent data, the next candidate frequent data of the candidate frequent data to be processed in the candidate frequent data sequence being the subsequent candidate frequent data; If the similarity index is less than a preset similarity threshold, the subsequent candidate frequent data is determined as invalid data, and the candidate frequent data to be processed is determined as target frequent data.

8. A computer device, characterized in that: include: CPU, memory and input / output interface; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that: The computer storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images