Data relationship discovery and mining method, device and equipment based on Spark framework and medium

Abstract

The method, device, equipment and medium for discovering and mining data relations based on a Spark framework provided by the embodiments of the present disclosure comprise: processing initial transaction data sets based on a compressed frequent pattern tree algorithm to obtain target transaction data sets; assigning data of compressed frequent pattern trees corresponding to different transactions to a computing node according to the target transaction data sets; mining the association relations of data items included in the compressed frequent pattern trees corresponding to each computing node based on a frequent pattern tree mining algorithm to determine the association relations corresponding to each computing node; and merging the association relations corresponding to each computing node according to the association relations corresponding to each computing node to obtain target association relations of the original transaction data sets. The comprehensive extraction and efficient association analysis and mining of massive normalized target network data feature elements are realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology and related technical fields, specifically to a method, apparatus, device, and medium for data relationship discovery and mining based on the Spark framework. Background Technology

[0002] Currently, network data in specific regions across different industries such as education, law, and catering exhibits characteristics such as diverse data structures and types, incompatible storage methods, differing sharing interface standards, and large data scale.

[0003] Existing technologies for traditional data relationship discovery and mining suffer from problems such as limited data capacity, low data processing efficiency, unsatisfactory data mining results, and poor integrity of tag data features.

[0004] Given the problems with existing technologies, there is an urgent need to propose a data relationship discovery and mining method based on the Spark framework. Summary of the Invention

[0005] The embodiments described herein provide a method, apparatus, device, and medium for data relationship discovery and mining based on the Spark framework, addressing the problems existing in the prior art.

[0006] Firstly, based on the content of this disclosure, a method for data relationship discovery and mining based on the Spark framework is provided, including: The initial transaction dataset is processed based on the compressed frequent pattern tree algorithm to obtain the target transaction dataset. The initial transaction dataset includes network data from different domains and multi-source heterogeneous network data. The target transaction dataset consists of compressed frequent pattern tree data corresponding to different transactions. Based on the target transaction dataset, the data of the compressed frequent pattern tree corresponding to different transactions are allocated to a computing node. The frequent pattern tree mining algorithm is used to mine the association relationships of the data items included in the compressed frequent pattern tree corresponding to each computing node, and to determine the association relationships corresponding to each computing node. Based on the association relationships corresponding to each computing node, the association relationships corresponding to each computing node are merged to obtain the target association relationships of the original transaction dataset.

[0007] In some embodiments of this disclosure, the process of processing the initial transaction dataset based on the compressed frequent pattern tree algorithm to obtain the target transaction dataset includes: Obtain the transactions, the data items included in each transaction, and the frequency values ​​of the data items included in each transaction from the initial transaction dataset; Based on the frequency values ​​of the data items included in the transaction, filter out the data items whose frequency values ​​meet the preset threshold; After sorting the selected data items in descending order by frequency value, a compressed frequent pattern tree corresponding to different transactions is constructed. The data from the compressed frequent pattern trees corresponding to different transactions constitute the target transaction dataset.

[0008] In some embodiments of this disclosure, the step of allocating the data of the compressed frequent pattern tree corresponding to different transactions to a computing node according to the target transaction dataset includes: The compressed frequent pattern tree included in the target transaction dataset is decomposed into a path list to obtain the path list included in the compressed frequent pattern tree, wherein the path list includes a path, a data item included in the path, and a frequency value corresponding to the data item included in the path. The path list of each compressed frequent pattern tree includes the data items and the corresponding frequency values ​​of the data items, which are then assigned to the corresponding computing nodes.

[0009] In some embodiments of this disclosure, the allocation of data items and corresponding frequency values ​​of the path lists of each compressed frequent pattern tree to the corresponding computing nodes includes: Obtain the status information of each computing node; Based on the path count information and the relationship between the average path length and the computing node status information included in the path list of each compression frequent pattern tree, the computing nodes corresponding to each compression frequent pattern tree are determined. The path list of each compressed frequent pattern tree includes the data items and the corresponding frequency values ​​of the data items, which are then assigned to the corresponding computing nodes.

[0010] In some embodiments of this disclosure, the method of mining the association relationships of the data items included in the compressed frequent pattern tree corresponding to each computing node based on the frequent pattern tree mining algorithm to determine the association relationships corresponding to each computing node includes: Based on the frequent pattern tree mining algorithm, the conditional pattern base and conditional pattern tree of the data items included in each computing node are determined; Based on the conditional schema base and conditional schema tree of the data items included in each computing node, the association relationship of the data items corresponding to each computing node is determined.

[0011] In some embodiments of this disclosure, determining the association relationship of data items corresponding to each computing node based on the conditional schema base and conditional schema tree of the data items included in each computing node includes: The FP-Growth algorithm is recursively executed on the conditional pattern tree to generate a set of data items containing any target data item. Based on the data item set containing the target data item, determine the confidence level of each rule in the data item set; Based on the confidence level of each rule in the data item set and the conditional pattern base of the data items included in each computing node, the association relationship of the data items corresponding to each computing node is determined.

[0012] In some embodiments of this disclosure, the step of merging the association relationships corresponding to each computing node based on the association relationships corresponding to each computing node to obtain the target association relationship of the original transaction dataset includes: Based on the association relationships corresponding to each computing node, the same association relationships of the same data item included in different computing nodes are deduplicated. Based on the frequency value of the data item corresponding to each computing node, the frequency values ​​of the same data item included in different computing nodes are normalized.

[0013] Secondly, based on the content of this disclosure, a data relationship discovery and mining device based on the Spark framework is provided, including: The target data determination module is used to process the initial transaction dataset based on the compressed frequent pattern tree algorithm to obtain the target transaction dataset. The initial transaction dataset includes network data from different domains and multi-source heterogeneous network data. The target transaction dataset is composed of compressed frequent pattern tree data corresponding to different transactions. The data allocation module is used to allocate the data of the compressed frequent pattern tree corresponding to different transactions to a computing node according to the target transaction dataset; The association determination module is used to mine the associations of the data items included in the compressed frequent pattern tree corresponding to each computing node based on the frequent pattern tree mining algorithm, and to determine the associations of each computing node. The target association determination module is used to merge the associations corresponding to each computing node based on the associations corresponding to each computing node, so as to obtain the target associations of the original transaction dataset.

[0014] Thirdly, according to the present disclosure, a computer device is provided, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any of the first aspects.

[0015] Fourthly, according to the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described in any of the first aspects.

[0016] The data relationship discovery and mining method, apparatus, device, and medium based on the Spark framework provided in this disclosure first process an initial transaction dataset using a compressed frequent pattern tree algorithm to obtain a target transaction dataset. Then, based on the target transaction dataset, the data from the compressed frequent pattern trees corresponding to different transactions are allocated to a computing node. Next, based on a frequent pattern tree mining algorithm, the data items included in the compressed frequent pattern trees corresponding to each computing node are used to mine association relationships, determining the association relationships corresponding to each computing node. Finally, based on the association relationships corresponding to each computing node, the association relationships corresponding to each computing node are merged to obtain the target association relationships of the original transaction dataset. The data relationship discovery and mining method based on the Spark framework can achieve comprehensive extraction and efficient association analysis and mining of feature elements from massive standardized target network data, based on the unified processing and standardized description of multi-source heterogeneous target network data such as Internet data resources, cryptographic resources, publicly available software tool resources, proprietary software tool resources, and technical knowledge bases.

[0017] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein: Figure 1 This is a flowchart illustrating a data relationship discovery and mining method based on the Spark framework provided in this embodiment of the disclosure; Figure 2 This is a schematic diagram of the structure of a data relationship discovery and mining device based on the Spark framework provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure.

[0019] In the accompanying diagram, markers with the same last two digits correspond to the same elements. It should be noted that the elements in the diagram are schematic and not drawn to scale. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.

[0021] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, the statement of “connecting” or “coupling” two or more parts together shall mean that these parts are directly joined together or joined through one or more intermediate components.

[0022] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of the phrase "embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists, A and B exist simultaneously, or B exists. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] Furthermore, in all embodiments of this disclosure, terms such as “first” and “second” are used only to distinguish one component (or part of a component) from another component (or another part of a component).

[0025] In the description of this application, unless otherwise stated, "multiple" means two or more (including two), and similarly, "multiple groups" means two or more (including two groups).

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0027] In view of the problems existing in the prior art, this disclosure provides a data relationship discovery and mining method based on the Spark framework. Figure 1 This is a flowchart illustrating a data relationship discovery and mining method based on the Spark framework provided in this disclosure embodiment, as follows: Figure 1 As shown, the specific process of data relationship discovery and mining based on the Spark framework includes: S110. The initial transaction dataset is processed based on the compressed frequent pattern tree algorithm to obtain the target transaction dataset.

[0028] The initial transaction dataset includes network data from different domains and heterogeneous network data from multiple sources, while the target transaction dataset consists of compressed frequent pattern trees corresponding to different transactions.

[0029] In a specific implementation, the initial transaction dataset is processed based on the compressed frequent pattern tree algorithm to obtain the target transaction dataset, including: obtaining the transactions, the data items included in the transactions, and the frequency values ​​of the data items included in the transactions in the initial transaction dataset; filtering out target data items whose frequency values ​​meet a preset threshold based on the frequency values ​​of the data items included in the transactions; and constructing compressed frequent pattern trees corresponding to different transactions after sorting the target data items in descending order according to their frequency values. The data of the compressed frequent pattern trees corresponding to different transactions constitute the target transaction dataset.

[0030] The Compressed Frequent Pattern Tree (CSFP-tree) algorithm is an efficient data mining technique for mining frequent itemsets. Its core idea is to compress the database by constructing a frequent pattern tree and frequently mining recursive itemsets on this tree.

[0031] In the data relationship discovery and mining method based on the Spark framework provided in this embodiment, the transactions, data items included in the transactions, and frequency values ​​of the data items included in the transactions are first obtained from the initial transaction dataset. Here, a transaction is regarded as an itemset, and an itemset contains multiple data items. The frequency value of the data item reflects the number of times the data item appears in the dataset, which is also the support.

[0032] After obtaining the transactions, data items included in the transactions, and frequency values ​​of the data items included in the transactions from the initial transaction dataset, data items whose frequency values ​​meet the preset threshold are selected from the data items based on the frequency values ​​of each data item. Then, the selected data items are sorted in descending order according to their frequency values ​​to form an item header table, and a compressed frequent pattern tree is constructed.

[0033] In the specific implementation process, for each transaction, the order of data items in the transaction is rearranged according to the arrangement order of data items in the item header table, and data items not in the item header table are deleted. A compressed frequent pattern tree is generated based on the processed data items. The implementation logic for generating the compressed frequent pattern tree based on the processed data items is as follows: if a data item already exists in the compressed frequent pattern tree, the count of that node is incremented; if a data item does not exist, a new node is created and linked to the compressed frequent pattern tree. At the same time, a head pointer list is maintained for the itemsets corresponding to the transactions included in the compressed frequent pattern tree to facilitate subsequent operations.

[0034] In this implementation, the Compressed Frequent Pattern Tree (CSFP-tree) technique is used to preprocess the initial transaction dataset. By scanning the initial transaction dataset, the transactions, the data items included in the transactions, and the frequency values ​​of the data items included in the transactions are determined. Infrequent items are deleted through pruning strategies (i.e., data items whose frequency values ​​do not meet the preset threshold are removed), generating a Compressed Frequent Pattern Tree with a smaller space footprint, effectively reducing subsequent data transmission overhead. Then, the preprocessed Compressed Frequent Pattern Tree is serialized and stored in HDFS (Hadoop Distributed File System) to achieve highly reliable and scalable data storage.

[0035] S120. Based on the target transaction dataset, allocate the data of the compressed frequent pattern tree corresponding to different transactions to a computing node.

[0036] In a specific implementation, based on the target transaction dataset, the data of the compressed frequent pattern tree corresponding to different transactions is allocated to a computing node, including: decomposing the compressed frequent pattern tree included in the target transaction dataset into a path list, wherein the path list includes a path, a data item included in the path, and a frequency value corresponding to the data item included in the path; and allocating the data item included in the path list of each compressed frequent pattern tree and the frequency value corresponding to the data item to the corresponding computing node.

[0037] After processing the initial transaction dataset to obtain the target transaction dataset in step S110, the target transaction dataset is stored in HDFS. Then, in step S120, the target transaction dataset is first retrieved from HDFS. The specific process for retrieving the target transaction dataset is as follows: The textFile or objectFile interface is called through SparkContext to read the data corresponding to each compressed frequent pattern tree stored in HDFS. (RDD), which serves as the basis for all subsequent distributed operations.

[0038] Among them, the Resilient Distributed Dataset (RDD) is the most basic data abstraction in Apache Spark, and it has the following five core characteristics: 1. Distributed storage: Data is distributed across the memory or disk of multiple nodes in the cluster, supporting large-scale parallel processing; 2. Immutability: Once created, it cannot be modified, ensuring consistency and fault tolerance; 3. Flexible fault tolerance; 4. Lazy evaluation; 5. It is partitionable, supporting parallel processing and scheduling optimization of each partition on different computing nodes.

[0039] In other words, in Spark's distributed computing architecture, a compute node is a worker machine in the cluster responsible for executing specific computing tasks. By distributing the compressed frequent pattern trees corresponding to different transactions in the target transaction dataset to different compute nodes, each compute node executes computing tasks in parallel, thereby improving data processing efficiency.

[0040] In the specific implementation, the data items and frequency values ​​corresponding to the path lists of each compression frequent pattern tree are allocated to the corresponding computing nodes, including: obtaining the status information of each computing node; and determining the computing nodes corresponding to each compression frequent pattern tree based on the relationship between the number of paths and the average path length in the path lists of each compression frequent pattern tree and the status information of the computing nodes. First, the number of paths and the average path length in the path list of each compression frequent pattern tree can reflect the amount of data corresponding to that compression frequent pattern tree. The status information of each computing node reflects the data processing capacity of each computing node. By understanding the relationship between the number of paths and the average path length in the path list of each compression frequent pattern tree and the status information of the computing nodes, the computing nodes corresponding to each compression frequent pattern tree can be determined.

[0041] In addition, during the above implementation process, the path decomposition of the compressed frequent pattern tree included in the target transaction dataset is performed to obtain the path list included in the compressed frequent pattern tree, in order to facilitate subsequent partitioning and distribution of computing nodes by data item.

[0042] Because the CSFP-tree algorithm was used to compress the frequent pattern tree to generate a compressed frequent pattern tree and remove data item nodes that are meaningless to the mining results, each data item node in the target transaction dataset is a node that is meaningful to the mining results, ensuring the accuracy of the subsequent determined association relationships.

[0043] S130. Based on the frequent pattern tree mining algorithm, perform association mining on the data items included in the compressed frequent pattern tree corresponding to each computing node to determine the association relationship corresponding to each computing node.

[0044] In a specific implementation, the association relationship of the target data items corresponding to each computing node is mined based on the frequent pattern tree mining algorithm to determine the association relationship of the target data items corresponding to each computing node. This includes: determining the conditional pattern base and conditional pattern tree of the target data items included in each computing node based on the frequent pattern tree mining algorithm; and determining the association relationship of the target data items corresponding to each computing node based on the conditional pattern base and conditional pattern tree of the target data items included in each computing node.

[0045] Specifically, the mapPartition operation is used to mine the compressed frequent pattern tree in each computing node using a frequent pattern tree mining algorithm to extract the conditional pattern base and conditional pattern tree for the corresponding data item. The conditional pattern base of a data item is defined as follows: for a given data item X, its conditional pattern base is the set of all data items preceding data item X in all transactions (i.e., itemsets) containing data item X in the initial transaction dataset, along with their occurrence paths. Specifically, it is represented as a set of prefix paths, each path accompanied by a frequency count. The conditional pattern tree is defined as: a locally frequent pattern tree reconstructed based on the conditional pattern base of a given data item X, used to mine all data item sets of data item X. This tree is smaller and more focused than the original CSFP-tree, specifically designed for mining data item sets ending with data item X.

[0046] Furthermore, based on the conditional schema base and conditional schema tree of the data items included in each computing node, the association relationship of the data items corresponding to each computing node is determined, including: recursively executing the FP-Growth algorithm on the conditional schema tree to generate a data item set containing any target data item; determining the confidence level of each rule in the data item set based on the data item set containing the target data item; and determining the association relationship of the data items corresponding to each computing node based on the confidence level of each rule in the data item set and the conditional schema base of the data items included in each computing node.

[0047] Specifically, the FP-Growth algorithm is recursively executed on the conditional pattern tree to generate all data item sets containing data item X. For example, the data item sets are represented as {A, B, C, X} and {D, E, X}. Then, for each data item set, it is divided into rules of the form L->R (where R contains X), and the confidence is calculated, that is, the confidence of rule {A, B, C} => {X} and the confidence of rule {D, E} => {X} are determined. Then, based on the conditional pattern base of the data items included in each computing node, the frequency values ​​of rule {A, B, C} => {X} and rule {D, E} => {X} are determined.

[0048] S140. Based on the association relationships corresponding to each computing node, merge the association relationships corresponding to each computing node to obtain the target association relationship of the original transaction dataset.

[0049] In the specific implementation, the association relationships of each computing node are merged according to the association relationships of each computing node to obtain the target association relationship of the original transaction dataset. This includes: performing deduplication on the same association relationships of the same data item included in different computing nodes according to the association relationships of each computing node; and normalizing the frequency values ​​of the same data item included in different computing nodes according to the frequency values ​​of the data items included in different computing nodes.

[0050] The final generated association satisfies the principles of completeness, deduplication, and consistency. The completeness principle means that all local rules generated by all computing nodes need to be fully collected. The deduplication principle means that if the same rule is generated on multiple nodes, it needs to be deduplicated. The consistency principle means that the support corresponding to the same data item set should take the original statistical value or the global aggregate value to ensure that the values ​​are consistent.

[0051] The data relationship discovery and mining method based on the Spark framework provided in this disclosure first processes the initial transaction dataset using a compressed frequent pattern tree algorithm to obtain a target transaction dataset. Then, based on the target transaction dataset, the data from the compressed frequent pattern trees corresponding to different transactions are allocated to a computing node. Next, based on a frequent pattern tree mining algorithm, the data items included in the compressed frequent pattern trees corresponding to each computing node are analyzed to determine the relationships between each computing node. Finally, based on the relationships between each computing node, the relationships between each computing node are merged to obtain the target relationships of the original transaction dataset. This data relationship discovery and mining method based on the Spark framework can achieve comprehensive extraction and efficient relationship analysis and mining of feature elements from massive standardized target network data, based on the unified processing and standardized description of multi-source heterogeneous target network data such as Internet data resources, cryptographic resources, publicly available software tools, proprietary software tools, and technical knowledge bases.

[0052] Based on the above embodiments, as a preferred implementation method, a combination of BERT and LSTM is used. First, the text is input into the BERT module, and BERT learns the encoding vector containing the text context information. At the same time, the labels are input into the label encoding layer, and the long short-term memory capability of LSTM is used to obtain the intrinsic relationship between the labels. Then, the BERT output vector and the LSTM output vector are subjected to an attention mechanism operation to obtain the specific relationship between each label and the text. The attention mechanism can explicitly show the prominent relationship between each document and each label. Finally, the sigmoid function is used to predict the independent distribution of each label to obtain the multi-label prediction sequence.

[0053] BERT predicts missing words in text through an automatic random masking mechanism, and simultaneously utilizes the NextSentence Prediction (NSP) task to jointly represent the sentence sequence of the text. BERT first inputs sentences into the model, separated by "[CLS]" and "[SEP]", with each word having an embedding dimension of 768. Then, a 12-layer Transformers Encoder structure transforms each word into a Tn rich in syntactic and semantic features, extracting the most significant features. The feature vector represents the global contextual information of the sentence. The “[CLS]” identifier does not represent a specific character or word in the text. It performs a self-attention operation with each word in the sentence, thus learning the contextual semantic information of each word in the sentence. Using it as a classification criterion can reflect a certain degree of fairness and rationality.

[0054] The label encoding layer first embeds and encodes the label text information to obtain the vector representation of the label; then, it uses the memory mechanism of the LSTM structure to obtain the correlation between labels; finally, it integrates the output layer of BERT for joint attention mechanism operation. The role of label encoding is to extract all label features through LSTM, expand the vector dimension of the labels, and provide computational convenience and interpretability for the subsequent attention mechanism; at the same time, by utilizing the long short-term memory structure features of LSTM, the inherent continuous features between different labels can be learned, such as the inclusion relationship or hierarchical relationship between labels.

[0055] Next, a separate attention fusion operation is performed on the CLS output layer of the BERT model and the LSTM hidden layer of the labels to obtain the CLS vector and the weights of each label. The weights are then assigned to the CLS vector. The attention mechanism can effectively highlight the explicit expression between text features and labels. Finally, the prediction result for each label is obtained by mapping to the label dimension through the Sigmoid function.

[0056] Steps S110-S140 constitute the structured relationship mining layer, focusing on extracting explicit co-occurrence relationships and association rules from large-scale transactional data; this step constitutes the semantic feature understanding layer, focusing on deep semantic modeling of unstructured / semi-structured text data to achieve complete description of implicit semantic relationships and multi-label classification prediction; by combining the above steps S110-S140 with this step, both structured relationship mining and implicit semantic relationship mining can be combined.

[0057] Based on the above embodiments, Figure 2 This is a schematic diagram of the structure of a data relationship discovery and mining device based on the Spark framework provided in an embodiment of this disclosure, as shown below. Figure 2 As shown, the data relationship discovery and mining device based on the Spark framework includes: The target data determination module 210 is used to process the initial transaction dataset based on the compressed frequent pattern tree algorithm to obtain the target transaction dataset. The initial transaction dataset includes network data from different fields and multi-source heterogeneous network data. The target transaction dataset is composed of compressed frequent pattern tree data corresponding to different transactions. Data allocation module 220 is used to allocate the data of the compressed frequent pattern tree corresponding to different transactions to a computing node according to the target transaction dataset; The association determination module 230 is used to perform association mining on the data items included in the compressed frequent pattern tree corresponding to each computing node based on the frequent pattern tree mining algorithm, and determine the association relationship corresponding to each computing node. The target association determination module 240 is used to merge the associations corresponding to each computing node according to the associations corresponding to each computing node to obtain the target associations of the original transaction dataset.

[0058] The data relationship discovery and mining apparatus based on the Spark framework provided in this disclosure first processes the initial transaction dataset using a compressed frequent pattern tree algorithm to obtain a target transaction dataset. Then, based on the target transaction dataset, the data from the compressed frequent pattern trees corresponding to different transactions are allocated to a computing node. Next, based on a frequent pattern tree mining algorithm, the data items included in the compressed frequent pattern trees corresponding to each computing node are used to mine association relationships, determining the association relationships corresponding to each computing node. Finally, based on the association relationships corresponding to each computing node, the association relationships corresponding to each computing node are merged to obtain the target association relationships of the original transaction dataset. The data relationship discovery and mining method based on the Spark framework can achieve comprehensive extraction and efficient association analysis and mining of feature elements from massive standardized target network data, based on the unified processing and standardized description of multi-source heterogeneous target network data such as Internet data resources, cryptographic resources, publicly available software tool resources, proprietary software tool resources, and technical knowledge bases.

[0059] In a specific implementation, the process of processing the initial transaction dataset based on the compressed frequent pattern tree algorithm to obtain the target transaction dataset includes: Obtain the transactions, the data items included in each transaction, and the frequency values ​​of the data items included in each transaction from the initial transaction dataset; Based on the frequency values ​​of the data items included in the transaction, filter out the data items whose frequency values ​​meet the preset threshold; After sorting the selected data items in descending order by frequency value, a compressed frequent pattern tree corresponding to different transactions is constructed. The data from the compressed frequent pattern trees corresponding to different transactions constitute the target transaction dataset.

[0060] In a specific implementation, the step of allocating the data of the compressed frequent pattern tree corresponding to different transactions to a computing node according to the target transaction dataset includes: The compressed frequent pattern tree included in the target transaction dataset is decomposed into a path list to obtain the path list included in the compressed frequent pattern tree, wherein the path list includes a path, a data item included in the path, and a frequency value corresponding to the data item included in the path. The path list of each compressed frequent pattern tree includes the data items and the corresponding frequency values ​​of the data items, which are then assigned to the corresponding computing nodes.

[0061] In a specific implementation, the step of allocating the data items and their corresponding frequency values ​​from the path list of each compressed frequent pattern tree to the corresponding computing nodes includes: Obtain the status information of each computing node; Based on the path count information and the relationship between the average path length and the computing node status information included in the path list of each compression frequent pattern tree, the computing nodes corresponding to each compression frequent pattern tree are determined. The path list of each compressed frequent pattern tree includes the data items and the corresponding frequency values ​​of the data items, which are then assigned to the corresponding computing nodes.

[0062] In a specific implementation, the step of mining the association relationships of the data items included in the compressed frequent pattern tree corresponding to each computing node based on the frequent pattern tree mining algorithm to determine the association relationships corresponding to each computing node includes: Based on the frequent pattern tree mining algorithm, the conditional pattern base and conditional pattern tree of the data items included in each computing node are determined; Based on the conditional schema base and conditional schema tree of the data items included in each computing node, the association relationship of the data items corresponding to each computing node is determined.

[0063] In a specific implementation, determining the association relationship of data items corresponding to each computing node based on the conditional schema base and conditional schema tree of the data items included in each computing node includes: The FP-Growth algorithm is recursively executed on the conditional pattern tree to generate a set of data items containing any target data item. Based on the data item set containing the target data item, determine the confidence level of each rule in the data item set; Based on the confidence level of each rule in the data item set and the conditional pattern base of the data items included in each computing node, the association relationship of the data items corresponding to each computing node is determined.

[0064] In a specific implementation, the step of merging the association relationships corresponding to each computing node to obtain the target association relationship of the original transaction dataset, based on the association relationships corresponding to each computing node, includes: Based on the association relationships corresponding to each computing node, the same association relationships of the same data item included in different computing nodes are deduplicated. Based on the frequency value of the data item corresponding to each computing node, the frequency values ​​of the same data item included in different computing nodes are normalized.

[0065] This application also provides a computer device, please refer to the following for details. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.

[0066] The computer device includes a memory 510 and a processor 520 that are interconnected via a system bus. It should be noted that only a computer device with components 510-520 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components may be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0067] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0068] The memory 510 includes at least one type of readable storage medium, including non-volatile memory or volatile memory, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. RAM may include static RAM or dynamic RAM. In some embodiments, the memory 510 may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory 510 may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the computer device. Of course, the memory 510 may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory 510 is typically used to store the operating system and various application software installed on the computer device, such as the program code of the method described above. In addition, the memory 510 may also be used to temporarily store various types of data that have been output or will be output.

[0069] The processor 520 is typically used to perform the overall operation of a computer device. In this embodiment, the memory 510 is used to store program code or instructions, including computer operation instructions. The processor 520 is used to execute the program code or instructions stored in the memory 510 or to process data, such as program code that runs the methods described above.

[0070] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus system can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0071] Another embodiment of this application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads computer-readable program code stored in the computer-readable medium, enabling the processor to execute the functional actions specified in each step or combination of steps in the above method; and to generate means for implementing the functional actions specified in each block or combination of blocks in the block diagram.

[0072] Computer-readable media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any suitable combination thereof, wherein the memory is used to store program code or instructions, the program code including computer operation instructions, and the processor is used to execute the program code or instructions of the above-described methods stored in the memory.

[0073] The definitions of memory and processor can be found in the description of the foregoing computer device embodiments, and will not be repeated here.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0075] In the various embodiments of this application, the functional units or modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] Unless otherwise expressly indicated by the context, the singular form of words used herein and in the appended claims includes the plural form, and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the terms “comprising” and “including” shall be interpreted as including rather than exclusively. Likewise, the terms “including” and “or” shall be interpreted as including unless such interpretation is expressly prohibited herein. Where the term “example” is used herein, particularly when it follows a set of terms, the “example” is merely exemplary and illustrative and should not be considered exclusive or extensive.

[0078] Further aspects and scope of adaptation become apparent from the description provided herein. It should be understood that various aspects of this application may be implemented individually or in combination with one or more other aspects. It should also be understood that the descriptions and specific embodiments herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0079] Several embodiments of this disclosure have been described in detail above. However, it is obvious that those skilled in the art can make various modifications and variations to the embodiments of this disclosure without departing from the spirit and scope of this disclosure. The scope of protection of this disclosure is defined by the appended claims.

Claims

1. A data relationship discovery and mining method based on the Spark framework, characterized in that, include: The initial transaction dataset is processed based on the compressed frequent pattern tree algorithm to obtain the target transaction dataset. The initial transaction dataset includes network data from different domains and multi-source heterogeneous network data. The target transaction dataset consists of compressed frequent pattern tree data corresponding to different transactions. Based on the target transaction dataset, the data of the compressed frequent pattern tree corresponding to different transactions are allocated to a computing node. The frequent pattern tree mining algorithm is used to mine the association relationships of the data items included in the compressed frequent pattern tree corresponding to each computing node, and to determine the association relationships corresponding to each computing node. Based on the association relationships corresponding to each computing node, the association relationships corresponding to each computing node are merged to obtain the target association relationships of the original transaction dataset.

2. The method according to claim 1, characterized in that, The initial transaction dataset is processed using the compressed frequent pattern tree algorithm to obtain the target transaction dataset, including: Obtain the transactions, the data items included in each transaction, and the frequency values ​​of the data items included in each transaction from the initial transaction dataset; Based on the frequency values ​​of the data items included in the transaction, filter out the data items whose frequency values ​​meet the preset threshold; After sorting the selected data items in descending order by frequency value, a compressed frequent pattern tree corresponding to different transactions is constructed. The data from the compressed frequent pattern trees corresponding to different transactions constitute the target transaction dataset.

3. The method according to claim 1, characterized in that, The step of allocating the data of the compressed frequent pattern tree corresponding to different transactions to a computing node according to the target transaction dataset includes: The compressed frequent pattern tree included in the target transaction dataset is decomposed into a path list to obtain the path list included in the compressed frequent pattern tree, wherein the path list includes a path, a data item included in the path, and a frequency value corresponding to the data item included in the path. The path list of each compressed frequent pattern tree includes the data items and the corresponding frequency values ​​of the data items, which are then assigned to the corresponding computing nodes.

4. The method according to claim 3, characterized in that, The process of assigning the data items and their corresponding frequency values ​​from the path list of each compressed frequent pattern tree to the corresponding computing nodes includes: Obtain the status information of each computing node; Based on the path count information and the relationship between the average path length and the computing node status information included in the path list of each compression frequent pattern tree, the computing nodes corresponding to each compression frequent pattern tree are determined. The path list of each compressed frequent pattern tree includes the data items and the corresponding frequency values ​​of the data items, which are then assigned to the corresponding computing nodes.

5. The method according to claim 1, characterized in that, The frequent pattern tree mining algorithm is used to mine the relationships between the data items in the compressed frequent pattern tree corresponding to each computing node, and to determine the relationships between each computing node, including: Based on the frequent pattern tree mining algorithm, the conditional pattern base and conditional pattern tree of the data items included in each computing node are determined; Based on the conditional schema base and conditional schema tree of the data items included in each computing node, the association relationship of the data items corresponding to each computing node is determined.

6. The method according to claim 5, characterized in that, The step of determining the association relationship of data items corresponding to each computing node based on the conditional schema base and conditional schema tree of the data items included in each computing node includes: The FP-Growth algorithm is recursively executed on the conditional pattern tree to generate a set of data items containing any target data item. Based on the data item set containing the target data item, determine the confidence level of each rule in the data item set; Based on the confidence level of each rule in the data item set and the conditional pattern base of the data items included in each computing node, the association relationship of the data items corresponding to each computing node is determined.

7. The method according to claim 1, characterized in that, The step of merging the association relationships corresponding to each computing node based on the association relationships corresponding to each computing node to obtain the target association relationship of the original transaction dataset includes: Based on the association relationships corresponding to each computing node, the same association relationships of the same data item included in different computing nodes are deduplicated. Based on the frequency value of the data item corresponding to each computing node, the frequency values ​​of the same data item included in different computing nodes are normalized.

8. A data relationship discovery and mining device based on the Spark framework, characterized in that, include: The target data determination module is used to process the initial transaction dataset based on the compressed frequent pattern tree algorithm to obtain the target transaction dataset. The initial transaction dataset includes network data from different domains and multi-source heterogeneous network data. The target transaction dataset is composed of compressed frequent pattern tree data corresponding to different transactions. The data allocation module is used to allocate the data of the compressed frequent pattern tree corresponding to different transactions to a computing node according to the target transaction dataset; The association determination module is used to mine the associations of the data items included in the compressed frequent pattern tree corresponding to each computing node based on the frequent pattern tree mining algorithm, and to determine the associations of each computing node. The target association determination module is used to merge the associations corresponding to each computing node based on the associations corresponding to each computing node, so as to obtain the target associations of the original transaction dataset.

9. A computer device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images