Big data relation mining method based on graph calculation

Abstract

The invention provides a big data relationship mining method based on graph calculation, which comprises the following steps: preprocessing multi-source heterogeneous data to generate a graph structure, mining frequent sub-graphs and dynamic relationships by utilizing a graph calculation technology, and constructing a relationship model, and relates to data cleaning, entity recognition, relationship extraction and graph calculation technologies. Wherein data cleaning is combined with locality sensitive hashing and machine learning denoising, entity recognition and relation extraction adopt a deep learning model, graph calculation fuses a graph convolutional network and distributed graph traversal, finally, relation prediction is carried out through a graph-based machine learning framework, and an analysis result is displayed by utilizing a graph visualization technology. According to the method, the accuracy and efficiency of big data relation mining are effectively improved, and the method is suitable for data analysis and decision support in a complex scene.

Description

Technical Field

[0001] The present invention relates to the field of data processing and mining technology, and more specifically, to a big data relationship mining method based on graph computing. Background Art

[0002] In the era of big data, the types and quantities of data are growing exponentially. Traditional data processing and mining methods have been unable to cope with these complex data relationships and massive data. Graph computing, as an effective tool for processing complex data relationships and structures, has gradually attracted widespread attention from researchers and the industry. Graph computing is a computing model based on graph theory that is used to process graph structured data composed of nodes and edges. It is suitable for various application scenarios, including social network analysis, recommendation systems, financial risk assessment, etc. Graph computing can effectively capture relationships and dependencies in data and reveal potential patterns and trends.

[0003] The prior art discloses a visualization character relationship mining management system and method based on big data. The visualization character relationship mining management system based on big data belongs to the field of big data technology and includes a mining module, an analysis module and a management module. The mining module includes a character mining unit, a time mining unit and a frequency mining unit. The character mining unit is used to count and calculate different accounts in the acquired mining data set to obtain a character data set. The time mining unit is used to count and mark the conversation time points in the character data set to obtain a time point set. The duration of the conversation is obtained according to the conversation time point and counted and marked to obtain a duration set. The time point set and the duration set constitute the time data set. The present invention also discloses a visualization character relationship mining management method based on big data. The present invention can solve the technical problems of poor depth and breadth of character relationship mining in the existing scheme and poor management and update effect after character relationship mining.

[0004] However, this data mining method may not be able to fully utilize the advantages of deep learning models when extracting relationships. When processing large-scale graph data, it faces computing performance bottlenecks and cannot improve the efficiency of subgraph mining and pattern matching. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, the present invention designs a big data relationship mining method based on graph computing, which can effectively overcome the defects of the prior art.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] A big data relationship mining method based on graph computing includes the following steps:

[0008] S1. Preprocessing multi-source heterogeneous big data, including data cleaning, entity recognition, relationship extraction and type labeling, to generate graph structure data, wherein the preprocessing step includes using natural language processing technology to perform semantic role labeling on text data to identify semantic relationships between entities;

[0009] S2. Mining frequent subgraphs and dynamic relationships from a graph using graph computing technology, wherein the graph computing technology includes a pattern matching algorithm that combines feature extraction of deep learning with traditional graph traversal technology to improve the accuracy and efficiency of subgraph mining;

[0010] S3. Build a relationship model based on the mining results and apply it to specific scenarios for data analysis and insights, where the relationship model includes a graph-based machine learning framework for training and predicting the development and changes of complex relationships between entities.

[0011] The data cleaning step includes using data deduplication and noise filtering technology, and the data deduplication uses a local sensitive hashing method, which includes the following steps:

[0012] Convert each data record into a feature vector;

[0013] Choose a hash function that suits the characteristics of your data;

[0014] Using the selected hash function, mapping each of the feature vectors to one or more hash buckets, each of the hash buckets containing a plurality of the feature vectors, and the feature vectors of each hash bucket are relatively close in the hash space;

[0015] For each new data record, calculate the hash value of the feature vector and map it to the corresponding hash bucket;

[0016] Performing duplicate detection in the hash bucket, comparing the similarity of all the feature vectors in the same hash bucket, identifying records with a similarity higher than a set threshold as duplicate data, removing the identified duplicate data from the data set, and retaining unique data records;

[0017] The noise filtering technology includes using a statistical method and a machine learning method, and specifically includes the following steps:

[0018] The statistical methods include: standard deviation analysis, calculating the standard deviation of each data point and identifying those data points that deviate greatly from the mean;

[0019] Distribution anomaly detection, using box-plot to identify outliers in the data;

[0020] Apply the Z-Score method, where data points with a Z-Score greater than 3 or less than -3 are considered outliers;

[0021] The machine learning-based method uses local anomaly factors, specifically:

[0022] Distance calculation, use Euclidean distance to calculate neighborhood distance, for each data point, calculate its distance to all other data points;

[0023] Construct local density, select k value, determine the size of local neighborhood (k value), that is, the k nearest neighbors of each data point;

[0024] Calculate k-distance. For each data point, find the maximum distance (k-distance) among its k nearest neighbors.

[0025] Calculate the reachable distance. For data points i and j, calculate their reachable distance, which is defined as the larger value between k distances i and j;

[0026] Local density, calculate the local density of each data point, that is, the density of the data points in its k neighborhoods;

[0027] Relative density, calculate the relative density of each data point, that is, the ratio of its local density to the local density of the data points in its neighborhood;

[0028] LOF value, calculate the LOF value of each data point, that is, the average value of its relative density, set the threshold, and set the threshold according to the LOF value. Data points with LOF values ​​exceeding the threshold are considered outliers;

[0029] Filter outliers to remove or mark these abnormal points from the dataset.

[0030] The entity recognition step includes using a named entity recognition (NER) model based on deep learning, wherein the recognition model is a model combining a long short-term memory network (Bi-LSTM) and a conditional random field (CRF), and the specific method steps are as follows:

[0031] Get annotated text datasets, which contain annotated entity information;

[0032] Building a vocabulary and a tag set, wherein the vocabulary maps words in the text to unique index values;

[0033] The tag set defines entity tag categories;

[0034] Convert words to vectors using pre-trained word embeddings (e.g. Word2Vec, GloVe) or custom word embedding layers;

[0035] Bi-LSTM Network:

[0036] The input layer uses word embedding as the input of Bi-LSTM. The Bi-LSTM layer uses bidirectional LSTM to capture contextual information.

[0037] CRF layer:

[0038] Sequence labeling: Use the CRF layer to label the output of Bi-LSTM and optimize entity boundary recognition;

[0039] Model training, using the log-likelihood of CRF as the loss function, using the Adam optimization algorithm to train the model, dividing the data into training set and validation set, performing model training, and adjusting hyperparameters to optimize performance.

[0040] The relation extraction step includes using a deep learning-based relation extraction model, which uses a pre-trained language model (such as BERT) to perform relation prediction, and includes the following steps:

[0041] Obtain a large-scale corpus containing entity pairs and relation annotations;

[0042] Organize the text into a format suitable for model input and divide the dataset into training, validation and test sets;

[0043] Text tokenization, using BERT’s tokenizer to split the text into vocabulary indices;

[0044] Generate input and create input samples for each entity pair, including:

[0045] Token IDs, vocabulary index of text;

[0046] Attention Masks, indicating which locations should be paid attention to by the model;

[0047] Token Type IDs, used to distinguish between two different parts of text;

[0048] Model building, loading pre-trained BERT, and selecting the appropriate BERT variant;

[0049] BERT layer, extracting contextual information from text;

[0050] Relation classification head, which adds a fully connected layer on the CLS token output by BERT for relation classification;

[0051] Define the loss function and use the cross entropy loss function to optimize the classification task;

[0052] Perform model training and input new text into the trained model to perform relationship prediction and map the relationship categories output by the model back to the actual relationship labels.

[0053] The pattern matching algorithm in the graph computing technology includes using a graph convolutional network (GCN) to perform feature learning and pattern recognition on graph structure data, wherein the GCN is used to learn local and global features of nodes, and the specific steps are:

[0054] Graph data collection, obtaining graph structure data with nodes and edges;

[0055] Data formatting: converting data into a format suitable for GCN input, including node feature matrix and adjacency matrix, and standardizing node features to improve training results;

[0056] Adjacency matrix processing: normalize the adjacency matrix and calculate the inverse square root of the degree matrix of each node's adjacency matrix;

[0057] Convolution operation, applies convolution operation to each node and its neighborhood to aggregate the information of neighboring nodes;

[0058] Activation function, use ReLU activation function to increase the nonlinear expression ability of the model;

[0059] Stack GCN layers and stack multiple graph convolutional layers as needed to capture multi-level features of nodes;

[0060] Output layer: Add appropriate output layer for subgraph mining tasks, such as fully connected layer or pooling layer, to train the constructed model;

[0061] Feature extraction, extracting node and subgraph features from GCN output;

[0062] Subgraph matching, by comparing the extracted features with the target pattern, identifying and extracting subgraphs that match a specific pattern;

[0063] Deploy the trained GCN model into the production environment, integrate it into practical applications, use the model for real-time subgraph matching and pattern recognition, and provide data analysis and decision support.

[0064] The graph traversal technology includes using a distributed graph computing framework (GraphX) to process large-scale graph data to accelerate the subgraph mining process;

[0065] Confirm the Spark cluster version and download the GraphX ​​package compatible with the Spark version;

[0066] Upload the GraphX ​​package to all nodes of the Spark cluster and place it in the specified library directory;

[0067] Adjust the spark.executor.memory and spark.driver.memory parameters to allocate appropriate memory;

[0068] Set the spark.cores.max and spark.executor.instances parameters to configure the number of cores and executor instances;

[0069] Data loading, using Spark's textFile or sequenceFile method to load data from HDFS, S3, or other storage systems;

[0070] Clean the data, remove invalid or erroneous records, and convert the raw data into vertex and edge formats;

[0071] Create vertex RDD and edge RDD, ensuring that the data types are consistent with GraphX ​​requirements;

[0072] Create a vertex RDD and use Spark's parallelize method to create a vertex RDD, assigning a unique ID and corresponding attributes to each vertex;

[0073] Create edge RDD, use Spark's parallelize method to create edge RDD, define the source vertex, target vertex and attributes of the edge;

[0074] Build a Graph object, pass the vertex RDD and edge RDD to the Graph constructor, create a Graph object, and set the default vertex attributes for the Graph;

[0075] Define graph calculation logic:

[0076] Select appropriate graph algorithms based on business needs and implement algorithm logic;

[0077] Use GraphXAPI to call GraphX's pregel and mapVertices APIs for graph calculations;

[0078] Write user-defined functions to manipulate vertex and edge properties;

[0079] Distributed Graph Computing

[0080] Graph partitioning, select the appropriate partitioning strategy;

[0081] Consider the characteristics of the graph to choose the best partitioning strategy;

[0082] Parallel computing uses the distributed computing capabilities of GraphX ​​to split graph data into multiple partitions and process them in parallel on the cluster, monitor the progress of tasks and resource usage, and ensure that computing tasks run efficiently.

[0083] The graph-based machine learning framework in the relational model includes using a graph neural network (GNN) to model and predict complex relationships between entities, wherein the GNN is used to capture dependencies and dynamic changes between entities.

[0084] The data analysis and insight step includes using graph-based visualization technology to present the mined relationships and patterns as interactive graphs to facilitate users to conduct in-depth analysis and understanding.

[0085] A computer storage medium stores computer instructions, which, when called, are used to execute the above-mentioned big data relationship mining method based on graph computing.

[0086] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: a big data relationship mining method based on graph computing uses a deep learning model (such as Bi-LSTM combined with CRF for entity recognition, BERT for relationship extraction) to improve the model's understanding of complex language and relationships, thereby improving the accuracy of relationship extraction. This approach makes up for the shortcomings of the prior art methods in semantic understanding, so that the model can better capture the deep-level relationships in the text, and adopts graph convolutional networks (GCN) and distributed graph computing frameworks (such as GraphX) to process large-scale graph data, which solves the performance bottleneck of traditional methods in processing big data, and uses distributed computing to improve the efficiency of subgraph mining and pattern matching. Through data deduplication and noise filtering technology (including local sensitive hashing, statistical methods and machine learning methods), the quality of data is effectively improved, providing more reliable input for subsequent mining, and the graph-based visualization technology presents the mined relationships and patterns in interactive graphs, helping users to more deeply understand the complex relationships and patterns in the data, and improving the intuitiveness and practicality of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived based on the provided drawings without paying any creative work.

[0088] Figure 1 This is a step diagram of a big data relationship mining method based on graph computing. DETAILED DESCRIPTION

[0089] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0090] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0091] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0092] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0093] Example

[0094] A big data relationship mining method based on graph computing, such as Figure 1 As shown, the following steps are included:

[0095] S1. Preprocessing multi-source heterogeneous big data, including data cleaning, entity recognition, relationship extraction and type labeling, to generate graph structure data, wherein the preprocessing step includes using natural language processing technology to perform semantic role labeling on text data to identify semantic relationships between entities;

[0096] S2. Mining frequent subgraphs and dynamic relationships from a graph using graph computing technology, wherein the graph computing technology includes a pattern matching algorithm that combines feature extraction of deep learning with traditional graph traversal technology to improve the accuracy and efficiency of subgraph mining;

[0097] S3. Build a relationship model based on the mining results and apply it to specific scenarios for data analysis and insights, where the relationship model includes a graph-based machine learning framework for training and predicting the development and changes of complex relationships between entities.

[0098] The data cleaning step includes using data deduplication and noise filtering technology, and the data deduplication uses a local sensitive hashing method, which includes the following steps:

[0099] Convert each data record into a feature vector;

[0100] Choose a hash function that is appropriate for the characteristics of your data; such as MinHash for Euclidean distance or LSH for cosine similarity.

[0101] Using the selected hash function, mapping each of the feature vectors to one or more hash buckets, each of the hash buckets containing a plurality of the feature vectors, and the feature vectors of each hash bucket are relatively close in the hash space;

[0102] For each new data record, calculate the hash value of the feature vector and map it to the corresponding hash bucket;

[0103] Performing duplicate detection in the hash bucket, comparing the similarity of all the feature vectors in the same hash bucket, identifying records with a similarity higher than a set threshold as duplicate data, removing the identified duplicate data from the data set, and retaining unique data records;

[0104] The noise filtering technology includes using a statistical method and a machine learning method, and specifically includes the following steps:

[0105] The statistical methods include: standard deviation analysis, calculating the standard deviation of each data point and identifying those data points that deviate greatly from the mean;

[0106] Distribution anomaly detection, using box-plot to identify outliers in the data;

[0107] Box plot, also known as box-line plot or box-whisker plot, is a statistical chart used to show the distribution of a set of data. It can provide summary statistical information of the minimum value, first quartile (Q1), median (Q2), third quartile (Q3) and maximum value of the data, and the shape of the box and the length of the whiskers can intuitively show the central tendency, dispersion and outliers of the data.

[0108] Apply the Z-Score method, where data points with a Z-Score greater than 3 or less than -3 are considered outliers;

[0109] The machine learning-based method uses local anomaly factors, specifically:

[0110] Distance calculation, use Euclidean distance to calculate neighborhood distance, for each data point, calculate its distance to all other data points;

[0111] Construct local density, select k value, determine the size of local neighborhood (k value), that is, the k nearest neighbors of each data point;

[0112] Calculate k-distance. For each data point, find the maximum distance (k-distance) among its k nearest neighbors.

[0113] Calculate the reachable distance. For data points i and j, calculate their reachable distance, which is defined as the larger value between k distances i and j;

[0114] Local density, calculate the local density of each data point, that is, the density of the data points in its k neighborhoods;

[0115] Relative density, calculate the relative density of each data point, that is, the ratio of its local density to the local density of the data points in its neighborhood;

[0116] LOF value: calculate the LOF value of each data point, that is, the average value of its relative density. The larger the LOF value, the more likely the data point is an outlier. Set the threshold: set the threshold according to the LOF value. Data points whose LOF value exceeds the threshold are considered outliers.

[0117] Filter outliers to remove or mark these abnormal points from the dataset.

[0118] The entity recognition step includes using a named entity recognition (NER) model based on deep learning, wherein the recognition model is a model combining a long short-term memory network (Bi-LSTM) and a conditional random field (CRF), and the specific method steps are as follows:

[0119] Get annotated text datasets, which contain annotated entity information, such as names of people, places, and organizations.

[0120] Building a vocabulary and a tag set, wherein the vocabulary maps words in the text to unique index values;

[0121] The tag set defines entity tag categories; for example, "B-PER" represents the beginning of a person's name, and "I-PER" represents the inside of a person's name.

[0122] Convert words to vectors using pre-trained word embeddings (e.g. Word2Vec, GloVe) or custom word embedding layers;

[0123] Bi-LSTM Network:

[0124] The input layer uses word embedding as the input of Bi-LSTM. The Bi-LSTM layer uses bidirectional LSTM to capture contextual information.

[0125] CRF layer:

[0126] Sequence labeling: Use the CRF layer to label the output of Bi-LSTM and optimize entity boundary recognition;

[0127] Model training, using the log-likelihood of CRF as the loss function, using the Adam optimization algorithm to train the model, dividing the data into training set and validation set, performing model training, and adjusting hyperparameters to optimize performance.

[0128] The relation extraction step includes using a deep learning-based relation extraction model, which uses a pre-trained language model (such as BERT) to perform relation prediction, and includes the following steps:

[0129] Obtain a large-scale corpus containing entity pairs and relation annotations; such as ACE and SemEval datasets.

[0130] Organize the text into a format suitable for model input and divide the dataset into training, validation and test sets;

[0131] Text tokenization, using BERT’s tokenizer to split the text into vocabulary indices;

[0132] Generate input and create input samples for each entity pair, including:

[0133] Token IDs, vocabulary index of text;

[0134] Attention Masks, indicating which locations should be paid attention to by the model;

[0135] Token Type IDs, used to distinguish between two different parts of text; such as a question and an answer.

[0136] Model building, loading pre-trained BERT, and selecting an appropriate BERT variant, such as BERT-base or BERT-large.

[0137] BERT layer, extracting contextual information from text;

[0138] Relation classification head, which adds a fully connected layer on the CLS token output by BERT for relation classification;

[0139] Define the loss function and use the cross entropy loss function to optimize the classification task;

[0140] Perform model training and input new text into the trained model to perform relationship prediction and map the relationship categories output by the model back to the actual relationship labels.

[0141] The pattern matching algorithm in the graph computing technology includes using a graph convolutional network (GCN) to perform feature learning and pattern recognition on graph structure data, wherein the GCN is used to learn local and global features of nodes, and the specific steps are:

[0142] Graph data collection, obtaining graph structure data with nodes and edges; (the data can come from social networks, molecular structures or other graph-based datasets

[0143] Data formatting: converting data into a format suitable for GCN input, including node feature matrix and adjacency matrix, and standardizing node features to improve training results;

[0144] Adjacency matrix processing: normalize the adjacency matrix and calculate the inverse square root of the degree matrix of each node's adjacency matrix;

[0145] Convolution operation, applies convolution operation to each node and its neighborhood to aggregate the information of neighboring nodes;

[0146] Activation function, use ReLU activation function to increase the nonlinear expression ability of the model;

[0147] Stack GCN layers and stack multiple graph convolutional layers as needed to capture multi-level features of nodes;

[0148] Output layer: Add appropriate output layer for subgraph mining tasks, such as fully connected layer or pooling layer, to train the constructed model;

[0149] Feature extraction, extracting node and subgraph features from GCN output;

[0150] Subgraph matching, by comparing the extracted features with the target pattern, identifying and extracting subgraphs that match a specific pattern;

[0151] Deploy the trained GCN model into the production environment, integrate it into practical applications, use the model for real-time subgraph matching and pattern recognition, and provide data analysis and decision support.

[0152] The graph traversal technology includes using a distributed graph computing framework (GraphX) to process large-scale graph data to accelerate the subgraph mining process;

[0153] Confirm the Spark cluster version and download the GraphX ​​package compatible with the Spark version;

[0154] Upload the GraphX ​​package to all nodes of the Spark cluster and place it in the specified library directory;

[0155] Adjust the spark.executor.memory and spark.driver.memory parameters to allocate appropriate memory;

[0156] Set the spark.cores.max and spark.executor.instances parameters to configure the number of cores and executor instances;

[0157] Data loading, using Spark's textFile or sequenceFile method to load data from HDFS, S3, or other storage systems;

[0158] Clean the data, remove invalid or erroneous records, and convert the original data into vertex format ((VertexId, VD)) and edge format (Edge[ED]);

[0159] Create vertex RDD and edge RDD, ensuring that the data types are consistent with GraphX ​​requirements;

[0160] Create a vertex RDD and use Spark's parallelize method to create a vertex RDD, assigning a unique ID and corresponding attributes to each vertex;

[0161] Create edge RDD, use Spark's parallelize method to create edge RDD, define the source vertex, target vertex and attributes of the edge;

[0162] Build a Graph object, pass the vertex RDD and edge RDD to the Graph constructor, create a Graph object, and set the default vertex attributes for the Graph;

[0163] Define graph calculation logic:

[0164] Select appropriate graph algorithms based on business needs and implement algorithm logic; for example, define the iterative calculation process of PageRank;

[0165] Use GraphXAPI to call GraphX's pregel and mapVertices APIs for graph calculations;

[0166] Write user-defined functions to manipulate vertex and edge properties;

[0167] Distributed Graph Computing

[0168] Graph partitioning, select an appropriate partitioning strategy, such as CanonicalRandomVertexCut, to optimize computing performance;

[0169] Consider the characteristics of the graph to choose the best partitioning strategy;

[0170] For connected graphs, CanonicalRandomVertexCut usually performs well;

[0171] Sparse graph or dense graph, EdgeCut may be more suitable for dense graph, while VertexCut is suitable for sparse graph.

[0172] Parallel computing uses the distributed computing capabilities of GraphX ​​to split graph data into multiple partitions and process them in parallel on the cluster, monitor the progress of tasks and resource usage, and ensure that computing tasks run efficiently.

[0173] The graph-based machine learning framework in the relational model includes using a graph neural network (GNN) to model and predict complex relationships between entities, wherein the GNN is used to capture dependencies and dynamic changes between entities.

[0174] The data analysis and insight step includes using graph-based visualization technology to present the mined relationships and patterns as interactive graphs to facilitate users to conduct in-depth analysis and understanding.

[0175] A computer storage medium stores computer instructions, which, when called, are used to execute the above-mentioned big data relationship mining method based on graph computing.

[0176] In the specific implementation process, step S1: data cleaning and graph structure generation

[0177] Each data record is converted into a feature vector, and the locality sensitive hashing (LSH) method is used to deduplicate. A hash function suitable for the data characteristics is selected, and the feature vector is mapped to a hash bucket. For each new data record, its hash value is calculated and mapped to the hash bucket. Duplicate detection is performed in the hash bucket. By calculating the similarity, duplicate data is identified and removed, and unique records are retained.

[0178] Noise Filtering:

[0179] Statistical methods, using standard deviation analysis, box plots and Z-Score methods to identify and remove outliers, machine learning methods: calculating the local outlier factor (LOF) of each data point, setting a threshold, and removing outliers whose LOF values ​​exceed the threshold.

[0180] Entity Recognition and Relation Extraction:

[0181] Entity recognition, using a deep learning model that combines long short-term memory network (Bi-LSTM) and conditional random field (CRF) for named entity recognition, building a vocabulary and label set, using pre-trained or custom word embedding to convert words into vectors, the Bi-LSTM layer captures contextual information, the CRF layer performs sequence labeling, and optimizes entity boundary recognition.

[0182] Relation extraction: Use a BERT-based deep learning model to predict relationships, organize the text into a format suitable for model input, use BERT to extract contextual information, and add a fully connected layer for relationship classification.

[0183] Generate graph structured data, convert the preprocessed data (including entities and relationships) into graph structured data, including nodes (entities) and edges (relationships).

[0184] Step S2: Graph computation and frequent subgraph mining

[0185] Graph computing technology, graph convolutional network (GCN): collect and format graph data, create node feature matrix and adjacency matrix, normalize the adjacency matrix, apply convolution operation and ReLU activation function in GCN, stack multiple GCN layers to capture multi-level features of nodes, extract node and subgraph features from GCN output, and perform pattern matching and subgraph recognition.

[0186] The pattern matching algorithm uses a graph convolutional network (GCN) for pattern recognition, identifies subgraphs that match the target pattern, and extracts frequent subgraphs.

[0187] Step S3: Relational model construction and application

[0188] Build a relational model, use graph neural network (GNN) to model and predict complex relationships between entities, and train graph neural network to capture the dependencies and dynamic changes between entities.

[0189] Apply the model to apply the trained relationship model to specific scenarios for data analysis and insights.

[0190] Step S4: Data analysis and visualization

[0191] Data analysis uses graph-based visualization technology to present the mined relationships and patterns as interactive graphs. Through the graphs, users can conduct in-depth analysis and understand the complex relationships and patterns in the data.

[0192] Visualization technology, choose appropriate graph visualization tools and platforms (such as Gephi, Cytoscape)

[0193] Design an interactive interface that allows users to query and analyze nodes and edges in the graph, and provide graph filtering, zooming, and layout functions to facilitate flexible exploration and analysis.

[0194] The same or similar reference numerals correspond to the same or similar components;

[0195] The terms used in the drawings to describe positional relationships are only used for illustrative purposes and should not be construed as limiting this patent;

[0196] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made on the basis of the above description. It is not necessary and impossible to list all the implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A big data relationship mining method based on graph computing, characterized in that: The following steps are involved: S1. Preprocessing multi-source heterogeneous big data, including data cleaning, entity recognition, relationship extraction and type labeling, to generate graph structure data, wherein the preprocessing step includes using natural language processing technology to perform semantic role labeling on text data to identify semantic relationships between entities; S2. Mining frequent subgraphs and dynamic relationships from a graph using graph computing technology, wherein the graph computing technology includes a pattern matching algorithm that combines feature extraction of deep learning with traditional graph traversal technology to improve the accuracy and efficiency of subgraph mining; S3. Build a relationship model based on the mining results and apply it to specific scenarios for data analysis and insights, where the relationship model includes a graph-based machine learning framework for training and predicting the development and changes of complex relationships between entities.

2. The big data relationship mining method based on graph computing according to claim 1 is characterized in that: The data cleaning step includes using data deduplication and noise filtering technology, and the data deduplication uses a local sensitive hashing method, which includes the following steps: Convert each data record into a feature vector; Choose a hash function that suits the characteristics of your data; Using the selected hash function, mapping each of the feature vectors to one or more hash buckets, each of the hash buckets containing a plurality of the feature vectors, and the feature vectors of each hash bucket are relatively close in the hash space; For each new data record, calculate the hash value of the feature vector and map it to the corresponding hash bucket; Performing duplicate detection in the hash bucket, comparing the similarity of all the feature vectors in the same hash bucket, identifying records with a similarity higher than a set threshold as duplicate data, removing the identified duplicate data from the data set, and retaining unique data records; The noise filtering technology includes using a statistical method and a machine learning method, and specifically includes the following steps: The statistical methods include: standard deviation analysis, calculating the standard deviation of each data point and identifying those data points that deviate greatly from the mean; Distribution anomaly detection, using box plots to identify outliers in the data; Apply the Z-Score method, where data points with a Z-Score greater than 3 or less than -3 are considered outliers; The machine learning-based method uses local anomaly factors, specifically: Distance calculation, use Euclidean distance to calculate neighborhood distance, for each data point, calculate its distance to all other data points; Construct the local density, select the k value, and determine the size of the local neighborhood, that is, the k nearest neighbors of each data point; Calculate k distances. For each data point, find the maximum distance among its k nearest neighbors. Calculate the reachable distance. For data points i and j, calculate their reachable distance, which is defined as the larger value between k distances i and j; Local density, calculate the local density of each data point, that is, the density of the data points in its k neighborhoods; Relative density, calculate the relative density of each data point, that is, the ratio of its local density to the local density of the data points in its neighborhood; LOF value, calculate the LOF value of each data point, that is, the average value of its relative density, set the threshold, and set the threshold according to the LOF value. Data points with LOF values ​​exceeding the threshold are considered outliers; Filter outliers to remove or mark these abnormal points from the dataset.

3. The big data relationship mining method based on graph computing according to claim 2 is characterized in that: The entity recognition step includes using a named entity recognition model based on deep learning, wherein the recognition model is a model combining a long short-term memory network and a conditional random field, and the specific method steps are as follows: Obtain an annotated text dataset, which contains annotated entity information; construct a vocabulary and a tag set, wherein the vocabulary maps words in the text to unique index values; The tag set defines entity tag categories; Convert words to vectors using pre-trained word embeddings or custom word embedding layers; Bi-LSTM Network: The input layer uses word embedding as the input of Bi-LSTM. The Bi-LSTM layer uses bidirectional LSTM to capture contextual information. CRF layer: Sequence labeling: Use the CRF layer to label the output of Bi-LSTM and optimize entity boundary recognition; Model training, using the log-likelihood of CRF as the loss function, using the Adam optimization algorithm to train the model, dividing the data into training set and validation set, performing model training, and adjusting hyperparameters to optimize performance.

4. The big data relationship mining method based on graph computing according to claim 3 is characterized in that: The relationship extraction step includes using a deep learning-based relationship extraction model, which uses a pre-trained language model to perform relationship prediction, and includes the following steps: Obtain a large-scale corpus containing entity pairs and relation annotations; Organize the text into a format suitable for model input and divide the dataset into training, validation and test sets; Text tokenization, using BERT’s tokenizer to split the text into vocabulary indices; Generate input and create input samples for each entity pair, including: Token IDs, vocabulary index of text; Attention Masks, indicating which locations should be paid attention to by the model; Token Type IDs, used to distinguish between two different parts of text; Model building, loading pre-trained BERT, and selecting the appropriate BERT variant; BERT layer, extracting contextual information from text; Relation classification head, which adds a fully connected layer on the CLS token output by BERT for relation classification; Define the loss function and use the cross entropy loss function to optimize the classification task; Perform model training and input new text into the trained model to perform relationship prediction and map the relationship categories output by the model back to the actual relationship labels.

5. The big data relationship mining method based on graph computing according to claim 4 is characterized in that: The pattern matching algorithm in the graph computing technology includes using a graph convolutional network to perform feature learning and pattern recognition on graph structure data, wherein the GCN is used to learn the local and global features of nodes. The specific steps are: Graph data collection, obtaining graph structure data with nodes and edges; Data formatting: converting data into a format suitable for GCN input, including node feature matrix and adjacency matrix, and standardizing node features to improve training results; Adjacency matrix processing: normalize the adjacency matrix and calculate the inverse square root of the degree matrix of each node's adjacency matrix; Convolution operation, applies convolution operation to each node and its neighborhood to aggregate the information of neighboring nodes; Activation function, use ReLU activation function to increase the nonlinear expression ability of the model; Stack GCN layers and stack multiple graph convolutional layers as needed to capture multi-level features of nodes; Output layer: Add appropriate output layer for subgraph mining tasks, such as fully connected layer or pooling layer, to train the constructed model; Feature extraction, extracting node and subgraph features from GCN output; Subgraph matching, by comparing the extracted features with the target pattern, identifying and extracting subgraphs that match a specific pattern; Deploy the trained GCN model into the production environment, integrate it into practical applications, use the model for real-time subgraph matching and pattern recognition, and provide data analysis and decision support.

6. The big data relationship mining method based on graph computing according to claim 5 is characterized in that: The graph traversal technology includes using a distributed graph computing framework to process large-scale graph data to accelerate the subgraph mining process; Confirm the Spark cluster version and download the GraphX ​​package compatible with the Spark version; Upload the GraphX ​​package to all nodes of the Spark cluster and place it in the specified library directory; Adjust the spark.executor.memory and spark.driver.memory parameters to allocate appropriate memory; Set the spark.cores.max and spark.executor.instances parameters to configure the number of cores and executor instances; Data loading, using Spark's textFile or sequenceFile method to load data from HDFS, S3, or other storage systems; Clean the data, remove invalid or erroneous records, and convert the raw data into vertex and edge formats; Create vertex RDD and edge RDD, ensuring that the data types are consistent with GraphX ​​requirements; Create a vertex RDD and use Spark's parallelize method to create a vertex RDD, assigning a unique ID and corresponding attributes to each vertex; Create edge RDD, use Spark's parallelize method to create edge RDD, define the source vertex, target vertex and attributes of the edge; Build a Graph object, pass the vertex RDD and edge RDD to the Graph constructor, create a Graph object, and set the default vertex attributes for the Graph; Define graph calculation logic: Select appropriate graph algorithms based on business needs and implement algorithm logic; Use GraphXAPI to call GraphX's pregel and mapVertices APIs for graph calculations; Write user-defined functions to manipulate vertex and edge properties; Distributed graph computing, graph partitioning, and choosing appropriate partitioning strategies; Consider the characteristics of the graph to choose the best partitioning strategy; Parallel computing uses the distributed computing capabilities of GraphX ​​to split graph data into multiple partitions and process them in parallel on the cluster, monitor the progress of tasks and resource usage, and ensure that computing tasks run efficiently.

7. The big data relationship mining method based on graph computing according to claim 6 is characterized in that: The graph-based machine learning framework in the relational model includes using a graph neural network to model and predict complex relationships between entities, wherein the graph neural network is used to capture dependencies and dynamic changes between entities.

8. The big data relationship mining method based on graph computing according to claim 7 is characterized in that: The data analysis and insight step includes using graph-based visualization technology to present the mined relationships and patterns as interactive graphs to facilitate users to conduct in-depth analysis and understanding.

9. A computer storage medium, characterized in that The computer storage medium stores computer instructions, which, when called, are used to execute a big data relationship mining method based on graph computing as described in any one of claims 1-8.

Images