Information analysis method based on big data processing framework

Abstract

The invention relates to the technical field of big data processing and information analysis, in particular to an information analysis method based on a big data processing framework, which constructs a domain-specific knowledge graph by introducing domain expert knowledge and literature data, and constructs a graph convolutional neural network model based on the knowledge graph. Semantic analysis and relation reasoning are carried out on the unstructured data, an association rule algorithm is utilized, modes and rules among the data are found, a more valuable association analysis result is generated, and the problem that when an existing information analysis technology is used for processing the multi-source heterogeneous unstructured data, the data processing efficiency is high is solved. The problems of difficulty in data fusion, insufficient semantic understanding and insufficient relationship mining depth exist.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing and information analysis, and in particular to an information analysis method based on a big data processing framework. Background Art

[0002] With the rapid development of the Internet and information technology, the amount of data generated by all walks of life is growing exponentially, especially unstructured data. The continuous maturity of big data processing frameworks has provided strong support for the storage, processing and analysis of massive data. In this context, how to extract valuable information from multi-source heterogeneous unstructured data has become a research hotspot.

[0003] Traditional information analysis methods mainly rely on structured data and use technical means such as statistics and machine learning. However, when faced with unstructured data, these methods have limitations in data understanding, semantic parsing, and relationship mining.

[0004] Existing technologies attempt to combine natural language processing (NLP) with machine learning to process unstructured data. However, these methods usually only target a single data source and lack the ability to integrate and comprehensively analyze multi-source data. In addition, in terms of relationship mining, traditional methods find it difficult to effectively capture the complex relationships between entities, resulting in insufficient accuracy and depth in information extraction. Although knowledge graphs, as an emerging technology, can represent domain knowledge in a structured manner, in practical applications, building high-quality knowledge graphs still faces the problems of sparse data and untimely updates. At the same time, how to effectively combine knowledge graphs with deep learning models to improve the effectiveness of information analysis remains a difficult problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract of the specification and the title of the invention of this application to avoid blurring the purpose of this section, the abstract of the specification and the title of the invention, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0006] In view of the above existing problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by the present invention is that when processing multi-source heterogeneous unstructured data, existing information analysis technologies have problems such as difficulty in data fusion, insufficient semantic understanding, and insufficient depth of relationship mining.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: by using a data acquisition module, unstructured data from multiple data sources are obtained, wherein the unstructured data includes text data, social media data, and sensor data, and data preprocessing is performed on the unstructured data to obtain a sample data set;

[0009] Combine expert experience and literature to obtain knowledge about concepts, entities, and relationships in the field, and structure the knowledge to build a domain-specific knowledge graph;

[0010] Building a graph convolutional neural network model based on the knowledge graph to vectorize entities and relationships in the knowledge graph;

[0011] Using the graph convolutional neural network model, semantic parsing and relational reasoning are performed on the unstructured data in the sample data set to identify key entities and their attribute information and relationships;

[0012] Based on the entities and the relationships, using association rule algorithms, discovering patterns and rules between data, and generating association analysis results;

[0013] The association analysis results are presented in the form of charts and reports.

[0014] As a preferred solution of the information analysis method based on the big data processing framework described in the present invention, the text data includes news articles, technical documents, and email text data;

[0015] The social media data includes microblogs, blogs, forum posts, and user-generated content;

[0016] The sensor data includes sensor readings generated by IoT devices.

[0017] As a preferred solution of the information analysis method based on the big data processing framework described in the present invention, the text data, the social media data, and the sensor data are preprocessed respectively, and the preprocessed data are organized in a unified format to form a sample data set. The expression form of the sample data set is:

[0018] Each record contains the document ID, the word sequence after word segmentation, and the part-of-speech tagging result;

[0019] D 文本 ={(ID i ,W i ,P i )|i=1,2,…,N 文本}

[0020] Among them, D 文本 represents the text dataset, N 文本is the total number of texts in the text dataset;

[0021] Each record contains user ID, publishing time, cleaned text content and sentiment annotation;

[0022] D 社交 ={(UID i ,T i ,C i ,S i )|i=1,2,…,N 社交}

[0023] Among them, D 社交 represents the social media dataset, N 社交 is the total number of contents in the social media dataset;

[0024] Each record contains the device ID, timestamp, and normalized sensor reading;

[0025] D 传感器 ={(SID i ,t i ,V i )|i=1,2,…,N 传感器}

[0026] Among them, D 传感器 represents the sensor dataset, N 传感器 is the total number of data in the sensor dataset.

[0027] As a preferred solution of the information analysis method based on the big data processing framework described in the present invention, natural language processing technology is used to extract entities and relationships from the sample data set, and the extracted entities and relationships are represented in the form of triples, which are recorded as (h, r, t);

[0028] Among them, h represents the head entity, r represents the relationship between entities, and t represents the tail entity.

[0029] As a preferred solution of the information analysis method based on the big data processing framework described in the present invention, a graph database is used to store the triples in the knowledge base as a graph structure, with nodes representing entities and edges representing relationships, to form a domain-specific knowledge graph G=(E, R);

[0030] Among them, E is the entity set, which contains all identified entities, and R is the relationship set, which contains various relationships between entities.

[0031] As a preferred solution of the information analysis method based on the big data processing framework described in the present invention, a graph convolutional neural network model is constructed based on the knowledge graph to vectorize entities and relationships. The feature update formula is:

[0032]

[0033] in, is the feature vector of node i in the l+1th layer, N(i) is the set of neighbor nodes of node i, d i and d j are the degrees (number of connected edges) of nodes i and j respectively, and W (l) is the weight matrix of the lth layer, b (l) is the bias vector of the lth layer, and σ is the nonlinear activation function.

[0034] As a preferred solution of the information analysis method based on the big data processing framework described in the present invention, the graph convolutional neural network model maps the preprocessed text data into a vector representation through training, and for each word w, its vector representation v is obtained by searching the embedding matrix. w , and then calculate the similarity between the text word vector and the entity vector in the knowledge graph through cosine similarity, and compare the similarity value with the set similarity threshold. When the similarity value is greater than the set similarity threshold, the word w matches the entity e, and the key entity in the text is identified.

[0035] As a preferred solution of the information analysis method based on the big data processing framework described in the present invention, the formula of the cosine similarity is:

[0036]

[0037] Among them, v w is the vector representation of word w, h e is the vector representation of entity e, ∥·∥ represents the Euclidean norm of the vector;

[0038] Set the similarity threshold θ, when sim(v w ,h e )>θ, word w is matched with entity e, and the key entity in the text is identified.

[0039] As a preferred solution of the information analysis method based on the big data processing framework described in the present invention, for each possible relationship r, the score f(h, r, t) is calculated, and the relationship with the highest score and exceeding the threshold is selected as the predicted relationship between entities.

[0040] Beneficial effects of the present invention:

[0041] 1. It realizes the unified collection and standardized processing of multi-source heterogeneous unstructured data, plays a role in eliminating data noise, filling data gaps, unifying data formats, and improving data quality and consistency;

[0042] 2. It realizes the conversion of professional knowledge in the field into a structured form that can be understood by machines, enriches the semantics of data, enhances the system's understanding of domain knowledge, improves the accuracy of information extraction and relationship mining, and enhances the model's ability to handle complex semantic relationships;

[0043] 3. It realizes the embedding of high-dimensional knowledge graph information into low-dimensional vector space, which plays a role in capturing the complex dependencies between entities and relationships, improving the model's learning ability for graph structure data, and providing efficient and accurate feature representation for semantic parsing and relational reasoning;

[0044] 4. It realizes the transformation of unstructured data into structured knowledge, plays a role in automatically extracting key information and understanding the deep semantics of data, improves the efficiency and accuracy of information extraction, reduces manual intervention, and enhances the intelligence of the system;

[0045] 5. It realizes deep pattern mining of structured data, which plays a role in revealing the association rules and trends hidden in the data, provides a strong basis for decision support and predictive analysis, and enhances the value of data;

[0046] 6. Help users quickly understand and interpret the role of analysis results and improve the efficiency of information transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0048] Figure 1 It is a flow chart of the information analysis method based on the big data processing framework shown in the present invention;

[0049] Figure 2 It is a schematic diagram of the information extraction accuracy comparison curve shown in the present invention;

[0050] Figure 3 It is a schematic diagram of the comparison curve of information extraction recall rate shown in the present invention;

[0051] Figure 4 It is a schematic diagram of the comparison curve of the average improvement of association rules shown in the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0053] Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without making any creative work should fall within the scope of protection of the present invention.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0055] According to an embodiment of the present invention, Figure 1 The flowchart shown is an information analysis method based on a big data processing framework, which specifically includes the following steps:

[0056] S1. Obtain unstructured data from various data sources through a data acquisition module. Unstructured data includes text data, social media data, and sensor data. Perform data preprocessing on the unstructured data to obtain a sample data set.

[0057] S2. Combine expert experience and literature to obtain knowledge about concepts, entities, and relationships in the field, and structure the knowledge to build a domain-specific knowledge graph;

[0058] S3. Build a graph convolutional neural network model based on the knowledge graph to vectorize the entities and relationships in the knowledge graph;

[0059] S4. Use the graph convolutional neural network model to perform semantic parsing and relational reasoning on the unstructured data in the sample dataset to identify key entities and their attribute information and relationships;

[0060] S5. Based on entities and relationships, use association rule algorithms to discover patterns and rules between data and generate association analysis results;

[0061] S6. Display the results of the association analysis in the form of charts and reports.

[0062] The following describes in more detail the implementation process and / or effects of certain embodiments of the present invention in conjunction with some preferred or optional examples of the present invention.

[0063]

Data collection and preprocessing

[0064] Use the data acquisition module to obtain unstructured data from various data sources, including but not limited to text data, social media data, and sensor data. The data types include character, numerical, and multimedia.

[0065] As an example, text data: includes data in the form of news articles, technical documents, and email texts;

[0066] As an example, social media data: includes user-generated content such as Weibo posts, blogs, and forum threads;

[0067] As an example, sensor data: includes sensor readings generated by Internet of Things devices, such as temperature, humidity, and location information;

[0068] Preprocess different types of unstructured data separately to obtain a sample dataset, which specifically includes the following steps:

[0069] (1) Preprocessing of text data

[0070] Unify character encoding: Convert all text data into a unified encoding format (such as UTF-8) to ensure consistent character sets;

[0071] Word segmentation: Use a Chinese word segmentation algorithm (such as Jieba) to divide the text into a sequence of words. For example, segment the sentence "Today's weather is very good" into "Today", "weather", "very", "good";

[0072] Stop word filtering: Remove commonly used but meaningless words, such as "de", "le", "he";

[0073] Lemmatization: Restore words to their original forms and unify synonyms and near-synonyms. For example, restore "running" and "running" to "run";

[0074] (2) Preprocessing of social media data:

[0075] Special symbol processing: Remove emojis, special characters, and URL links, and retain useful text information;

[0076] Spelling correction: Use a spelling check algorithm to correct typos and colloquial expressions. For example, correct "虾米" to "什么";

[0077] Sentiment annotation: Perform sentiment analysis on the text and annotate the sentiment tendency (positive, neutral, negative);

[0078] (3) Preprocessing of sensor data

[0079] Data cleaning: Remove outliers and noisy data. For example, delete temperature values that exceed a reasonable range;

[0080] Missing value filling: Use methods such as mean and interpolation to fill in missing data points;

[0081] Data normalization: Scale the data to a specific range (such as [0,1]) to eliminate the influence of different dimensions;

[0082] Furthermore, the preprocessed data is organized in a unified format to form a sample data set. The data set is expressed as follows:

[0083] Text dataset: Each record contains the document ID, the word sequence after word segmentation, and the part-of-speech tagging result;

[0084] D 文本 ={(ID i ,W i ,P i )|i=1,2,…,N 文本}

[0085] Among them, D 文本 represents the text dataset, N 文本 is the total number of texts in the text dataset;

[0086] For example, suppose there is a document with document ID 1 and content “The weather is very good today”. After word segmentation, the word sequence is: W1 = [“today”, “weather”, “very”, “good”], and the part-of-speech tagging result P1

[0087] =[noun, noun, adverb, adjective], the record is represented as: (ID1=1, W1, P1);

[0088] Social media dataset: Each record contains user ID, posting time, cleaned text content and sentiment annotation;

[0089] D 社交 ={(UID i ,T i ,C i ,S i )|i=1,2,…,N 社交}

[0090] Among them, D 社交 represents the social media dataset, N 社交 is the total number of contents in the social media dataset;

[0091] For example, suppose there is a piece of social media content, the user ID is "user123", the publishing time is "2024-11-07 10:00:00", and the original content is "I am so happy today!" The cleaned content is: C i

[0092] = "I am very happy today"; emotion labeling result: S i = positive; the record is represented by:

[0093] (UID i = "user123", T i="2024-11-07 10:00:00",C i ,S i );

[0094] Sensor dataset: Each record contains device ID, timestamp, and normalized sensor readings;

[0095] D 传感器 ={(SID i ,t i ,V i )|i=1,2,…,N 传感器}

[0096] Among them, D 传感器 represents the sensor dataset, N 传感器 is the total number of data in the sensor dataset;

[0097] For example, suppose there is a sensor data with device ID "sensor001", timestamp "2024-11-0710:00:00", measurement parameters are temperature and humidity, and original reading: Temperature: Temp raw =25℃, humidity: Hum raw =60%, after normalization:

[0098] V i =[v i1 ,v i2 ]=[0.5,0.6]

[0099] The record is represented as: (SID i = "sensor001", t i ="2024-11-07 10:00:00",V i ).

[0100]

Building a knowledge graph

[0101] Through expert experience and literature, concepts, entities and relationships in the field are obtained, including professional terms, definitions and logical relationships; among which:

[0102] Expert experience: Organize discussions with experts in the field to sort out professional terms, definitions and logical relationships;

[0103] Literature: Use web crawling technology to capture and analyze papers, patents, and technical reports in the field to extract valuable knowledge points;

[0104] Entities and relations are extracted from the preprocessed data using natural language processing techniques, using named entity recognition (NER) and relation extraction algorithms;

[0105] The extracted entities and relations are represented in the form of triples, denoted as (h, r, t), where h represents the head entity, r represents the relationship between entities, and t represents the tail entity.

[0106] Use a graph database (such as Neo4j) to store the triples in the knowledge base as a graph structure, with nodes representing entities and edges representing relationships, to form a domain-specific knowledge graph G = (E, R), where:

[0107] E is the entity set, which contains all identified entities;

[0108] R is a relationship set, which contains various relationships between entities;

[0109] As an example, use the pre-trained NER model to identify entities in the text, such as people, places, and organizations. For example, in the sentence "Li Lei studies at Peking University", the entities "Li Lei" and "Peking University" are identified;

[0110] Use relation extraction algorithms to identify relationships between entities, such as "belongs to" and "is located in";

[0111] The extracted entities and relations are represented in the form of triples to form a knowledge base, for example:

[0112] (Li Lei, studying at Peking University)

[0113] (Peking University, located in,Beijing).

[0114] [Identification of entities and relationships]

[0115] Assign an initial feature vector to each entity and relation in the knowledge graph, using either random initialization or pre-trained embeddings:

[0116] Entity feature vector Where d is the vector dimension;

[0117] Relation feature vector:

[0118] A graph convolutional neural network (GCN) model is established to vectorize entities and relationships. The feature update formula is:

[0119]

[0120] in, is the feature vector of node i in the l+1th layer, N(i) is the set of neighbor nodes of node i, d i and d j are the degrees (number of connected edges) of nodes i and j respectively, and W (l)is the weight matrix of the lth layer, b (l) is the bias vector of the lth layer, σ is a nonlinear activation function, such as ReLU,

[0121] Using known entity relationship pairs, the model is trained using supervised learning, and the loss function is:

[0122]

[0123] Among them, S is the positive sample set, which contains the real triples (h, r, t), S ′ is a negative sample set, containing randomly sampled error triplets (h ′ ,r ′ ,t ′ ), is a scoring function, which indicates the matching degree between entities h and t through relationship r, and σ is a Sigmoid function;

[0124] During the model training process, the stochastic gradient descent (SGD) optimization algorithm is used to update the model parameters. The parameter update formula is:

[0125]

[0126] Among them, θ (t) is the model parameter of the tth iteration, η is the learning rate, is the gradient of the loss function with respect to the parameter θ;

[0127] Using the trained graph convolutional neural network model, the preprocessed text data is mapped into a vector representation. For each word w, its vector representation v is obtained by looking up the embedding matrix. w ;

[0128] Calculate the similarity between the text word vector and the entity vector in the knowledge graph using cosine similarity, the formula is:

[0129]

[0130] Among them, v w is the vector representation of word w, h e is the vector representation of entity e, ∥·∥ represents the Euclidean norm of the vector;

[0131] Set the similarity threshold θ, when sim(v w ,h e )>θ, word w matches entity e and identifies the key entity in the text;

[0132] For each possible relationship r, calculate the score f(h, r, t), and select the relationship with the highest score that exceeds the threshold as the predicted relationship between entities;

[0133] For the identified entity pair (h, t), the relationship score is calculated using the relationship prediction function f(h, r, t):

[0134]

[0135] in, and h t are the vector representations of the head entity h and the tail entity t, respectively, r is the weight matrix of relation r, b r is the bias term of relation r;

[0136] For entities that are not directly connected, define the optimization objective function:

[0137]

[0138] Among them, U is the set of unconnected entity pairs, cos(h h ,h t ) is the cosine similarity between entity vectors;

[0139] By minimizing L ′ , mining potential relationships between entities and improving the depth of relationship mining;

[0140] Exemplarily, in the relational reasoning and mining steps, the attention mechanism is used to optimize the graph neural network, and the attention coefficient calculation formula is:

[0141]

[0142] Among them, α ij is the attention coefficient of node i to its neighbor node j, a T is the learnable weight vector, [Wh i ||Wh j ] represents the concatenation of the features of node i and node j, and LeakyReLU is the activation function, which is defined as:

[0143]

[0144] Generate association analysis results

[0145] The identified entities and their relationships are grouped into a transaction dataset D. Each transaction T i It is expressed as:

[0146] T i ={h i ,r i ,t i}

[0147] For example, transaction T1 = {Li Lei, studying at, Peking University};

[0148] Use the Apriori algorithm, set the minimum support δ, and find frequent item sets. Specifically, the following steps are included:

[0149] Generate candidate item sets: construct candidate sets of item sets;

[0150] Support calculation: Count the number of times each candidate item set appears in D and calculate the support;

[0151] Pruning: remove itemsets with support less than δ;

[0152] For example, support is defined as:

[0153]

[0154] Among them, A is the item set;

[0155] Generate association rules that satisfy the minimum confidence γ from the frequent item set, and the rule form is A→B;

[0156] Among them, A and B are frequent item sets, and

[0157] The confidence level is defined as:

[0158]

[0159] Among them, Supp(A) is the support of item set A, that is, the frequency of occurrence of A, and Conf(A→B) is the confidence of the rule, which indicates the probability of B occurring when A occurs;

[0160] Calculate indicators such as lift, evaluate the effectiveness of the rules, and calculate lift (A→B):

[0161]

[0162] Among them, Lift(A→B) is the lift of the rule, which measures the strength of the association between A and B;

[0163] When the lift is greater than 1, it means that the rule has a positive correlation;

[0164] Exemplarily, using visualization tools (such as D3.js), the association analysis results are displayed in the form of a network diagram or a heat map, for example, a network diagram showing the relationship between entities, where nodes represent entities and edges represent relationships; a report containing analysis results and suggestions is automatically generated, including:

[0165] Abstract: summarizes the main findings and conclusions of the analysis;

[0166] Detailed results: List important association rules, frequent itemsets and their support, confidence and lift;

[0167] Recommendations: Propose feasible strategies and action plans based on the analysis results;

[0168] For example, in an analysis report in the financial field, it was found that there is a strong correlation between "high income" and "purchasing high-end financial products", and it is recommended to promote high-end financial products to high-income customers.

[0169] It should be noted that traditional information analysis methods usually rely on keyword-based text mining or simple machine learning models, such as naive Bayes classifiers and support vector machines. These methods mainly target a single type of data and lack the ability to comprehensively process multi-source heterogeneous unstructured data. In addition, traditional methods have limitations in semantic understanding and relational reasoning, and are unable to fully explore the deep semantics and potential associations in the data, resulting in low accuracy in information extraction and incomplete association analysis results.

[0170] In order to fully verify the effectiveness and superiority of the method of the present invention in the field of information analysis, this embodiment selects the financial field as the experimental object. The data in the financial field is complex and diverse, and the accuracy and depth of information analysis are required to be high, which is suitable as a test scenario for the method of the present invention.

[0171] This experiment aims to verify the effectiveness of the method of the present invention in the field of information analysis, especially its advantages in processing multi-source heterogeneous unstructured data, deep semantic analysis and relational reasoning. By comparing with traditional methods, the improvement of the method of the present invention in key indicators such as information extraction accuracy, relationship mining depth and association analysis effect is evaluated.

[0172]

Experimental data

[0173] (1) Text data

[0174] Source: A collection of 1,000 news reports and market analysis reports from authoritative financial media, including The Wall Street Journal, Financial Times, and Reuters;

[0175] Content: covers macroeconomic indicators, company financial reports, market trend analysis, etc., involving multiple financial sub-fields such as stocks, bonds, and foreign exchange;

[0176] (2) Social media data

[0177] Source: 1,000 user comments and discussion posts from Weibo, Xueqiu, Zhihu and other platforms were obtained;

[0178] Content: reflects investor sentiment, market expectations, and discussions on hot issues;

[0179] (3) Sensor data

[0180] Source: 1,000 pieces of real-time market data from stock trading systems, including price, trading volume, and order depth;

[0181] Content: Covers the high-frequency trading data of the constituent stocks of the CSI 300 Index;

[0182] A total of 3,000 unstructured data items, covering a variety of data types and sources, are representative and contain professional terms, spoken expressions, emotional factors, and high-frequency numerical data, which increases the difficulty of information extraction and analysis. They contain noise, redundancy, and inconsistency, and require effective preprocessing and cleaning;

[0183]

Experimental environment conditions

[0184] CPU: Intel Xeon Platinum 8280, 28 cores, main frequency 2.7GHz, supports multi-threaded parallel computing;

[0185] Memory: 512GB DDR4 ECC memory to meet the needs of large-scale data processing and model training;

[0186] GPU: NVIDIA Tesla V100 with 32GB video memory, supporting accelerated training of deep learning models;

[0187] Storage: 4TB NVMe SSD high-speed solid-state hard drive to ensure data read and write speed and system response efficiency;

[0188] Operating system: Ubuntu 20.04LTS, providing a stable server operating environment;

[0189] Programming language: Python 3.8, with rich scientific computing and machine learning libraries;

[0190] Deep learning framework: PyTorch 1.7, used to build and train graph convolutional neural network models;

[0191] Data processing library: Pandas 1.2.0, for data reading, cleaning and preprocessing;

[0192] Natural language processing libraries: NLTK 3.5, SpaCy 2.3, for word segmentation, part-of-speech tagging, and named entity recognition;

[0193] Graph processing library: NetworkX 2.5, for building and manipulating knowledge graphs;

[0194] Association rule mining library: mlxtend 0.18, used to implement the Apriori algorithm;

[0195] Visualization library: Matplotlib 3.3, Seaborn 0.11, for data visualization and result presentation;

[0196] Random seed: To ensure the repeatability of the experiment, the random seed is set to 42;

[0197] Graph Convolutional Neural Network (GCN): embedding dimension 128, learning rate 0.01, training epochs 300, batch size 64;

[0198] Association rule mining: minimum support 0.05, minimum confidence 0.6;

[0199] Evaluation indicators: information extraction accuracy, recall rate, F1 value, and average improvement of association rules;

[0200]

Experimental indicators

[0201] Information extraction accuracy (Precision), information extraction recall (Recall), F1 value (F1Score), average lift of association rules (Lift), model training time (Training Time);

[0202] [Experimental implementation process]

[0203] Traditional method implementation

[0204] Data preprocessing: perform word segmentation and stop word filtering on text data and social media data, generate bag-of-words models, and standardize sensor data;

[0205] Information extraction: Use the naive Bayes classifier to extract key entities based on TF-IDF features;

[0206] Association analysis: Based on the extracted entities, the Apriori algorithm is used to mine association rules;

[0207] Implementation of the method of the present invention

[0208] Data preprocessing: perform word segmentation, part-of-speech tagging, and named entity recognition on text data and social media data, and normalize and process outliers on sensor data;

[0209] Knowledge graph construction: Organize concepts, entities, and relationships in the financial field, including companies, stocks, and market events, and build a knowledge graph with 1,000 nodes and 5,000 edges;

[0210] Model training: Use the GCN model to train the knowledge graph to obtain vectorized representations of entities and relationships;

[0211] Semantic parsing and relational reasoning: Use the trained GCN model to perform deep semantic parsing on the sample data set, identify key entities and their attributes, perform relational reasoning, and mine potential associations between entities;

[0212] Association analysis: Use improved association rule algorithms to mine patterns and regularities among data based on identified entities and relationships.

[0213] Reference Figure 2 It can be seen intuitively that with the increase in the number of training rounds, the accuracy of the two methods has improved. However, the accuracy of the method of the present invention is always higher than that of the traditional method, and the improvement is more obvious. When the number of training rounds is 300, the accuracy of the method of the present invention reaches 92%, while the accuracy of the traditional method is only 74%. This shows that the method of the present invention can more effectively extract key information and has higher accuracy when processing multi-source heterogeneous unstructured data.

[0214] Reference Figure 3 The recall rate of the method of the present invention is always ahead of the traditional method, and the gap widens with the increase of the number of training rounds. When the number of training rounds is 300, the recall rate of the method of the present invention reaches 91%, which is 21 percentage points higher than that of the traditional method. This shows that the method of the present invention has significant advantages in comprehensively identifying relevant information, can effectively reduce the missed detection rate, and improve the integrity of information extraction.

[0215] Reference Figure 4 , with the increase of the minimum support, the average lift of the proposed method shows an overall upward trend, and reaches a peak of 4.0 when the minimum support is 0.05, and then decreases slightly. This is because at a higher minimum support, the mined association rules are more representative and relevant; in contrast, the average lift of the traditional method reaches 2.0 when the minimum support is 0.03, and then tends to be stable, and even decreases slightly when the minimum support continues to increase, which shows that the traditional method has limitations in mining high-quality association rules and cannot fully utilize the potential patterns in the data.

[0216] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An information analysis method based on a big data processing framework, characterized in that: include: Obtain unstructured data from a variety of data sources through a data acquisition module, wherein the unstructured data includes text data, social media data, and sensor data, and perform data preprocessing on the unstructured data to obtain a sample data set; Combine expert experience and literature to obtain knowledge about concepts, entities, and relationships in the field, and structure the knowledge to build a domain-specific knowledge graph; Building a graph convolutional neural network model based on the knowledge graph to vectorize entities and relationships in the knowledge graph; Using the graph convolutional neural network model, semantic parsing and relational reasoning are performed on the unstructured data in the sample data set to identify key entities and their attribute information and relationships; Based on the entities and the relationships, using association rule algorithms, discovering patterns and rules between data, and generating association analysis results; The association analysis results are presented in the form of charts and reports.

2. The information analysis method based on the big data processing framework according to claim 1 is characterized in that: The text data includes news articles, technical documents, and email text data; The social media data includes microblogs, blogs, forum posts, and user-generated content; The sensor data includes sensor readings generated by IoT devices.

3. The information analysis method based on the big data processing framework according to claim 1 or 2 is characterized in that: The text data, the social media data, and the sensor data are preprocessed respectively, and the preprocessed data are organized in a unified format to form a sample data set. The expression form of the sample data set is: Each record contains the document ID, the word sequence after word segmentation, and the part-of-speech tagging result; D 文本 ={(ID i ,W i ,P i )|i=1,2,…,N 文本 } Among them, D 文本 represents the text dataset, N 文本 is the total number of texts in the text dataset; Each record contains user ID, publishing time, cleaned text content and sentiment annotation; D 社交 ={(UID i ,T i ,C i ,S i )|i=1,2,…,N 社交 } Among them, D 社交 represents the social media dataset, N 社交 is the total number of contents in the social media dataset; Each record contains the device ID, timestamp, and normalized sensor reading; D 传感器 ={(SID i ,t i ,V i )|i=1,2,…,N 传感器 } Among them, D 传感器 represents the sensor dataset, N 传感器 is the total number of data in the sensor dataset.

4. The information analysis method based on the big data processing framework according to claim 3 is characterized in that: Utilizing natural language processing technology, entities and relations are extracted from the sample data set, and the extracted entities and relations are represented in the form of triples, which are recorded as (h, r, t); Among them, h represents the head entity, r represents the relationship between entities, and t represents the tail entity.

5. The information analysis method based on the big data processing framework according to claim 4 is characterized in that: Use a graph database to store the triples in the knowledge base as a graph structure, where nodes represent entities and edges represent relationships, to form a domain-specific knowledge graph G = (E, R); Among them, E is the entity set, which contains all identified entities, and R is the relationship set, which contains various relationships between entities.

6. The information analysis method based on the big data processing framework according to claim 5 is characterized in that: A graph convolutional neural network model is constructed based on the knowledge graph to vectorize entities and relationships. The feature update formula is: in, is the feature vector of node i in the l+1th layer, N(i) is the set of neighbor nodes of node i, d i and d j are the degrees (number of connected edges) of nodes i and j respectively, and W (l) is the weight matrix of the lth layer, b (l) is the bias vector of the lth layer, and σ is the nonlinear activation function.

7. The information analysis method based on the big data processing framework according to claim 6 is characterized in that: The graph convolutional neural network model is trained to map the preprocessed text data into vector representations. For each word w, its vector representation v is obtained by looking up the embedding matrix. w , and then calculate the similarity between the text word vector and the entity vector in the knowledge graph through cosine similarity, and compare the similarity value with the set similarity threshold. When the similarity value is greater than the set similarity threshold, the word w matches the entity e, and the key entity in the text is identified.

8. The information analysis method based on the big data processing framework according to claim 7 is characterized in that: The formula for the cosine similarity is: Among them, v w is the vector representation of word w, h e is the vector representation of entity e, ∥·∥ represents the Euclidean norm of the vector; Set the similarity threshold θ, when sim(v w ,h e )>θ, word w is matched with entity e, and the key entity in the text is identified.

9. The information analysis method based on the big data processing framework according to claim 7 is characterized in that: For each possible relationship r, the score f(h, r, t) is calculated, and the relationship with the highest score that exceeds the threshold is selected as the predicted relationship between entities.

Images