Multi-source heterogeneous data fusion and analysis method and system

Abstract

The invention discloses a multi-source heterogeneous data fusion and analysis method and system. The method comprises the following steps: step 1, collecting data from a plurality of different types of data sources; step 2, carrying out preprocessing of data cleaning on the collected data; 3, fusing the data by using a natural language processing technology and a machine learning algorithm to generate a unified data view; 4, analyzing the fused data by adopting a data mining and analysis technology, and extracting required information; and 5, displaying a result obtained through analysis in the step 4 through a visual tool. The invention further discloses a system corresponding to the method. According to the method, the efficiency and accuracy of multi-source heterogeneous data processing can be remarkably improved, and the manual operation time is shortened; efficient acquisition, cleaning, fusion and analysis of data are realized through an automation technology; the generated analysis result is accurate and reliable, and powerful support can be provided for decision making.

Description

Technical Field

[0001] The invention discloses a multi-source heterogeneous data fusion and analysis method and system, and relates to the technical field of computer information technology and data processing and analysis. Background Art

[0002] With the advent of the big data era, diverse information sources such as city official websites, social media, news websites, and corporate data have made data processing and analysis increasingly complex. These data sources usually involve multiple heterogeneous formats, including structured database records, semi-structured XML or JSON files, and unstructured text, images, and videos.

[0003] Existing technologies have several limitations and challenges:

[0004] 1. Insufficient heterogeneous data fusion: Although existing technologies support data collection from multiple data sources, there are still deficiencies in data fusion. Existing methods often cannot efficiently handle the fusion of data in different formats and structures, resulting in high time cost and low efficiency of data integration.

[0005] 2. Challenges of data cleaning and quality control: Existing technologies have not yet fully solved the problems of automatic data cleaning and quality control. In particular, the automatic removal of data redundancy and errors, as well as the processing of missing data, still rely on a large number of manual operations, which increases the complexity of the processing flow.

[0006] 3. Real-time data processing capability: With the continuous increase in data flow, how to process large-scale data in real time and provide fast data analysis and decision support is an important technical challenge.

[0007] 4. The need for intelligent decision support systems: Faced with complex data relationships and a rapidly changing market environment, more intelligent decision support systems are needed to achieve efficient data analysis and accurate forecasting recommendations. Summary of the invention

[0008] The technical problem to be solved by the present invention is: in view of the defects of the prior art, a multi-source heterogeneous data fusion and analysis method and system are provided to solve the problems of data fusion, cleaning, analysis and real-time processing mentioned in the background technology. Through the automated data processing flow and efficient data analysis mechanism, the method and system disclosed in the present invention not only improve the efficiency of data processing, but also ensure the reliability and accuracy of data processing.

[0009] The present invention adopts the following technical solution to solve the above technical problems: The present invention discloses a multi-source heterogeneous data fusion and analysis method, the method comprising:

[0010] Step 1: Collect data from multiple different types of data sources, including structured data, semi-structured data, and unstructured data.

[0011] Wherein, the data source includes:

[0012] 101. City official website: Use web crawler technology to regularly capture policies, announcements, and data information released on the city’s official website.

[0013] 102. Social Media: Use API interfaces to obtain real-time data on user content, comments, and interactions on social media.

[0014] 103. News website: Collect the latest news reports and articles through RSS subscription or web crawler technology.

[0015] 104. Enterprise data: Obtain relevant business data from the enterprise's internal database or API interface.

[0016] The data collection frequency and strategy formula used is:

[0017]

[0018] Among them, F d Indicates the frequency of data collection, N d It indicates the number of data requests per unit time, and T indicates the time period.

[0019] Step 2: Preprocess the collected data by cleaning the data. The data preprocessing includes: data deduplication, missing value processing and noise filtering.

[0020] Among them, data preprocessing specifically includes the following steps:

[0021] 201. Data deduplication: Use hash functions to generate unique identifiers for data records, identify and delete duplicate data records;

[0022] 202. Missing value processing: Use K nearest neighbor interpolation method or expectation maximization algorithm to process missing values;

[0023] 203. Noise filtering: Use regular expressions, stop words, and domain dictionaries to filter out noise and irrelevant information in data;

[0024] The data cleaning, deduplication and missing value processing formula used is:

[0025] D c =D raw -D dup -D miss

[0026] Among them, Dc represents the cleaned data set, D raw represents the original data set, D dup Indicates repeated data, D miss Indicates missing data.

[0027] Step 3: Use natural language processing technology and machine learning algorithms to integrate data and generate a unified data view. Specifically include:

[0028] 301. Entity Recognition: Use NLP technology to identify entities in data, and use bidirectional long short-term memory networks and conditional random field models to ensure the accuracy of entity recognition;

[0029] 302. Relationship Extraction: Use machine learning algorithms to extract the relationships between entities in data, and improve the effect of relationship extraction through word embedding technology and dependency syntax analysis;

[0030] 303. Data alignment: Align the same entity from different data sources to generate a unified entity view; use graph matching algorithms to ensure the accuracy of data alignment;

[0031] 304. Data integration: Integrate the aligned data into a unified data view and improve the quality of data integration through data normalization and consistency checking;

[0032] The entity recognition and alignment formula used in data fusion is:

[0033]

[0034] Among them, E f Represents the fused entity set, E i represents the entity set of the i-th data source, α i represents the weight of the i-th data source;

[0035] The relation extraction formula used is:

[0036]

[0037] Among them, R f represents the fused relation set, R i represents the relationship set of the i-th data source, β i represents the weight of the i-th data source;

[0038] The data integration formula used is:

[0039]

[0040] Among them, D f represents the integrated data set, D irepresents the dataset of the ith data source, and γ represents the function of data normalization and consistency check.

[0041] Step 4: Use data mining and analysis technology to analyze the fused data and extract the required information. Specifically include:

[0042] 401. Classification and Clustering: Use machine learning algorithms to classify and cluster data to discover patterns and trends in the data;

[0043] 402. Association analysis: Use association rule mining technology to analyze the association relationship in the data and find feature items with high association through frequent item set mining;

[0044] 403. Time Series Analysis: Analyze time series data to predict future trends and changes, and use autoregressive integrated moving average models and long short-term memory networks to improve the accuracy of predictions;

[0045] 404. Sentiment analysis: Conduct sentiment analysis on social media data to obtain public sentiment and public opinion trends; use sentiment dictionaries and sentiment classification models to ensure the accuracy of sentiment analysis;

[0046] The classification and clustering formulas used are:

[0047]

[0048] Among them, C represents the category, k represents the number of categories, and x i represents a data point, μ k represents the category center, δ represents the distance metric function;

[0049] The time series analysis formula used is:

[0050] y t =φ1y t_1 +φ2y t-2 +…+φ p y t-p +θ1∈ t-1 +θ2∈ t-2 +…+θ q ∈ t-q

[0051] Among them, y t represents the time series value, φ i represents the autoregressive parameter, θ i represents the moving average parameter, ∈ t represents white noise;

[0052] The sentiment analysis formula used is:

[0053]

[0054] Among them, S represents the sentiment score, w i Represents the weight of sentiment words, s i Indicates the sentiment intensity of sentiment words.

[0055] Step 5: Display the results obtained from step 4 through visualization tools. Specifically include:

[0056] 501. Real-time display of key indicators and analysis results through interactive dashboards;

[0057] 502. Use charts and graphs to present data analysis results, including;

[0058] 503. Automatically write and update analytical reports through natural language generation technology;

[0059] The data visualization formula used in the result display is:

[0060] V=f(D analysis )

[0061] Among them, V represents the visualization result, D analysis represents the analyzed data, and f represents the visualization function.

[0062] The present invention also discloses a multi-source heterogeneous data fusion and analysis system, which is equipped with the multi-source heterogeneous data fusion and analysis method mentioned above. The system architecture of the system includes a data acquisition module, a data cleaning module, a data fusion module, a data analysis module and a data display module which are connected in sequence.

[0063] Compared with the prior art, the present invention adopts the above technical scheme and has the following technical effects: the present invention can significantly improve the efficiency and accuracy of multi-source heterogeneous data processing and reduce manual operation time; through automation technology, efficient data collection, cleaning, fusion and analysis can be achieved; the generated analysis results are accurate and reliable, and can provide strong support for decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a system architecture diagram of the present invention, showing the composition of the entire system and the relationship between each module;

[0065] Figure 2 It is a data collection flow chart in the method of the present invention, which describes in detail the operation process of data collection;

[0066] Figure 3 is a data cleaning flow chart of the present invention, showing the operation steps of data cleaning;

[0067] Figure 4is a data fusion flow chart in the method of the present invention, showing the operation steps of data fusion;

[0068] Figure 5 It is a data analysis flow chart in the method of the present invention, showing the specific operation steps of data analysis. DETAILED DESCRIPTION

[0069] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0070] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0071] The system architecture diagram of the present invention is as follows: Figure 1 As shown, the system includes a data acquisition module, a data cleaning module, a data fusion module, a data analysis module and a data display module which are connected in sequence.

[0072] A multi-source heterogeneous data fusion and analysis method disclosed in the present invention comprises the following steps:

[0073] Step 1: Data collection: Automatically collect data from multiple different types of data sources, including structured data (such as databases), semi-structured data (such as JSON, XML), and unstructured data (such as text, images, audio, etc.).

[0074] Step 2: Data cleaning: Preprocess the collected data, including data deduplication, missing value processing, noise filtering, etc., to ensure data quality.

[0075] Step 3: Data fusion: Use natural language processing (NLP) technology and machine learning algorithms to fuse data from different sources and in different formats to generate a unified data view.

[0076] Step 4: Data analysis: Use data mining and analysis techniques to conduct in-depth analysis of the integrated data to extract valuable information and knowledge.

[0077] As a further preferred embodiment of the present invention, the method further comprises:

[0078] Step 5. Result display: Display the analysis results through visualization tools to provide intuitive decision support.

[0079] During the data collection process of step 1, the system automatically collects data from multiple different types of data sources, including but not limited to:

[0080] 101. City official website: Use web crawler technology to regularly crawl policies, announcements, data and other information released on the city’s official website. The crawler uses a distributed crawler architecture to ensure efficient data collection through task distribution and scheduling systems.

[0081] 102. Social Media: Use API interfaces to obtain real-time data on user content, comments, interactions, etc. on social media. Use the OAuth2.0 authentication mechanism to ensure the legitimacy and security of data collection.

[0082] 103. News website: Collect the latest news reports and articles through RSS subscription or web crawler technology. Use full-text indexing technology to improve the efficiency of news data collection.

[0083] 104. Enterprise data: Obtain relevant business data from the enterprise's internal database or API interface. Ensure the efficiency of data collection through database connection pool and parallel query technology.

[0084] The data collection frequency and strategy formula used is:

[0085]

[0086] Among them, F d Indicates the frequency of data collection, N d It indicates the number of data requests per unit time, and T indicates the time period.

[0087] In the data cleaning process of step 2, the collected data is first preprocessed, including the following steps:

[0088] 201. Data deduplication: Use hash functions (such as MD5, SHA-256) to generate unique identifiers for data records, identify and delete duplicate data records.

[0089] 202. Missing value processing: Use K-Nearest Neighbors Imputation or Expectation-Maximization Algorithm to process missing values.

[0090] 203. Noise filtering: Use regular expressions, stop words list and domain-specific dictionary to filter noise and irrelevant information in the data.

[0091] The data cleaning, deduplication and missing value processing formula used is:

[0092] D c=D raw -D dup -D miss

[0093] Among them, D c represents the cleaned data set, D raw represents the original data set, D dup Indicates repeated data, D miss Indicates missing data.

[0094] In the data fusion process of step 3, natural language processing (NLP) technology and machine learning algorithms are used to fuse data from different sources and in different formats, including:

[0095] 301. Entity Recognition: Use NLP techniques (such as named entity recognition, NER) to identify entities in the data (such as names of people, places, and institutions). Use bidirectional long short-term memory networks (Bi-LSTM) and conditional random fields (CRF) models to ensure the accuracy of entity recognition.

[0096] 302. Relationship Extraction: Use machine learning algorithms (such as support vector machines, SVM) to extract the relationship between entities in the data. Use word embedding technology (such as Word2Vec) and dependency parsing to improve the effect of relationship extraction.

[0097] 303. Data alignment: Align the same entity from different data sources to generate a unified entity view. Use graph matching algorithms (such as Hungarian Algorithm) to ensure the accuracy of data alignment.

[0098] 304. Data integration: Integrate the aligned data into a unified data view for subsequent analysis. Improve the quality of data integration through data normalization and consistency check.

[0099] The entity recognition and alignment formula used in data fusion is:

[0100]

[0101] Among them, E f Represents the fused entity set, E i represents the entity set of the i-th data source, α i Represents the weight of the i-th data source.

[0102] The relation extraction formula used is:

[0103]

[0104] Among them, R f represents the fused relation set, R i represents the relationship set of the i-th data source, β i Represents the weight of the i-th data source.

[0105] The data integration formula used is:

[0106]

[0107] Among them, D f represents the integrated data set, D i represents the dataset of the ith data source, and γ represents the function of data normalization and consistency check.

[0108] In the data analysis process of step 4, data mining and analysis techniques are used to conduct in-depth analysis on the fused data, including:

[0109] 401. Classification and Clustering: Use machine learning algorithms (such as K-means clustering; support vector machine, SVM) to classify and cluster data and discover patterns and trends in the data.

[0110] 402. Association analysis: Use association rule mining technology (such as Apriori Algorithm) to analyze the association relationship in the data. Through frequent item set mining (Frequent Itemset Mining), find feature items with high association.

[0111] 403. Time Series Analysis: Analyze time series data to predict future trends and changes. Use the Autoregressive Integrated Moving Average (ARIMA) model and Long Short-Term Memory (LSTM) network to improve the accuracy of predictions.

[0112] 404. Sentiment Analysis: Conduct sentiment analysis on social media data to understand public sentiment and public opinion trends. Use sentiment lexicon and sentiment classification models (such as Bidirectional Encoder Representation Transformation, BERT) to ensure the accuracy of sentiment analysis.

[0113] The classification and clustering formulas used are:

[0114]

[0115] Among them, C represents the category, k represents the number of categories, and x i represents a data point, μ k represents the category center, and δ represents the distance metric function.

[0116] The time series analysis formula used is:

[0117] y t =φ1y t-1 +φ2y t-2 +…+φ p y t-p +θ1∈ t-1 +θ2∈ t-2 +…+θ q ∈ t-q

[0118] Among them, y t represents the time series value, φ i represents the autoregressive parameter, θ i represents the moving average parameter, ∈ t Represents white noise.

[0119] The sentiment analysis formula used is:

[0120]

[0121] Among them, S represents the sentiment score, w i Represents the weight of sentiment words, s i Indicates the sentiment intensity of sentiment words.

[0122] In the process of displaying the results of step 5, visualization tools are used to display the analysis results, including:

[0123] 501. Dashboard: Display key indicators and analysis results in real time, so you can quickly understand the overall situation. Through interactive dashboards (such as Tableau, Power BI), dynamic display and multi-dimensional analysis of data can be achieved.

[0124] 502. Charts and graphs: Use bar charts, line charts, pie charts and other charts to present data analysis results. Use chart libraries such as D3.js and ECharts to improve the aesthetics and interactivity of charts.

[0125] 503. Analysis Report: Automatically generate analysis reports, providing detailed analysis results and suggestions. Through natural language generation (NLG) technology, the report can be automatically written and updated.

[0126] The data visualization formula used in the result display is:

[0127] V=f(D analysis )

[0128] Among them, V represents the visualization result, D analysis represents the analyzed data, and f represents the visualization function.

[0129] The following two specific embodiments are used to illustrate the implementation process and effect of the method of the present invention.

[0130] Example 1: Collect data from the city's official website and news website, and perform data cleaning, fusion and analysis.

[0131] 1. Data collection:

[0132] -Use web crawler technology to regularly capture policies, announcements, data and other information released on the city’s official website, and use RSS subscriptions and web crawler technology to collect the latest reports and articles from news websites.

[0133] -Configure the crawler task scheduling system and set the data collection frequency F d and task allocation strategies to ensure efficient data collection.

[0134] 2. Data cleaning:

[0135] -Use hash functions to generate unique identifiers for data records, identify and delete duplicate data records.

[0136] - K nearest neighbor interpolation method is used to handle missing values ​​and ensure data integrity.

[0137] -Use regular expressions and domain dictionaries to filter out noise and irrelevant information in the data and improve data quality.

[0138] 3. Data Fusion:

[0139] - Use named entity recognition (NER) technology to identify entities in the data, and use Bi-LSTM and CRF models to ensure the accuracy of recognition.

[0140] -Extract the relationship between entities through the SVM algorithm, and use Word2Vec and dependency syntactic analysis technology to improve the relationship extraction effect.

[0141] -Use graph matching algorithms to align the same entity from different data sources and generate a unified entity view.

[0142] -Normalize the aligned data to ensure data consistency and integration quality.

[0143] 4. Data Analysis:

[0144] - Perform cluster analysis on data using K-means clustering algorithm to discover patterns and trends in the data.

[0145] -Use Apriori algorithm to mine association rules and analyze the association relationships in data.

[0146] -Analyze and forecast time series data through ARIMA model to provide predictions of future trends.

[0147] -Use sentiment lexicon and BERT model to conduct sentiment analysis on news data to understand public sentiment and public opinion trends.

[0148] 5. Results display:

[0149] -Use interactive dashboards to display key indicators and analysis results in real time, and realize dynamic display and multi-dimensional analysis of data.

[0150] -Use chart libraries such as D3.js and ECharts to present analysis results and provide intuitive charts and graphs.

[0151] -Automatically generate analysis reports through natural language generation technology, providing detailed analysis results and suggestions.

[0152] Example 2: Collect data from social media and enterprise databases, and perform data cleaning, fusion and analysis.

[0153] 1. Data collection:

[0154] -Use the API interface to obtain real-time data such as user content, comments, interactions, etc. on social media, and ensure the legitimacy and security of data collection through the OAuth2.0 authentication mechanism.

[0155] - Obtain relevant business data from the company's internal database or API interface, and use database connection pool and parallel query technology to ensure the efficiency of data collection.

[0156] 2. Data cleaning:

[0157] -Use hash functions to generate unique identifiers for data records, identify and delete duplicate data records.

[0158] -The expectation maximization algorithm is used to handle missing values ​​and ensure data integrity.

[0159] -Use regular expressions and stop word lists to filter out noise and irrelevant information in the data to improve data quality.

[0160] 3. Data Fusion:

[0161] -Use NER technology to identify entities in the data, and adopt Bi-LSTM and CRF models to ensure the accuracy of recognition.

[0162] -Extract the relationship between entities through the SVM algorithm, and use Word2Vec and dependency syntactic analysis technology to improve the relationship extraction effect.

[0163] -Use graph matching algorithms to align the same entity from different data sources and generate a unified entity view.

[0164] -Normalize the aligned data to ensure data consistency and integration quality.

[0165] 4. Data Analysis:

[0166] -Use SVM algorithm to classify and analyze enterprise data to discover patterns and trends in the data.

[0167] -Use Apriori algorithm to mine association rules and analyze the association relationships in data.

[0168] -Analyze and predict time series data through LSTM model to provide predictions of future trends.

[0169] -Use sentiment lexicon and BERT model to conduct sentiment analysis on social media data to understand public sentiment and public opinion dynamics.

[0170] 5. Results display:

[0171] -Use interactive dashboards to display key indicators and analysis results in real time, and realize dynamic display and multi-dimensional analysis of data.

[0172] -Use chart libraries such as D3.js and ECharts to present analysis results and provide intuitive charts and graphs.

[0173] -Automatically generate analysis reports through natural language generation technology, providing detailed analysis results and suggestions.

[0174] The above is a detailed description of the implementation mode of the present invention in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation mode. Various changes can be made within the knowledge scope of ordinary technicians in the field without departing from the purpose of the present invention. The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical content disclosed above to an equivalent embodiment of equivalent changes without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principle of the present invention, still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A multi-source heterogeneous data fusion and analysis method, characterized in that: The method comprises: Step 1: Collect data from multiple different types of data sources; Step 2: Preprocess the collected data for data cleaning; Step 3: Use natural language processing technology and machine learning algorithms to integrate data and generate a unified data view; Step 4: Use data mining and analysis techniques to analyze the fused data and extract the required information.

2. The multi-source heterogeneous data fusion and analysis method according to claim 1, characterized in that: The method comprises step five: displaying the result obtained by the analysis in step four through a visualization tool.

3. A multi-source heterogeneous data fusion and analysis method as claimed in claim 1 or 2, characterized in that: In the step 1, the collected data includes structured data, semi-structured data and unstructured data.

4. A multi-source heterogeneous data fusion and analysis method as claimed in claim 1 or 2, characterized in that: In the step 2, data preprocessing includes: data deduplication, missing value processing and noise filtering.

5. The multi-source heterogeneous data fusion and analysis method according to claim 3, characterized in that: In the step 1, data is collected from different types of data sources, and the data sources include:

101. City official website: Use web crawler technology to regularly capture policies, announcements, and data information released on the city’s official website; 102. Social Media: Use API interfaces to obtain real-time data on user posts, comments, and interactions on social media; 103. News website: collect the latest news reports and articles through RSS subscription or web crawler technology; 104. Enterprise data: obtain relevant business data from the enterprise's internal database or API interface; The data collection frequency and strategy formula used is: Among them, F d Indicates the frequency of data collection, N d It indicates the number of data requests per unit time, and T indicates the time period.

6. A multi-source heterogeneous data fusion and analysis method as claimed in claim 4, characterized in that: In the step 2, data preprocessing specifically includes the following steps:

201. Data deduplication: Use hash functions to generate unique identifiers for data records, identify and delete duplicate data records; 202. Missing value processing: Use K nearest neighbor interpolation method or expectation maximization algorithm to process missing values; 203. Noise filtering: Use regular expressions, stop words, and domain dictionaries to filter out noise and irrelevant information in data; The formula used for data cleaning, deduplication and missing value processing is: D c =D raw -D dup -D miss Among them, D c represents the cleaned data set, D raw represents the original data set, D dup Indicates repeated data, D miss Indicates missing data.

7. A multi-source heterogeneous data fusion and analysis method as claimed in claim 1 or 2, characterized in that: The step three specifically includes:

301. Entity Recognition: Use NLP technology to identify entities in data, and use bidirectional long short-term memory networks and conditional random field models to ensure the accuracy of entity recognition; 302. Relationship Extraction: Use machine learning algorithms to extract the relationships between entities in data, and improve the effect of relationship extraction through word embedding technology and dependency syntax analysis; 303. Data alignment: Align the same entity from different data sources to generate a unified entity view; use graph matching algorithms to ensure the accuracy of data alignment; 304. Data integration: Integrate the aligned data into a unified data view and improve the quality of data integration through data normalization and consistency checking; The entity recognition and alignment formula used in data fusion is: Among them, E f Represents the fused entity set, E i represents the entity set of the i-th data source, α i represents the weight of the i-th data source; The relation extraction formula used is: Among them, R f represents the fused relation set, R i represents the relationship set of the i-th data source, β i represents the weight of the i-th data source; The data integration formula used is: Among them, D f represents the integrated data set, D i represents the dataset of the ith data source, and γ represents the function of data normalization and consistency check.

8. A multi-source heterogeneous data fusion and analysis method as claimed in claim 1 or 2, characterized in that: The step 4 specifically includes:

401. Classification and Clustering: Use machine learning algorithms to classify and cluster data to discover patterns and trends in the data; 402. Association analysis: Use association rule mining technology to analyze the association relationship in the data and find feature items with high association through frequent item set mining; 403. Time Series Analysis: Analyze time series data to predict future trends and changes, and use autoregressive integrated moving average models and long short-term memory networks to improve the accuracy of predictions; 404. Sentiment analysis: Conduct sentiment analysis on social media data to obtain public sentiment and public opinion trends; use sentiment dictionaries and sentiment classification models to ensure the accuracy of sentiment analysis; The classification and clustering formula used is: Among them, C represents the category, k represents the number of categories, and x i represents a data point, μ k represents the category center, δ represents the distance metric function; The time series analysis formula used is: y t =φ1y t-1 +φ2y t-2 +…+φ p y t-p +θ1∈ t-1 +θ2∈ t-2 +…+θ q ∈ t-q Among them, y t represents the time series value, φ i represents the autoregressive parameter, θ i represents the moving average parameter, ∈ t represents white noise; The sentiment analysis formula used is: Among them, S represents the sentiment score, w i Represents the weight of sentiment words, s i Indicates the sentiment intensity of sentiment words.

9. A multi-source heterogeneous data fusion and analysis method as claimed in claim 1 or 2, characterized in that: The analysis results are displayed through visualization tools as described in step 5, including:

501. Real-time display of key indicators and analysis results through interactive dashboards; 502. Use charts and graphs to present data analysis results, including; 503. Automatically write and update analytical reports through natural language generation technology; The data visualization formula used in the result display is: V=f(D analysis ) Among them, V represents the visualization result, D analysis represents the analyzed data, and f represents the visualization function.

10. A multi-source heterogeneous data fusion and analysis system, characterized by: The system uses any one of the multi-source heterogeneous data fusion and analysis methods in claims 1-9 above, and the system architecture of the system includes a data acquisition module, a data cleaning module, a data fusion module, a data analysis module and a data display module connected in sequence.

Images