Internet multidimensional data fusion analysis system based on artificial intelligence

Abstract

The invention is applied to the technical field of data analysis, and particularly discloses an Internet multi-dimensional data fusion analysis system based on artificial intelligence, which comprises a central processing unit and a plurality of end processing units, and the central processing unit is connected with the end processing units at the same time. According to the Internet multidimensional data fusion analysis system based on artificial intelligence, the system is provided with the central processing unit and the plurality of end processing units connected with the central processing unit for cooperative work, and the end processing units collect multidimensional data from different data sources by adopting a plurality of modes such as web crawlers, application program interfaces and Internet of Things equipment at the same time; the data preprocessing module in each end processing unit processes the original data set in parallel and performs operations such as data cleaning, abnormal data marking and data conversion on the original data, and the preprocessed data is input into the central processing unit for subsequent processing, so that the data processing pressure can be dispersed, and compared with a single processor processing mode, the data processing efficiency is improved. And the data processing speed is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to an Internet multi-dimensional data fusion analysis system based on artificial intelligence. Background Art

[0002] With the rapid development of Internet technology, the amount of data obtained by the Internet is large, and the sources of data are wide and diverse, covering multiple dimensions such as text, images, audio, and video. In order to obtain the information contained in Internet data, data analysis systems are usually used to extract information from Internet data, mine valuable information, and provide support for decision-making in various industries.

[0003] However, traditional data analysis systems usually use a single processor to analyze and process data. However, Internet data is massive and growing rapidly. Using a single processor to process data is inefficient and cannot adapt to the needs of large-scale data processing. In addition, since Internet data contains information in multiple dimensions, traditional data analysis systems lack the ability to mine data relationships, making it difficult to discover the deep connections hidden between multidimensional data and unable to fully tap the potential value of multidimensional data, thus affecting the efficiency of multidimensional data analysis and the reliability of analysis results. Summary of the Invention

[0004] The purpose of the present invention is to provide an Internet multidimensional data fusion analysis system based on artificial intelligence to solve the problems raised in the above background technology, such as low processing efficiency of a single processor and insufficient correlation analysis of multidimensional data.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an Internet multi-dimensional data fusion and analysis system based on artificial intelligence, comprising a central processing unit and a terminal processing unit, wherein the central processing unit is connected to a plurality of terminal processing units at the same time, and the central processing unit and the plurality of terminal processing units constitute a structure with the central processing unit as the center and a plurality of terminal processing units distributedly arranged, the central processing unit is bidirectionally connected to a data storage module, and the data storage module is bidirectionally connected to each terminal processing unit at the same time, the terminal processing unit comprises a terminal processing device, a data acquisition module and a data preprocessing module, the terminal processing device has a built-in data acquisition module connected, the terminal processing device has a built-in data preprocessing module, the central processing unit comprises a central processing device, a data fusion module, a data analysis module and a data visualization module, the central processing device has a built-in data fusion module, the central processing device has a built-in data analysis module, the central processing device has a built-in data visualization module, the data fusion module is connected to the data analysis module, the data analysis module is connected to the data visualization module, and the central processing device is bidirectionally communicated with a plurality of terminal processing devices at the same time; Preferably, the data acquisition module collects multidimensional data from multiple data sources on the Internet. The data acquisition module collects data including web crawlers, application programming interfaces (APIs) and Internet of Things (IoT) devices. The web crawler acquires web page information data, the application programming interface connects to databases and third-party platforms to acquire data, and the IoT devices collect data. The data collected by the data acquisition module is marked as raw data, and a data set consisting of multiple raw data is marked as a raw data group.

[0006] By adopting the above technical solution, the original data can be collected from the Internet using the data collection module.

[0007] Preferably, the data preprocessing module receives the original data group from the data acquisition module for data preprocessing, and the data preprocessing includes data cleaning, abnormal data marking and data conversion. The data cleaning includes noise data removal, duplicate data removal and error data removal. The abnormal data marking identifies and processes abnormal data points through an outlier detection algorithm. The data conversion normalizes the original data group and converts it into a standard format that can be processed by the system. The data of the original data group after preprocessing is marked as a preprocessed data group.

[0008] By adopting the above technical solution, the raw data can be preprocessed using the data preprocessing module to improve the efficiency of subsequent data analysis.

[0009] Preferably, the data fusion module receives the pre-processed data group in each terminal processing unit, and the data fusion module uses a data association algorithm to perform data fusion on the pre-processed data group. The data obtained after the data fusion processing of the pre-processed data group is marked as a fused data group. The fusion process of the pre-processed data group by the data association algorithm is as follows: Feature extraction process: extract data features from the data of the preprocessing data group, remove redundant features, and select the most representative data features; Data matching process: Based on the extracted data features, similarity algorithms are used to calculate the similarity between data. By analyzing the correlation between data features, the correlation relationship between data is mined. Based on the similarity calculation results and the correlation mining structure, similar or related data in different data are matched and associated to generate matching data between the data. A threshold is set for the matching data, and the data that reaches the threshold is considered as associated data; Data fusion process: Data fusion is performed based on the matching data of the data. The neural network model is used to fuse the data whose matching data reaches the threshold. The fused data is marked as a fused data group.

[0010] By adopting the above technical solution and utilizing the data fusion module, the fusion of multi-dimensional data can be achieved, providing a basis for subsequent data analysis.

[0011] Preferably, the data analysis module receives the fused data group processed by the data fusion module and analyzes the fused data group. The data analysis module uses a deep learning model to process the fused data group, mines the deep-level features of the fused data group, and obtains the trends and rules of the data.

[0012] By adopting the above technical solution, the fused data can be analyzed using the data analysis module to obtain the patterns of the data.

[0013] Preferably, the data visualization module receives the data analysis results from the data analysis module. The data visualization module has a built-in data visualization component to display the data analysis results in a visual manner. The data visualization includes statistical chart display and interactive functions. The statistical chart display displays the data in the form of statistical charts, and the interactive function enables users to interactively explore the statistical charts.

[0014] By adopting the above technical solution, the analyzed data can be visualized using the data visualization module, which is convenient for users to observe.

[0015] Preferably, the data storage module stores the data in the central processing unit and the terminal processing unit in real time respectively. The data storage module has a built-in data compression algorithm, which compresses the stored data to reduce the data storage space. The data is stored in partitions within the data storage module.

[0016] By adopting the above technical solution, data can be stored in real time using the data storage module, making it easier to trace the data.

[0017] Compared with the prior art, the present invention has the following beneficial effects: the Internet multi-dimensional data fusion analysis system based on artificial intelligence: 1. The system of the present invention employs a central processing unit (CPU) and multiple connected terminal processing units (DPUs) to work together. The DPUs simultaneously collect multidimensional data from various data sources using various methods, such as web crawlers, application programming interfaces (APIs), and IoT devices. The data preprocessing modules within each DPU process the raw data sets in parallel, performing operations such as data cleaning, abnormal data marking, and data conversion. The preprocessed data is then fed into the CPU for subsequent processing, thereby distributing the data processing pressure and increasing the data processing speed compared to single-processor processing methods. 2. In the present invention, the data association algorithm set in the data fusion module is used to match the correlation between multidimensional data, providing a basis for subsequent data analysis. Based on the data correlation, the data analysis module uses a deep learning model to process the fused data group, explore deep-level features, and derive data trends and patterns, thereby improving the reliability of the data analysis results. The results of the data analysis are visualized through the data visualization module. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the structure of the system of the present invention; Figure 2 This is a structural diagram of the data processing flow of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] See also Figure 1-Figure 2 , the present invention provides a technical solution: an Internet multi-dimensional data fusion analysis system based on artificial intelligence.

[0021] The central processing unit is connected to multiple terminal processing units at the same time. The central processing unit and the multiple terminal processing units form a structure with the central processing unit as the center and multiple terminal processing units distributedly arranged. The central processing unit is bidirectionally connected to the data storage module. The data storage module is bidirectionally connected to each terminal processing unit at the same time. The terminal processing unit includes a terminal processing device, a data acquisition module and a data preprocessing module. The terminal processing device has a built-in data acquisition module connected. The terminal processing device has a built-in data preprocessing module. The central processing unit includes a central processing device, a data fusion module, a data analysis module and a data visualization module. The central processing device has a built-in data fusion module. The central processing device has a built-in data analysis module. The central processing device has a built-in data visualization module. The data fusion module is connected to the data analysis module. The data analysis module is connected to the data visualization module. The central processing device is bidirectionally communicated with multiple terminal processing devices at the same time. The data storage module stores the data in the central processing unit and the terminal processing unit in real time respectively. The data storage module has a built-in data compression algorithm. The data compression algorithm compresses the stored data to reduce data storage space. The data storage module partitions the data for storage. like Figure 2As shown, in the process of using this system to analyze Internet data, the data acquisition module in the terminal processing unit is used to connect and collect data, and at the same time, the data preprocessing module in the terminal processing device synchronously preprocesses the data. The preprocessed data is input from the terminal processing unit to the central processing unit, thereby reducing the data processing volume of the central processing unit and improving the data analysis efficiency. The preprocessed data is input into the data fusion module for fusion of related data, and the fused data is input into the data analysis module for analysis and processing. The analysis results are visualized in the data visualization module. In this process, the data storage module stores the data in real time to ensure the integrity of the data, and the data compression algorithm built into the data storage module compresses the data to reduce data storage space.

[0022] The data acquisition module collects multidimensional data from multiple data sources on the Internet. The data acquisition module collects data including web crawlers, application program interfaces and Internet of Things devices. The web crawlers acquire web page information data, the application program interfaces connect to databases and third-party platforms to acquire data, and the Internet of Things devices collect data. The data collected by the data acquisition module is marked as raw data, and the data set composed of multiple raw data is marked as a raw data group. The data preprocessing module receives the raw data group from the data acquisition module for data preprocessing. Data preprocessing includes data cleaning, abnormal data marking and data conversion. Data cleaning includes noise data removal, duplicate data removal and error data removal. Abnormal data marking identifies and processes abnormal data points through an outlier detection algorithm. Data conversion normalizes the raw data group and converts it into a standard format that can be processed by the system. The data after preprocessing of the raw data group is marked as a preprocessed data group. like Figure 1 As shown, a data acquisition module is built into the terminal processing device in the form of a program, so that the data acquisition module can stably run the web crawler program, normally connect with the application interface, and receive data from the Internet of Things device. The data acquisition module is used to collect data from the Internet. The web crawler program extracts text, pictures, links and other data from the web page according to pre-set rules, and the application interface connects with the database and the third-party platform to obtain the corresponding data. For the Internet of Things device, the data acquisition module obtains the collected data of the Internet of Things device through communication. These data are collected into the system as raw data. The raw data group is transmitted to the data preprocessing module. First, data cleaning is performed to remove noise data, duplicate data and error data, and an outlier detection algorithm is used to identify and mark abnormal data for subsequent further analysis and processing. The processed data is converted into a standard format that can be processed by the system for subsequent data analysis. The preprocessed data is marked as a preprocessed data group.

[0023] The data fusion module receives the pre-processed data group in each terminal processing unit, and the data fusion module uses the data association algorithm to perform data fusion on the pre-processed data group. The data obtained after the pre-processed data group is processed by the data fusion is marked as the fusion data group. The fusion process of the pre-processed data group by the data association algorithm is as follows: Feature extraction process: extract data features from the data of the pre-processed data group, remove redundant features, and select the most representative data features; Data matching process: Based on the extracted data features, use the similarity algorithm to calculate the similarity between the data, analyze the correlation between the data features, and mine the correlation relationship between the data according to the similarity. Calculation results and correlation mining structure are used to match and associate similar or related data in different data, generate matching data between the data, set a threshold for the matching data, and the data that reaches the threshold is the associated data; data fusion process: data fusion is performed based on the matching data of the data, and a neural network model is used to fuse the data whose matching data reaches the threshold. The fused data is marked as a fused data group. The data analysis module receives the fused data group processed by the data fusion module and analyzes the fused data group. The data analysis module uses a deep learning model to process the fused data group, mine the deep-level features of the fused data group, and obtain the trends and patterns of the data; like Figure 1 As shown, the data fusion module receives the pre-processed data group of each terminal processing unit, uses the feature extraction algorithm to extract key data features from the data, removes redundant features in the process, retains the most representative data features, reduces data dimensions, and improves data processing efficiency. Based on the extracted data features, a similarity algorithm is used to calculate the similarity between the data. At the same time, based on the correlation between the data features, the potential correlation between the data is mined, and finally the matching data between the data is calculated. Based on the value of the matching data, it is determined whether the data is related data, and a threshold is set for the matching data. For data whose matching data exceeds the threshold, the neural network model is used to fuse the data whose matching data reaches the threshold. The fused data group is marked as a fused data group, and the fused data group is input into the data analysis module. The deep learning model is used to process the fused data group, and the deep-level features of the fused data group are mined to obtain the trends and laws of the data.

[0024] The data visualization module receives the data analysis results from the data analysis module. The data visualization module has built-in data visualization components to display the data analysis results in a visual manner. Data visualization includes statistical chart display and interactive functions. The statistical chart display displays the data in the form of statistical charts, and the interactive function enables users to interactively explore the statistical charts. like Figure 1As shown, the data visualization module receives the results from the data analysis module and uses built-in visualization components to display the data in the form of statistical charts. The appropriate chart type is selected according to the characteristics of the data. For example, a bar chart is used to compare the size of data of different categories, a line chart is used to show the trend of data changes over time, and a pie chart is used to show the proportion of each part of the data in the whole. Users can use mouse clicks, zooming, filtering and other operations to view the data details in depth, thereby improving the user experience.

[0025] Working principle: The system collects raw data through the data acquisition module of the terminal processing unit, and preprocesses the raw data in the data preprocessing module of the terminal processing unit to reduce the amount of data processing in the subsequent central processing unit. After the preprocessed data is input into the central processing unit, data fusion is performed in the data fusion module. The fused data is input into the data analysis module for analysis and processing. The data analysis results are visualized by the data visualization module to facilitate users to observe the data analysis results.

[0026] While the embodiments of the present invention have been shown and described, it will be apparent to those skilled in the art that various changes, modifications, substitutions, and alterations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. An Internet multi-dimensional data fusion analysis system based on artificial intelligence, comprising a central processing unit and a terminal processing unit, characterized in that: The central processing unit is connected to multiple terminal processing units at the same time. The central processing unit and the multiple terminal processing units constitute a structure with the central processing unit as the center and multiple terminal processing units distributedly arranged. The central processing unit is bidirectionally connected to the data storage module. The data storage module is bidirectionally connected to each terminal processing unit at the same time. The terminal processing unit includes a terminal processing device, a data acquisition module and a data preprocessing module. The terminal processing device has a built-in data acquisition module connected. The terminal processing device has a built-in data preprocessing module. The central processing unit includes a central processing device, a data fusion module, a data analysis module and a data visualization module. The central processing device has a built-in data fusion module. The central processing device has a built-in data analysis module. The central processing device has a built-in data visualization module. The data fusion module is connected to the data analysis module. The data analysis module is connected to the data visualization module. The central processing device is bidirectionally communicated with multiple terminal processing devices at the same time.

2. The artificial intelligence-based Internet multidimensional data fusion analysis system according to claim 1, characterized in that: The data acquisition module collects multidimensional data from multiple data sources on the Internet. The data acquisition module collects data including web crawlers, application programming interfaces (APIs), and Internet of Things (IoT) devices. The web crawlers acquire web page information data, the application programming interfaces (APIs) connect to databases and third-party platforms to acquire data, and the IoT devices collect data. The data collected by the data acquisition module is marked as raw data, and a data set consisting of multiple raw data is marked as a raw data group.

3. The artificial intelligence-based Internet multi-dimensional data fusion analysis system according to claim 1, characterized in that: The data preprocessing module receives the original data group from the data acquisition module for data preprocessing. The data preprocessing includes data cleaning, abnormal data marking and data conversion. The data cleaning includes noise data removal, duplicate data removal and error data removal. The abnormal data marking identifies and processes abnormal data points through an outlier detection algorithm. The data conversion normalizes the original data group and converts it into a standard format that can be processed by the system. The data of the original data group after preprocessing is marked as a preprocessed data group.

4. The artificial intelligence-based Internet multi-dimensional data fusion analysis system according to claim 1, characterized in that: The data fusion module receives the pre-processed data group in each terminal processing unit, and performs data fusion on the pre-processed data group using a data association algorithm. The data obtained after the pre-processed data group is fused is marked as a fused data group. The fusion process of the pre-processed data group by the data association algorithm is as follows: Feature extraction process: extract data features from the data of the preprocessing data group, remove redundant features, and select the most representative data features; Data matching process: Based on the extracted data features, similarity algorithms are used to calculate the similarity between data. By analyzing the correlation between data features, the correlation relationship between data is mined. Based on the similarity calculation results and the correlation mining structure, similar or related data in different data are matched and associated to generate matching data between the data. A threshold is set for the matching data, and the data that reaches the threshold is considered as associated data; Data fusion process: Data fusion is performed based on the matching data of the data. The neural network model is used to fuse the data whose matching data reaches the threshold. The fused data is marked as a fused data group.

5. The artificial intelligence-based Internet multi-dimensional data fusion analysis system according to claim 1, characterized in that: The data analysis module receives the fused data group processed by the data fusion module and analyzes the fused data group. The data analysis module uses a deep learning model to process the fused data group, explores the deep-level features of the fused data group, and obtains the trends and rules of the data.

6. The artificial intelligence-based Internet multi-dimensional data fusion analysis system according to claim 1, characterized in that: The data visualization module receives the data analysis results from the data analysis module. The data visualization module has a built-in data visualization component to display the data analysis results in a visual manner. The data visualization includes statistical chart display and interactive functions. The statistical chart display displays the data in the form of statistical charts, and the interactive function enables users to interactively explore the statistical charts.

7. The artificial intelligence-based Internet multi-dimensional data fusion analysis system according to claim 1, characterized in that: The data storage module stores the data in the central processing unit and the terminal processing unit in real time respectively. The data storage module has a built-in data compression algorithm, which compresses the stored data to reduce the data storage space. The data is partitioned and stored in the data storage module.

Images