Hyper-fusion panoramic visual analysis system and method based on multi-source data fusion

Abstract

The invention discloses a hyper-fusion panoramic visual analysis system and method based on multi-source data fusion, and the system comprises a data source access module, a data collection module, a data preprocessing module, a multi-data fusion module and a visual analysis module, the data source access module is provided with a plurality of data interfaces, and is used for connecting different data sources; the data acquisition module is used for acquiring data from the data source connected with the data source access module; the data preprocessing module is used for cleaning data and carrying out standardization processing on the data; the multi-data fusion module adopts a specific data fusion algorithm to fuse the collected multi-source data; and the visual analysis module carries out visual processing on the fused data. According to the invention, redundant and useless information in the data is removed through the data preprocessing module, and the remaining data is standardized, so that the data of a plurality of data sources can be conveniently integrated, and meanwhile, the data of the plurality of data sources can be conveniently fused.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a hyper-converged panoramic visual analysis system and method based on multi-source data fusion. Background Art

[0002] With the rapid development of information technology, the data generated by various data sources has exploded, including but not limited to sensor data, database data, network log data, image data, text data, etc. These data often have different formats, structures and semantics. It is difficult to fully and deeply understand the operating conditions and laws of complex systems by analyzing the data of a single data source in isolation. At present, although there are some data fusion and visualization methods, most of them cannot effectively integrate multi-source heterogeneous data to achieve all-round and in-depth panoramic visual analysis. In addition, there is a lack of accurate consideration of data quality and feature weights in the data fusion process, which limits the accuracy and practicality of the analysis results.

[0003] In the prior art, a Chinese patent application with publication number CN118485756A discloses a hyper-convergent visualization processing method and system based on three-dimensional digital twins, including: obtaining a view tag corresponding to the interactive event information according to the interactive operation, judging whether it is necessary to globally update the 3D twin scene view according to the view tag; when it is necessary to globally update the 3D twin scene view, updating the new view element model resources and the corresponding materials, labels and data according to the view tag, and rendering and displaying the element model according to the updated new view element model resources and the corresponding materials, labels and data; when it is necessary to locally update the 3D twin scene view, updating the associated element model resources and the corresponding materials, labels and data according to the local element tag, and rendering and displaying the associated element model according to the updated associated element model resources and the corresponding materials, labels and data. This solution can improve the efficiency of element expression, highlight the visual effect, and improve the efficiency of operation and maintenance work.

[0004] The hyper-converged visualization processing method and system provided by the above patent mainly improves the efficiency of element expression, highlights the visual effect, and improves the efficiency of operation and maintenance work. In practical applications, the visualization data processing system often needs to collect data from multiple data sources, and the data from multiple data sources will have different problems such as format, structure and semantics, which makes data processing more troublesome. However, the existing visualization data processing system does not have the function of standardizing data, nor can it perform fusion processing on data from multiple data sources. Summary of the invention

[0005] The purpose of the present invention is to provide a hyper-converged panoramic visual analysis system and method based on multi-source data fusion, aiming to improve the problem that the existing visual data processing system does not have the function of standardizing data and cannot fuse data from multiple data sources.

[0006] The present invention is achieved in that:

[0007] In order to achieve the above-mentioned object, according to one aspect of the present invention, the present invention provides a hyper-converged panoramic visual analysis system based on multi-source data fusion, including a data source access module, having a plurality of data interfaces for connecting different data sources;

[0008] A data collection module, used to collect data from a data source connected to the data source access module;

[0009] Data preprocessing module, used to clean and standardize data;

[0010] The multi-data fusion module uses a specific data fusion algorithm to fuse the collected multi-source data;

[0011] The visualization analysis module visualizes the fused data.

[0012] Preferably, the data preprocessing module includes a data cleaning unit and a standardization processing unit. The data cleaning unit performs data cleaning operations according to the data characteristics of different data sources to remove noise data, duplicate data and abnormal values; the standardization processing unit is used to standardize the data so that the data from different data sources are comparable in numerical range and dimension, and adopts the following standardization formula:

[0013]

[0014] Among them, x is the original data value, μ is the mean of the data column, σ is the standard deviation, and x new is the standardized data value.

[0015] Preferably, the multi-data fusion module includes a feature extraction unit, a similarity calculation unit, a weight allocation unit and a data fusion processing unit; the feature extraction unit is used to extract features from the preprocessed data of each data source and convert the high-dimensional data into a low-dimensional feature vector representation; the similarity calculation unit is used to calculate the similarity of the extracted feature vectors; the weight allocation unit is used to allocate fusion weights to each data source according to the reliability, relevance and importance of the data; the data fusion processing unit is used to perform fusion processing on the feature vectors of each data source.

[0016] Preferably, the feature extraction unit converts high-dimensional data into low-dimensional feature vector representation; for text data, a bag-of-words model or F-ID algorithm is used to extract features; for image data, a convolutional neural network is used to extract feature maps of the image; for structured data, key features are selected according to data fields and business logic; let the feature vector extracted from the i-th data source be F i =[f i1 ,f i2 ,…,f im ], where m is the feature dimension; the similarity calculation unit uses the cosine similarity method to calculate the similarity between the feature vectors of each data source, which is used to evaluate the correlation between the data; the two feature vectors F i and F j The cosine similarity calculation formula is:

[0017]

[0018] Where · represents the vector dot product, |F i | and |F j |respectively vector F i and F j The model, sim(F i ,F j ) is F i and F j The cosine similarity of .

[0019] Preferably, when the weight distribution unit evaluates the data, the reliability is evaluated based on the historical accuracy and stability of the data source, the relevance is evaluated based on the calculation results of the similarity calculation module, and the importance is evaluated based on the business domain knowledge; let the weight of the i-th data source be w i , calculated by the following formula:

[0020]

[0021] Among them, α, β, and γ are adjustable weight coefficients, satisfying α+β+γ=1, n is the number of data sources, and Credibility i is the data reliability index of data source i, sim(F i ,F j ) is the data source feature vector F i and F j The cosine similarity of is used to represent the correlation index of data; Importance i is the data importance index of data source i;

[0022] The data fusion processing unit performs weighted fusion on the feature vectors of each data source based on the calculated weights, and obtains the fused feature vector F as follows:

[0023] F=[f1,f2,…,f m ];

[0024] in,

[0025] Preferably, the visualization analysis module includes a visualization mapping unit and an interaction design unit, wherein the visualization mapping unit is used to select a suitable visualization method according to the characteristics of the fused data and the analysis requirements of the user; and the interaction design unit is used to provide rich interactive functions, allowing the user to zoom, translate, filter, drill and perform associated query operations on the visualization results.

[0026] Preferably, it also includes a knowledge discovery module, a decision support module and a user management interaction module, wherein the knowledge discovery module divides data objects with similar characteristics into different clusters through a clustering algorithm so that users can identify different categories and patterns in the data; and uses an association rule mining algorithm to find frequent associations between data items; the decision support module provides users with decision suggestions and predictive analysis based on the discovered knowledge and visual analysis results; the user management interaction module includes a user management unit and a user interaction unit, wherein the user management unit is responsible for user registration, login, authority management and other functions to ensure that different users access and operate corresponding data and functional modules according to their authority; the user interaction unit is used to provide a friendly and intuitive graphical user interface, through which users can conveniently perform data query, analysis task setting, visualization parameter adjustment, view analysis results and decision suggestions.

[0027] According to a second aspect of the present invention, a hyper-fusion panoramic visual analysis method based on multi-source data fusion is provided, and the specific steps of the analysis method are as follows:

[0028] S100, connecting a data collection module to multiple data sources through a data source access module, and the data collection module collects data from each data source through the data source access module;

[0029] S200, the collected data is transmitted to a data preprocessing module, and the data preprocessing module cleans the collected data and performs standardization processing on the data;

[0030] S300, the processed data is transmitted to a multi-data fusion module, the multi-data fusion module processes the data and enters into weight allocation, and then fuses the multi-data according to the allocated weights;

[0031] S400: The fused data is transmitted to the visualization analysis module, and the visualization analysis module selects a suitable visualization method for panoramic display according to the characteristics of the fused data and the analysis requirements of the user.

[0032] Preferably, in step S300, the specific steps of multi-data fusion are as follows:

[0033] S310, converting high-dimensional data into low-dimensional feature vector representation through a feature extraction unit; assuming that the feature vector extracted from the i-th data source is F i =[f i1 ,f i2 ,…,f im ], where m is the feature dimension;

[0034] S320, using the cosine similarity calculation method to calculate the similarity of the collected feature vectors; two feature vectors F i and F j The cosine similarity calculation formula is:

[0035]

[0036] Where · represents the vector dot product, |F i | and |F j |respectively vector F i and F j The model, sim(F i ,F j ) is F i and F j The cosine similarity of

[0037] S330, after the similarity calculation of the feature vectors is completed, the relevance of the data can be determined, and then the weight allocation unit evaluates the reliability of the data through the historical accuracy and stability of the data source, and the importance is evaluated based on the business domain knowledge; then the indicators of the reliability, relevance and importance of the data are substituted into the weight calculation formula of the data source for calculation; let the weight of the i-th data source be w i , calculated by the following formula:

[0038]

[0039] Among them, α, β, and γ are adjustable weight coefficients, satisfying α+β+γ=1, n is the number of data sources, and Credibility i is the data reliability index of data source i, sim(F i ,F j ) is the data source feature vector F i and F j The cosine similarity of is used to represent the correlation index of data; Importance i is the data importance index of data source i;

[0040] S340, the data fusion processing unit performs weighted fusion on the feature vectors of each data source according to the calculated weights, and obtains the fused feature vector F as follows:

[0041] F=[f1,f2,…,f m ];

[0042] in,

[0043] Preferably, in step S400, after the visualization analysis module selects a suitable visualization method for panoramic display of the data, the visualization analysis system is provided with a knowledge discovery module, a decision support module and a user management interaction module. The knowledge discovery module clustering algorithm divides data objects with similar characteristics into different clusters so that users can identify different categories and patterns in the data; the association rule mining algorithm is used to find out the frequent associations between data items; the decision support module provides users with decision suggestions and predictive analysis based on the discovered knowledge and visualization analysis results; finally, the user management interaction module is responsible for user registration, login, authority management and other functions, ensuring that different users access and operate corresponding data and functional modules according to their authority, and providing a friendly and intuitive graphical user interface through which users can conveniently query data, set analysis tasks, adjust visualization parameters, view analysis results and decision suggestions.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention removes redundant and useless information from the data through a data preprocessing module, and standardizes the remaining data, which can facilitate the integration of data from multiple data sources, and also facilitates the fusion processing of data from multiple data sources; and fuses the data through a multi-data fusion module, which greatly facilitates the centralized processing of data from multiple data sources.

[0046] 2. The present invention uses data mining and machine learning algorithms, such as cluster analysis, classification algorithms, association rule mining, etc., through the knowledge discovery module to conduct in-depth analysis of the fused data and discover potential knowledge patterns and rules.

[0047] 3. The present invention is responsible for interacting with users through the user management interaction module, receiving user query requests, analysis task settings, visualization parameter adjustment and other operations, and presenting the results to users. Users can interact with the system through a graphical interface to conduct panoramic visual analysis conveniently and quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a structural block diagram of the present invention as a whole;

[0049] Figure 2 It is a structural block diagram of the data preprocessing module of the present invention;

[0050] Figure 3 It is a structural block diagram of the multi-data fusion module of the present invention;

[0051] Figure 4 It is a structural block diagram of the visual analysis module of the present invention;

[0052] Figure 5 It is a structural block diagram of the user management interaction module of the present invention;

[0053] Figure 6 It is a flowchart of the analysis method of the present invention. DETAILED DESCRIPTION

[0054] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0055] The following is a further description with reference to the accompanying drawings and specific embodiments:

[0056] Example 1

[0057] like Figure 1 As shown, a hyper-converged panoramic visual analysis system based on multi-source data fusion includes a data source access module, a data acquisition module, a data preprocessing module, a multi-data fusion module and a visual analysis module. The data source access module has multiple data interfaces for connecting different data sources; including but not limited to relational databases, non-relational databases (such as NoSQL databases), file systems (such as CSV, XML, JSON format files), network interfaces (for collecting real-time data, such as network traffic data), sensor interfaces (such as temperature, humidity, pressure and other physical quantity sensors), etc. The data acquisition module is used to collect data from the data source connected to the data source access module; the data preprocessing module is used to clean up the data and perform standardization processing on the data; the multi-data fusion module uses a specific data fusion algorithm to fuse the collected multi-source data; and the visual analysis module visualizes the fused data.

[0058] like Figure 2As shown, the data preprocessing module includes a data cleaning unit and a standardization processing unit. The data cleaning unit performs data cleaning operations according to the data characteristics of different data sources to remove noise data, duplicate data and outliers. For example, for sensor data, an outlier detection method based on a statistical model is used. If the deviation between the sensor reading at a certain moment and the mean value of historical data exceeds a certain threshold, it is determined to be an outlier and corrected or removed. The standardization processing unit is used to standardize the data so that the data from different data sources are comparable in numerical range and dimension. The following standardization formula is used:

[0059]

[0060] Among them, x is the original data value, μ is the mean of the data column, σ is the standard deviation, and x new is the standardized data value.

[0061] like Figure 3 As shown in the figure, the multi-data fusion module includes a feature extraction unit, a similarity calculation unit, a weight assignment unit and a data fusion processing unit; the feature extraction unit is used to extract features from the preprocessed data sources and convert high-dimensional data into low-dimensional feature vector representation; the similarity calculation unit is used to calculate the similarity of the extracted feature vectors; the weight assignment unit is used to assign fusion weights to each data source according to the reliability, relevance and importance of the data; the data fusion processing unit is used to perform fusion processing on the feature vectors of each data source. The feature extraction unit converts high-dimensional data into low-dimensional feature vector representation; for text data, the bag-of-words model or F-ID algorithm is used to extract features; for image data, a convolutional neural network is used to extract feature maps of the image; for structured data, key features are selected according to data fields and business logic; let the feature vector extracted from the i-th data source be F i =[f i1 ,f i2 ,…,f im ], where m is the feature dimension; the similarity calculation unit uses the cosine similarity method to calculate the similarity between the feature vectors of each data source to evaluate the correlation between the data; the two feature vectors F i and F j The cosine similarity calculation formula is:

[0062]

[0063] Where · represents the vector dot product, |F i | and |F j |respectively vector F i and F j The model, sim(F i ,F j ) is Fi and F j When the weight allocation unit evaluates the data, the reliability is evaluated based on the historical accuracy and stability of the data source, the relevance is evaluated based on the calculation results of the similarity calculation module, and the importance is evaluated based on the business domain knowledge; let the weight of the i-th data source be w i , calculated by the following formula:

[0064]

[0065] Among them, α, β, and γ are adjustable weight coefficients, satisfying α+β+γ=1, n is the number of data sources, and Credibility i is the data reliability index of data source i, sim(F i ,F j ) is the data source feature vector F i and F j The cosine similarity of is used to represent the correlation index of data; Importance i is the data importance index of data source i; the data fusion processing unit performs weighted fusion on the feature vectors of each data source based on the calculated weights, and obtains the fused feature vector F as follows:

[0066] F=[f1,f2,…,f m ];

[0067] in,

[0068] like Figure 4 As shown in the figure, the visualization analysis module includes a visualization mapping unit and an interaction design unit. The visualization mapping unit is used to select a suitable visualization method according to the characteristics of the fused data and the analysis needs of the user; such as bar charts, line charts, scatter charts, pie charts, maps, heat maps, network diagrams, etc., to map data features to visualization elements to achieve intuitive display of data. For example, for data related to geographic space, map visualization is used to mark data points on the map, and the attribute values ​​of the data are represented by visual variables such as color and size. The interaction design unit is used to provide rich interactive functions, allowing users to zoom, pan, filter, drill and associate queries on the visualization results. For example, the user can click on a column in the bar chart with the mouse, and the system will automatically pop up detailed data information and related associated data charts, which is convenient for users to deeply analyze the laws and relationships behind the data.

[0069] Example 2

[0070] like Figure 1As shown, a hyper-converged panoramic visual analysis system based on multi-source data fusion includes a data source access module, a data acquisition module, a data preprocessing module, a multi-data fusion module and a visual analysis module. The data source access module has multiple data interfaces for connecting different data sources; including but not limited to relational databases, non-relational databases (such as NoSQL databases), file systems (such as CSV, XML, JSON format files), network interfaces (for collecting real-time data, such as network traffic data), sensor interfaces (such as temperature, humidity, pressure and other physical quantity sensors), etc. The data acquisition module is used to collect data from the data source connected to the data source access module; the data preprocessing module is used to clean up the data and perform standardization processing on the data; the multi-data fusion module uses a specific data fusion algorithm to fuse the collected multi-source data; and the visual analysis module visualizes the fused data.

[0071] like Figure 2 As shown, the data preprocessing module includes a data cleaning unit and a standardization processing unit. The data cleaning unit performs data cleaning operations according to the data characteristics of different data sources to remove noise data, duplicate data and outliers. For example, for sensor data, an outlier detection method based on a statistical model is used. If the deviation between the sensor reading at a certain moment and the mean value of historical data exceeds a certain threshold, it is determined to be an outlier and corrected or removed. The standardization processing unit is used to standardize the data so that the data from different data sources are comparable in numerical range and dimension. The following standardization formula is used:

[0072]

[0073] Among them, x is the original data value, μ is the mean of the data column, σ is the standard deviation, and x new is the standardized data value.

[0074] like Figure 3 As shown in the figure, the multi-data fusion module includes a feature extraction unit, a similarity calculation unit, a weight assignment unit and a data fusion processing unit; the feature extraction unit is used to extract features from the preprocessed data sources and convert high-dimensional data into low-dimensional feature vector representation; the similarity calculation unit is used to calculate the similarity of the extracted feature vectors; the weight assignment unit is used to assign fusion weights to each data source according to the reliability, relevance and importance of the data; the data fusion processing unit is used to perform fusion processing on the feature vectors of each data source. The feature extraction unit converts high-dimensional data into low-dimensional feature vector representation; for text data, the bag-of-words model or F-ID algorithm is used to extract features; for image data, a convolutional neural network is used to extract feature maps of the image; for structured data, key features are selected according to data fields and business logic; let the feature vector extracted from the i-th data source be Fi =[f i1 ,f i2 ,…,f im ], where m is the feature dimension; the similarity calculation unit uses the cosine similarity method to calculate the similarity between the feature vectors of each data source to evaluate the correlation between the data; the two feature vectors F i and F j The cosine similarity calculation formula is:

[0075]

[0076] Where · represents the vector dot product, |F i | and |F j |respectively vector F i and F j The model, sim(F i ,F j ) is F i and F j When the weight allocation unit evaluates the data, the reliability is evaluated based on the historical accuracy and stability of the data source, the relevance is evaluated based on the calculation results of the similarity calculation module, and the importance is evaluated based on the business domain knowledge; let the weight of the i-th data source be w i , calculated by the following formula:

[0077]

[0078] Among them, α, β, and γ are adjustable weight coefficients, satisfying α+β+γ=1, n is the number of data sources, and Credibility i is the data reliability index of data source i, sim(F i ,F j ) is the data source feature vector F i and F j The cosine similarity of is used to represent the correlation index of data; Importance i is the data importance index of data source i; the data fusion processing unit performs weighted fusion on the feature vectors of each data source based on the calculated weights, and obtains the fused feature vector F as follows:

[0079] F=[f1,f2,…,f m ];

[0080] in,

[0081] like Figure 4As shown in the figure, the visualization analysis module includes a visualization mapping unit and an interaction design unit. The visualization mapping unit is used to select a suitable visualization method according to the characteristics of the fused data and the analysis needs of the user; such as bar charts, line charts, scatter charts, pie charts, maps, heat maps, network diagrams, etc., to map data features to visualization elements to achieve intuitive display of data. For example, for data related to geographic space, map visualization is used to mark data points on the map, and the attribute values ​​of the data are represented by visual variables such as color and size. The interaction design unit is used to provide rich interactive functions, allowing users to zoom, pan, filter, drill and associate queries on the visualization results. For example, the user can click on a column in the bar chart with the mouse, and the system will automatically pop up detailed data information and related associated data charts, which is convenient for users to deeply analyze the laws and relationships behind the data.

[0082] like Figure 1 and Figure 5 As shown, it also includes a knowledge discovery module, a decision support module and a user management interaction module. The knowledge discovery module divides data objects with similar characteristics into different clusters through a clustering algorithm so that users can identify different categories and patterns in the data; the association rule mining algorithm is used to find out the frequent associations between data items; the decision support module provides users with decision suggestions and predictive analysis based on the discovered knowledge and visual analysis results. For example, in the financial risk assessment scenario, based on the fusion analysis of historical data and current market data, the volatility trend of the financial market is predicted, and corresponding investment suggestions are given, such as adjusting the asset allocation ratio, avoiding certain high-risk investment products, etc.; the user management interaction module includes a user management unit and a user interaction unit. The user management unit is responsible for user registration, login, authority management and other functions to ensure that different users access and operate corresponding data and functional modules according to their permissions; for example, administrator users have advanced permissions such as system configuration, data source management and user authority allocation; ordinary users can only perform operations such as data query, visual analysis and obtaining decision suggestions. The user interaction unit is used to provide a friendly and intuitive graphical user interface, through which users can easily query data, set analysis tasks, adjust visualization parameters, view analysis results and make decision recommendations; the interface design follows the principles of human-computer interaction, focuses on the convenience of operation and readability of information, and uses interactive elements such as menus, buttons, charts, and tables to enable users to quickly get started and efficiently complete data analysis tasks.

[0083] Example 3

[0084] like Figure 1 As shown, a hyper-converged panoramic visual analysis method based on multi-source data fusion is provided. The specific steps of the analysis method are as follows:

[0085] S100, connecting a data collection module to multiple data sources through a data source access module, and the data collection module collects data from each data source through the data source access module;

[0086] S200, the collected data is transmitted to a data preprocessing module, and the data preprocessing module cleans the collected data and performs standardization processing on the data;

[0087] S300, the processed data is transmitted to the multi-data fusion module, the multi-data fusion module processes the data and enters the weight allocation, and then fuses the multi-data according to the allocated weights; the specific steps of multi-data fusion are as follows:

[0088] S310, converting high-dimensional data into low-dimensional feature vector representation through a feature extraction unit; assuming that the feature vector extracted from the i-th data source is F i =[f i1 ,f i2 ,…,f im ], where m is the feature dimension;

[0089] S320, using the cosine similarity calculation method to calculate the similarity of the collected feature vectors; two feature vectors F i and F j The cosine similarity calculation formula is:

[0090]

[0091] Where · represents the vector dot product, |F i | and |F j |respectively vector F i and F j The model, sim(F i ,F j ) is F i and F j The cosine similarity of

[0092] S330, after the similarity calculation of the feature vectors is completed, the relevance of the data can be determined, and then the weight allocation unit evaluates the reliability of the data through the historical accuracy and stability of the data source, and the importance is evaluated based on the business domain knowledge; then the indicators of the reliability, relevance and importance of the data are substituted into the weight calculation formula of the data source for calculation; let the weight of the i-th data source be w i , calculated by the following formula:

[0093]

[0094] Among them, α, β, and γ are adjustable weight coefficients, satisfying α+β+γ=1, n is the number of data sources, and Credibilityi is the data reliability index of data source i, sim(F i ,F j ) is the data source feature vector F i and F j The cosine similarity of is used to represent the correlation index of data; Importance i is the data importance index of data source i;

[0095] S340, the data fusion processing unit performs weighted fusion on the feature vectors of each data source according to the calculated weights, and obtains the fused feature vector F as follows:

[0096] F=[f1,f2,…,f m ];

[0097] in,

[0098] S400, the fused data is transmitted to the visualization analysis module, and the visualization analysis module selects a suitable visualization method for panoramic display according to the fused data characteristics and the user's analysis needs. After the visualization analysis module selects a suitable visualization method for panoramic display of the data, the visualization analysis system is provided with a knowledge discovery module, a decision support module and a user management interaction module. The knowledge discovery module clustering algorithm divides data objects with similar characteristics into different clusters so that users can identify different categories and patterns in the data; the association rule mining algorithm is used to find the frequent association relationship between data items; the decision support module provides decision suggestions and predictive analysis for users based on the discovered knowledge and visualization analysis results; finally, the user management interaction module is responsible for user registration, login, authority management and other functions, ensuring that different users access and operate corresponding data and functional modules according to their authority, and at the same time provides a friendly and intuitive graphical user interface, through which users can easily query data, set analysis tasks, adjust visualization parameters, view analysis results and decision suggestions.

[0099] In summary, compared with the prior art, the present application removes redundant and useless information from the data through a data preprocessing module, and standardizes the remaining data, which can facilitate the integration of data from multiple data sources, and also facilitates the fusion processing of data from multiple data sources; and the data is fused through a multi-data fusion module, which greatly facilitates the centralized processing of data from multiple data sources. Through the knowledge discovery module, data mining and machine learning algorithms, such as cluster analysis, classification algorithms, association rule mining, etc., are used to deeply analyze the fused data to discover potential knowledge patterns and laws. The user management interaction module is responsible for interacting with users, receiving user query requests, analysis task settings, visualization parameter adjustments, and other operations, and presenting the results to users. Users can interact with the system through a graphical interface to quickly and easily perform panoramic visual analysis.

[0100] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A hyper-converged panoramic visual analysis system based on multi-source data fusion, characterized in that: It includes a data source access module with multiple data interfaces for connecting to different data sources; A data collection module, used to collect data from a data source connected to the data source access module; Data preprocessing module, used to clean and standardize data; The multi-data fusion module uses a specific data fusion algorithm to fuse the collected multi-source data; The visualization analysis module visualizes the fused data.

2. According to claim 1, a hyper-converged panoramic visual analysis system based on multi-source data fusion is characterized in that: The data preprocessing module includes a data cleaning unit and a standardization processing unit. The data cleaning unit performs data cleaning operations according to the data characteristics of different data sources to remove noise data, duplicate data and abnormal values; the standardization processing unit is used to standardize the data so that the data from different data sources are comparable in numerical range and dimension, using the following standardization formula: in, is the original data value, is the mean of the data column, is the standard deviation, is the standardized data value.

3. The hyper-fusion panoramic visual analysis system based on multi-source data fusion according to claim 1 is characterized in that: The multi-data fusion module includes a feature extraction unit, a similarity calculation unit, a weight allocation unit and a data fusion processing unit; the feature extraction unit is used to extract features from the pre-processed data of each data source and convert the high-dimensional data into a low-dimensional feature vector representation; the similarity calculation unit is used to calculate the similarity of the extracted feature vectors; the weight allocation unit is used to allocate a fusion weight to each data source according to the reliability, relevance and importance of the data; The data fusion processing unit is used to perform fusion processing on the feature vectors of various data sources.

4. The hyper-fusion panoramic visual analysis system based on multi-source data fusion according to claim 3 is characterized in that: The feature extraction unit converts high-dimensional data into low-dimensional feature vector representation; for text data, the bag-of-words model or F-ID algorithm is used to extract features; for image data, the convolutional neural network is used to extract the feature map of the image; for structured data, key features are selected according to data fields and business logic; From The feature vector extracted from the data source is ,in is the feature dimension; The similarity calculation unit uses the cosine similarity method to calculate the similarity between the feature vectors of each data source, so as to evaluate the correlation between the data; two feature vectors and The cosine similarity calculation formula is: where · represents the vector dot product, and The vectors are and The model, for and The cosine similarity of .

5. The hyper-fusion panoramic visual analysis system based on multi-source data fusion according to claim 4 is characterized in that: When the weight distribution unit evaluates the data, the reliability is evaluated based on the historical accuracy and stability of the data source, the relevance is evaluated based on the calculation results of the similarity calculation module, and the importance is evaluated based on the business domain knowledge; let the weight of the i-th data source be w i , calculated by the following formula: Among them, α, β, and γ are adjustable weight coefficients, satisfying α+β+γ=1, n is the number of data sources, and Credibility i is the data reliability index of data source i, sim(F i ,F j ) is the data source feature vector F i and F j The cosine similarity of is used to represent the correlation index of data; Importance i is the data importance index of data source i; The data fusion processing unit performs weighted fusion on the feature vectors of each data source based on the calculated weights, and obtains the fused feature vector F as follows: F=[f1,f2,…,f m ]; in, 6. The hyper-fusion panoramic visual analysis system based on multi-source data fusion according to claim 1 is characterized in that: The visualization analysis module includes a visualization mapping unit and an interaction design unit. The visualization mapping unit is used to select a suitable visualization method according to the characteristics of the fused data and the user's analysis requirements; the interaction design unit is used to provide rich interactive functions, allowing the user to zoom, translate, filter, drill and perform associated query operations on the visualization results.

7. A hyper-converged panoramic visual analysis system based on multi-source data fusion according to any one of claims 1 to 6, characterized in that: It also includes a knowledge discovery module, a decision support module and a user management interaction module. The knowledge discovery module divides data objects with similar characteristics into different clusters through a clustering algorithm so that users can identify different categories and patterns in the data; Use association rule mining algorithms to find frequent associations between data items; The decision support module provides users with decision suggestions and predictive analysis based on the discovered knowledge and visual analysis results; the user management interaction module includes a user management unit and a user interaction unit. The user management unit is responsible for user registration, login, authority management and other functions to ensure that different users access and operate corresponding data and functional modules according to their permissions; the user interaction unit is used to provide a friendly and intuitive graphical user interface, through which users can easily query data, set analysis tasks, adjust visual parameters, view analysis results and decision suggestions.

8. A hyper-converged panoramic visual analysis method based on multi-source data fusion, using the hyper-converged panoramic visual analysis system based on multi-source data fusion according to any one of claims 1 to 7, characterized in that: The specific steps of this analysis method are as follows: S100, connecting a data collection module to multiple data sources through a data source access module, and the data collection module collects data from each data source through the data source access module; S200, the collected data is transmitted to a data preprocessing module, and the data preprocessing module cleans the collected data and performs standardization processing on the data; S300, the processed data is transmitted to a multi-data fusion module, the multi-data fusion module processes the data and enters into weight allocation, and then fuses the multi-data according to the allocated weights; S400: The fused data is transmitted to the visualization analysis module, and the visualization analysis module selects a suitable visualization method for panoramic display according to the characteristics of the fused data and the analysis requirements of the user.

9. The method for hyper-fusion panoramic visual analysis based on multi-source data fusion according to claim 8, characterized in that: In step S300, the specific steps of fusion of most data are as follows: S310, converting high-dimensional data into low-dimensional feature vector representation through a feature extraction unit; assuming that the feature vector extracted from the i-th data source is F i =[f i1 ,f i2 ,…,f im ], where m is the feature dimension; S320, using the cosine similarity calculation method to calculate the similarity of the collected feature vectors; two feature vectors F i and F j The cosine similarity calculation formula is: Where · represents the vector dot product, |F i | and |F j |respectively vector F i and F j The model, sim(F i ,F j ) is F i and F j The cosine similarity of S330, after the similarity calculation of the feature vectors is completed, the relevance of the data can be determined, and then the weight allocation unit evaluates the reliability of the data through the historical accuracy and stability of the data source, and the importance is evaluated based on the business domain knowledge; then the indicators of the reliability, relevance and importance of the data are substituted into the weight calculation formula of the data source for calculation; let the weight of the i-th data source be w i , calculated by the following formula: Among them, α, β, and γ are adjustable weight coefficients, satisfying α+β+γ=1, n is the number of data sources, and Credibility i is the data reliability index of data source i, sim(F i ,F j ) is the data source feature vector F i and F j The cosine similarity of is used to represent the correlation index of data; Importance i is the data importance index of data source i; S340, the data fusion processing unit performs weighted fusion on the feature vectors of each data source according to the calculated weights, and obtains the fused feature vector F as follows: F=[f1,f2,…,f m ]; in, 10. The method for hyper-converged panoramic visual analysis based on multi-source data fusion according to claim 8, characterized in that: In step S400, after the visualization analysis module selects a suitable visualization method for the data to display the whole picture, the visualization analysis system is provided with a knowledge discovery module, a decision support module and a user management interaction module, and the data objects with similar characteristics are divided into different clusters through the clustering algorithm of the knowledge discovery module, so that the user can identify different categories and patterns in the data; Use association rule mining algorithms to find frequent associations between data items; The decision support module provides users with decision suggestions and predictive analysis based on the discovered knowledge and visual analysis results; finally, the user management interaction module is responsible for user registration, login, permission management and other functions, ensuring that different users access and operate corresponding data and functional modules according to their permissions, and providing a friendly and intuitive graphical user interface through which users can easily query data, set analysis tasks, adjust visual parameters, view analysis results and decision suggestions.

Images