Intelligent visualization system for dynamic aggregation of big data

Abstract

An intelligent visualization system for big data dynamic aggregation comprises a self-adaptive dynamic weighted aggregation algorithm module, a deep learning-based anomaly detection module and the like, can automatically identify and connect various data sources, dynamically adjust weights according to historical variances of the data sources to perform aggregation calculation, and is used for anomaly detection based on deep learning when the variances are small and the weights are high. The multi-level visual framework has a basis layer, an analysis layer, a decision layer and an interactive design layer, a feedback learning mechanism can adjust the weight according to user scores and data source accuracy, the method has the advantages of automatically and dynamically adjusting the weight, supporting real-time aggregation analysis, being modularized and extensible and the like, the accuracy and reliability of the system are improved, the user experience is enhanced, and the user experience is improved. On the whole, the method has wide application value in the fields of finance, e-commerce, logistics and the like, and can better meet the requirements of users.

Description

Technical Field

[0001] The present invention relates to the transmission of digital information, and in particular to an intelligent visualization system for dynamic aggregation of big data. Background Art

[0002] In today's digital age, the sources of data are extremely broad and complex, and there is a large amount of data from different databases. For example, in business operations, different internal departments often use their own databases to manage specific businesses, such as the customer relationship management database of the sales department and the supply chain database of the production department. The data in these databases need to be integrated in the overall analysis. At the same time, different companies also need to integrate their respective data based on cooperation, data sharing and other needs. In order to solve these problems, technical personnel are constantly working hard and have proposed many technical solutions. In this field, the more cutting-edge existing technical solutions are: Chinese invention patent CN119052161A, a method, system, and device for dynamic multi-source data aggregation and scheduling based on the Internet of Things, discloses a method, system, and device for dynamic multi-source data aggregation and scheduling based on the Internet of Things. The method sets multiple constraints in the process of dynamic multi-source data aggregation and scheduling based on the Internet of Things, where the multi-source data is data from different Internet of Things nodes. When the multiple constraints are met, a first objective function is constructed, which is to minimize data redundancy and maximize transmission efficiency for the data transmitted during the dynamic multi-source data aggregation and scheduling based on the Internet of Things. Based on the first objective function, a global target path set is determined. When the global target path set is used to aggregate and schedule the multi-source data, the ratio between the used resources of all Internet of Things nodes in the global target path set and the total resources of the nodes is calculated. Based on the ratio, the resource allocation during the aggregation and scheduling of the multi-source data is adjusted. This application can improve task scheduling efficiency and reduce transmission delay.

[0003] Chinese invention patent CN118916241A discloses a method, device, terminal, and medium for data aggregation, which includes: receiving aggregation rules, determining the aggregation time and aggregation attributes of target aggregated data; reading initial reported data, and determining the reporting scenario of the initial reported data based on the aggregation time, wherein the reporting scenario includes a first reporting scenario and a second reporting scenario; performing single attribute identification on the initial reported data of the first reporting scenario to determine the first target reported data, wherein the first target reported data is the reported data of the non-aggregated attribute; performing single attribute identification on the initial reported data of the second reporting scenario to determine the second target reported data, wherein the second target reported data is the reported data of the aggregated attribute; and obtaining the target aggregated data based on the first target reported data and the first target reported data. The present invention can accurately perform data aggregation based on attributes, thereby improving the integrity and accuracy of reported data.

[0004] However, existing data integration technologies have exposed numerous shortcomings when faced with such complex multi-source data. For one thing, they often struggle to adapt to diverse and complex data sources. When data source quality fluctuates, such as when certain data sources suddenly experience data errors or are not updated in a timely manner, existing integration technologies lack effective response mechanisms and are unable to flexibly adjust to the real-time status of the data source, affecting the accuracy of the integration results. Furthermore, the architectural design of many integration systems is relatively rigid and inflexible, making it difficult to easily expand them to meet the diverse needs of users in different scenarios, thus limiting their application in a wider range of fields.

[0005] With the continuous advancement of data analysis technology, the demand for data visualization is increasing, showing a trend towards diversification and high interactivity. Different user groups and specific analysis scenarios have distinct requirements for visualization formats. For example, when displaying data changing over time, a line chart can more intuitively demonstrate the changing trend; while when comparing the proportion of different data categories, a pie chart or a bar chart is more appropriate. Furthermore, users often want to be able to interact with the visualization interface, such as zooming and filtering, to deeply explore the underlying information and better support decision-making. However, existing visualization tools have significant limitations. Their relatively simple functions only provide a limited number of visualization formats, which cannot meet the diverse needs of users. Moreover, many visualization tools lack integration with other data processing modules and cannot work effectively with data integration and analysis modules. As a result, visualization cannot fully play its role in supporting decision-making in the entire data analysis process. Summary of the Invention

[0006] In view of the problems described in the background technology, the present invention proposes a technical solution for an intelligent visualization system for dynamic aggregation of big data, the specific contents of which are as follows: An intelligent visualization system for dynamic aggregation of big data includes an adaptive dynamic weighted aggregation algorithm, anomaly detection based on deep learning, a multi-level visualization framework, a feedback learning mechanism, a data source identification module, a data connection module, a data preprocessing module, and a data analysis module. The system can automatically identify and connect various types of data sources and adaptively perform dynamic weighted aggregation calculations based on the weights of the data sources. Assuming the weight of the data source Di is wi, the formula for dynamic weighted aggregation is:

[0007] in, Is the data source The historical variance reflects the stability of the data. ϵ is the weight adjustment constant, which can correct errors according to actual conditions and avoid zero division errors. The value of is the weight; By introducing historical variance to dynamically adjust weights, the reliability of the data source can be better reflected. The smaller the variance, the greater the weight, and the greater the variance, the smaller the weight.

[0008] The deep learning-based anomaly detection uses a long short-term memory network to detect anomalies in time series data. The model input is the data of the past T time points, and the output is the predicted value at the next time point. The mean square error is used as the loss function:

[0009] in: is the actual value, It is the predicted value. By training the LSTM model, the system can identify abnormal fluctuations in the data, issue early warnings in time, and improve the accuracy of data analysis.

[0010] The multi-level visualization framework includes: a base layer that displays time series graphs of raw data; an analysis layer that displays aggregated data and calculates moving averages and standard deviation indicators; a decision layer that uses decision trees or random forest models to help users understand the potential impact of different choices and displays visualization results of different decision paths; and an interactive design layer that allows users to select a time range through a slider, and the system updates the visualization results in real time to enhance the user experience.

[0011] In the feedback learning mechanism, users can rate the accuracy and visualization effect of the data during use. The system adjusts the weights and model parameters based on user feedback. The feedback update formula sets the feedback score to f, and the weight update formula is:

[0012] Where: α is the learning rate, which controls the influence of feedback on the weight. is the new weight of the i-th data source, is the original weight of the i-th data source.

[0013] The data preprocessing module includes data cleaning and data standardization functions, and integrates data validation technology pattern matching and logic checking to further improve data quality. It can support real-time monitoring of data flow and immediate processing of new data entering the system to ensure the accuracy and timeliness of subsequent analysis.

[0014] The adaptive dynamic weighted aggregation algorithm uses a dynamic weighting method to dynamically adjust weights and dynamically update them using a machine learning model to improve the accuracy of aggregation results. Specifically, this mechanism employs reinforcement learning methods, allowing the system to self-learn and optimize in a constantly changing data environment, enabling rapid response and adjustment to emergencies or abnormal data. It optimizes data source weights based on the accuracy of historical data and real-time feedback. Deep learning technology is introduced to analyze data trends over long time spans, automatically adjusting weighting strategies to adapt to changes in data source reliability over different time periods, thereby improving the overall accuracy of data aggregation.

[0015] The data analysis module includes statistical analysis to identify specific situations in the data and make trend forecasts. It integrates time series analysis and forecasting models to make more accurate predictions about future trends and provide customized early warning services to help users prepare countermeasures in advance.

[0016] The multi-level visualization framework provides a variety of visualization forms, including line charts, bar charts and heat maps. Users can choose different visualization forms according to their needs and support highly customized visual configurations. Users can adjust color schemes, chart styles, etc. to meet specific display needs and enhance the flexibility and aesthetics of data expression. Users can select different data sources and time ranges for dynamic analysis. It also provides an integration interface with external systems, allowing users to export visualization results directly to other platforms or share analysis results with other systems.

[0017] The data connection module can support multiple connection protocols, including ODBC, JDBC and REST API, to achieve connection with different data sources, and add support for emerging database technologies NoSQL databases and distributed file systems, so as to access the widest range of data sources.

[0018] The feedback learning mechanism regularly evaluates the accuracy of each data source and adjusts the weight based on the evaluation results. It combines artificial intelligence technology to automatically identify key factors affecting the accuracy of data sources by learning from large amounts of historical data, and formulates personalized evaluation criteria accordingly to achieve more accurate weight adjustments.

[0019] Compared with the prior art, the present invention has the following beneficial technical effects: 1. An intelligent visualization system for dynamically aggregating big data uses automated identification and connection of multi-source data. The system can automatically identify and connect to various data sources (such as databases, APIs, files, and streaming data), supporting multiple data formats (such as CSV, JSON, and SQL) and protocols (such as HTTP, FTP, and WebSocket), adapting to complex data environments. Through intelligent algorithms, the system quickly establishes connections and monitors status in real time to ensure data continuity and integrity. In the event of a connection failure, the system automatically retries or switches to an alternate data source to ensure stability and reliability. This automated and intelligent design significantly reduces manual intervention and improves data integration efficiency, making it suitable for multi-source data integration scenarios such as enterprise data warehouses and IoT data platforms.

[0020] 2. An intelligent visualization system for dynamic big data aggregation uses a dynamic weighted aggregation algorithm. The core of this algorithm is to dynamically adjust weights based on the historical variance of the data source, reflecting its stability: smaller variances are associated with larger weights, while larger variances are associated with smaller weights. By introducing a tuning constant ϵ, the algorithm not only avoids division-by-zero errors but also smooths out extreme values, enhancing robustness. This historical variance-based weighting mechanism ensures that highly reliable data sources receive a greater weight in the aggregated results, significantly improving the accuracy of the results.

[0021] 3. An intelligent visualization system for dynamic big data aggregation uses weighting based on historical variance. This intelligent and transparent weighting calculation, with clear mathematical meaning, ensures that users can understand and trust the weighting results. By dynamically adjusting weights, the system can automatically adapt to changes in data sources. For example, if a data source exhibits instability (increased variance) over a period of time, its weight is automatically reduced, minimizing the negative impact on the aggregation results. This intelligent design not only enhances the system's adaptability but also strengthens user trust. Furthermore, the introduction of the tuning constant ϵ avoids over-reliance on a single data source, further improving the system's robustness.

[0022] 4. An intelligent visualization system for dynamic big data aggregation supports real-time data aggregation and analysis, rapidly responding to data changes. The dynamic weighted aggregation algorithm is computationally simple and efficient, making it suitable for large-scale data processing and real-time computing scenarios. Through parallel computing and optimization algorithms, the system efficiently completes data integration and analysis tasks. For example, in real-time monitoring systems (such as stock market monitoring and industrial equipment status monitoring), the system can quickly integrate multi-source data and generate accurate aggregated results, helping users make timely decisions. This efficiency and real-time performance give the system a significant advantage in modern data analysis.

[0023] 5. An intelligent visualization system for dynamic big data aggregation uses a modular design, enabling flexible expansion to support new data sources and analytical methods, meeting ever-changing data analysis needs. Through a dynamic weighted aggregation algorithm, the system significantly improves the accuracy and reliability of data integration and reduces the impact of low-quality data sources. Its automation, intelligence, and efficiency make it suitable for scenarios such as multi-source data fusion, real-time data aggregation, and data quality improvement. For example, in the financial sector, the system can integrate data from different exchanges to generate accurate market analysis reports; in the Internet of Things, the system can monitor device status in real time, providing early warnings and analytical support. Overall, the system offers significant advantages in multi-source data integration, real-time data analysis, and data quality improvement, meeting the complex needs of modern data analysis.

[0024] 6. An intelligent visualization system for dynamic big data aggregation uses a dynamic weighted aggregation algorithm that automatically adjusts weights based on real-time feedback and historical accuracy of data sources. This flexibility enables the system to better adapt to changes in data sources and improve the accuracy of aggregation results.

[0025] 7. The modular design of an intelligent visualization system for dynamically aggregated big data allows users to flexibly expand the system based on their needs, supporting a variety of data sources and analysis methods. This flexibility is rare in existing technologies and can meet the specific needs of different users.

[0026] 8. An intelligent visualization system for dynamic big data aggregation. By combining multiple visualization tools, such as Matplotlib, Seaborn, and Plotly, this technical solution provides rich visualization forms, enhancing the intuitiveness and interactivity of data presentation. This makes it easier for users to understand and analyze data.

[0027] 9. An intelligent visualization system for dynamically aggregated big data incorporates a real-time feedback mechanism that allows for timely assessment of data source accuracy and adjustment of weights. This mechanism, relatively uncommon in existing technologies, effectively improves the reliability of data integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is an overall logical architecture diagram of an intelligent visualization system that dynamically aggregates big data. DETAILED DESCRIPTION

[0029] An intelligent visualization system for dynamic aggregation of big data includes an adaptive dynamic weighted aggregation algorithm, anomaly detection based on deep learning, a multi-level visualization framework, a feedback learning mechanism, a data source identification module, a data connection module, a data preprocessing module, and a data analysis module. The system can automatically identify and connect various types of data sources and adaptively perform dynamic weighted aggregation calculations based on the weights of the data sources. Assuming the weight of the data source Di is wi, the formula for dynamic weighted aggregation is: -1 in, Is the data source The historical variance reflects the stability of the data. ϵ is the weight adjustment constant, which can correct errors according to actual conditions and avoid zero division errors. The value of is the weight.

[0030] By introducing historical variance to dynamically adjust weights, the reliability of the data source can be better reflected. The smaller the variance, the greater the weight, and the larger the variance, the smaller the weight.

[0031] In a specific embodiment, suppose a financial institution wants to build an intelligent visualization system that dynamically aggregates big data to analyze market conditions. The system needs to integrate data from different data sources, such as data from major stock exchanges, data from professional financial data providers, and data on hot discussions about the financial market on social media.

[0032] Step 1: The data source identification module will identify these data sources. For stock exchange data, its data stability is relatively high and the historical variance is small; while the stability of social media data is relatively low and the historical variance is large. The data connection module establishes connections with these data sources through the corresponding protocols.

[0033] Step 2: The data preprocessing module cleans and standardizes the data obtained from the connection. For example, the data of the stock exchange may have some irregular formats or missing values. These problems can be solved through data cleaning. At the same time, the data from different data sources are standardized to make them comparable.

[0034] Step 3: Set the stock exchange data source when performing dynamic weighted aggregation calculations The weight of , professional financial data provider data source The weight of , social media data sources The weight of According to Formula 1, due to the historical variance of stock exchange data Smaller, its corresponding weight Large; while the historical variance of social media data Larger, its corresponding weight The weight adjustment constant ϵ can be adjusted according to the actual situation. For example, it can be set to 0.01 to avoid division by zero errors and correct the data errors to a certain extent. In this way, the system can dynamically adjust the weight according to the reliability of the data source to achieve more accurate aggregation calculations.

[0035] The deep learning-based anomaly detection uses a long short-term memory network to detect anomalies in time series data. The model input is the data of the past T time points, and the output is the predicted value at the next time point. The mean square error is used as the loss function: -2 in: is the actual value, It is the predicted value. By training the LSTM model, the system can identify abnormal fluctuations in the data, issue early warnings in time, and improve the accuracy of data analysis.

[0036] In a specific embodiment, taking the sales data of an e-commerce platform as an example, the deep learning-based anomaly detection function of the system is used. The e-commerce platform generates a large amount of sales data every day, and these data show certain time series characteristics.

[0037] Step 1: The system collects sales data from the past period (assuming 30 days) as a training set. The Long Short-Term Memory (LSTM) model takes as input data from the past T time points (T can be set to 7 days), that is, the daily sales figures for the past week. The model's goal is to predict the sales figures for the next time point (day 8).

[0038] Step 2: During the training process, the mean square error is used as the loss function formula 2, where is the actual sales amount, is the sales amount predicted by the model. The system continuously adjusts the parameters of the LSTM model to reduce the value of the loss function.

[0039] Step 3. In actual application, the system acquires new sales data in real time and feeds the past seven days' data into the trained LSTM model to generate a predicted sales amount for the next day. If the error between the actual sales amount and the predicted amount exceeds a certain threshold, the system identifies this as an abnormal fluctuation and issues a timely warning. For example, on a particular day, the model predicts sales of 100,000 yuan, but actual sales reach 200,000 yuan. The system calculates the mean squared error and detects this significant discrepancy. It then issues an anomaly warning, alerting e-commerce platform operators that promotional activities may have exceeded expectations, data entry errors, or sudden changes in market demand, thereby improving the accuracy of data analysis.

[0040] The multi-level visualization framework includes: a base layer that displays time series graphs of raw data; an analysis layer that displays aggregated data and calculates moving averages and standard deviation indicators; a decision layer that uses decision trees or random forest models to help users understand the potential impact of different choices and displays visualization results of different decision paths; and an interactive design layer that allows users to select a time range through a slider, and the system updates the visualization results in real time to enhance the user experience.

[0041] In the feedback learning mechanism, users can rate the accuracy and visualization effect of the data during use. The system adjusts the weights and model parameters based on user feedback. The feedback update formula sets the feedback score to f, and the weight update formula is:

[0042] Where: α is the learning rate, which controls the influence of feedback on the weight. is the new weight of the i-th data source, is the original weight of the i-th data source.

[0043] The data preprocessing module includes data cleaning and data standardization functions, and integrates data validation technology pattern matching and logic checking to further improve data quality. It can support real-time monitoring of data flow and immediate processing of new data entering the system to ensure the accuracy and timeliness of subsequent analysis.

[0044] In a specific embodiment, the system receives a batch of new sales data. The data preprocessing module first cleans the data to remove duplicate records and outliers. The system then normalizes the data, converting sales figures into units of ten thousand yuan. Finally, the system ensures data integrity and consistency through pattern matching and logical checks.

[0045] In a specific embodiment, a logistics company uses this system to manage and analyze logistics data. Logistics data comes from a wide range of sources, including order systems, warehouse management systems, transportation management systems, etc. These data may have problems such as irregular formats, duplicate values, and missing values.

[0046] Regarding data cleansing, the system performs format checks on order numbers in the order system. For example, order numbers should be a fixed-length combination of numbers and letters. If an order number does not conform to this format, the system will mark it as abnormal data and take appropriate action, which may include deleting the record or correcting the order number by comparing it with other relevant data. For inventory data in the warehouse management system, duplicate records may exist. The system uses data cleansing functions to identify and delete these duplicate records to ensure data uniqueness.

[0047] Regarding data standardization, data in different systems may use different units and scales. For example, the weight of goods in an order system may be expressed in kilograms, while the weight of goods in a transportation management system may be expressed in tons. The system converts this data into the same unit to facilitate subsequent analysis and processing.

[0048] Integrated data validation technology, pattern matching and logic checking. When processing order data, the system checks the logical relationship between the order placement time, shipping time, and receipt time. If the order placement time is later than the shipping or receipt time, this is clearly illogical, and the system will flag and process these abnormal data.

[0049] In terms of real-time data flow monitoring, when new order data enters the system, the data preprocessing module immediately processes it. For example, new order data includes customer information, product information, and order status. The system immediately cleans and standardizes this data to ensure that subsequent data analysis can be carried out accurately and timely, providing a reliable basis for logistics companies' decision-making.

[0050] The adaptive dynamic weighted aggregation algorithm uses a dynamic weighting method to dynamically adjust weights and dynamically update them using a machine learning model to improve the accuracy of aggregation results. Specifically, this mechanism employs reinforcement learning methods, allowing the system to self-learn and optimize in a constantly changing data environment, enabling rapid response and adjustment to emergencies or abnormal data. It optimizes data source weights based on the accuracy of historical data and real-time feedback. Deep learning technology is introduced to analyze data trends over long time spans, automatically adjusting weighting strategies to adapt to changes in data source reliability over different time periods, thereby improving the overall accuracy of data aggregation.

[0051] In a specific embodiment, an energy company uses the system to analyze energy consumption data. Energy consumption data is affected by many factors, such as seasonal changes, economic activity levels, weather conditions, etc., and the data environment is constantly changing.

[0052] The adaptive dynamic weighted aggregation algorithm uses a dynamic weighting method and a machine learning model for dynamic updates. The system collects energy consumption data over a period of time (e.g., one year), including consumption data for different energy types (e.g., electricity, natural gas), as well as relevant influencing factors (e.g., temperature, humidity, and whether it is a weekday).

[0053] Application of reinforcement learning methods: The system compares aggregated energy consumption data with actual energy consumption and assigns rewards or penalties based on the comparison. For example, if the aggregated results closely match the actual situation, the system will award a positive reward; if the deviation is significant, a negative reward will be given. Through continuous learning and adjustment, the system can self-learn and optimize in an ever-changing data environment. When faced with unexpected situations, such as sudden extreme weather in a certain area causing unusual fluctuations in energy consumption, the system can respond quickly and adjust the weighting of data sources.

[0054] Data source weights are optimized based on historical data accuracy and real-time feedback. The system analyzes the accuracy of data provided by different data sources over time, appropriately increasing the weight of data sources with high accuracy and reducing the weight of data sources with low accuracy. Simultaneously, user feedback is collected in real time. For example, if energy company staff discover anomalies in the data from a particular data source and promptly provide feedback to the system, the system will adjust the weight of that data source based on this feedback.

[0055] Introducing deep learning technology: The system analyzes data trends over long time spans (such as five years) and automatically adjusts weighting strategies through deep learning models. For example, if it is found that energy consumption in the summer increases significantly due to the use of air conditioning each year, the system will automatically adjust the weights of data sources related to summer energy consumption before the summer arrives to adapt to changes in the reliability of data sources in different time periods, thereby improving the overall accuracy of data aggregation.

[0056] In a specific embodiment, the system monitors multiple stock market data sources and dynamically adjusts the weight of each data source using a reinforcement learning model. When a particular market experiences unusual fluctuations, the system automatically reduces the weight of that data source, minimizing its impact on the aggregated results. Furthermore, the system uses a deep learning model to analyze data trends over long time spans and automatically adjusts the weighting strategy to accommodate changes in data source reliability over time.

[0057] The data analysis module includes statistical analysis to identify specific situations in the data and predict trends. Integrating time series analysis and forecasting models enables more accurate predictions of future trends and provides customized early warning services to help users prepare for countermeasures in advance.

[0058] Specific implementation example: The system analyzed sales data from the past year and found a clear upward trend in the fourth quarter of each year. Based on this, the system used a time series analysis model to predict sales for the coming year and provided customized early warning services to help users prepare inventory in advance.

[0059] The multi-level visualization framework provides a variety of visualization forms, including line charts, bar charts and heat maps. Users can choose different visualization forms according to their needs and support highly customized visual configurations. Users can adjust color schemes, chart styles, etc. to meet specific display needs and enhance the flexibility and aesthetics of data expression. Users can select different data sources and time ranges for dynamic analysis. It also provides an integration interface with external systems, allowing users to export visualization results directly to other platforms or share analysis results with other systems.

[0060] Specific implementation example: A market research company uses this system to analyze market data. Market data includes market share, consumer satisfaction, sales growth rate, etc. of different products. The multi-level visualization framework provides multiple visualization forms. For market share data, users can choose a bar chart for visualization. The bar chart can clearly compare the market share of different products, and the height of each bar represents the market share of the corresponding product. For consumer satisfaction data, users can choose a line chart, and through the trend of the line, they can intuitively see how consumer satisfaction changes over time. For sales growth rate data, users can choose a heat map, which can use the depth of color to indicate the sales growth rate of different time periods and different products, allowing users to quickly understand the distribution of sales growth.

[0061] Users can adjust color schemes and chart styles to suit their needs. For example, in a bar chart, you can set the color of the columns representing best-selling products to red and the columns representing slow-selling products to blue to enhance data visibility. Users can also adjust the chart's font size, line thickness, and other styles to meet specific display needs.

[0062] Users can select different data sources and timeframes for dynamic analysis. For example, if a user wants to analyze market data changes in a specific region over the past year, they simply need to select the corresponding data source (data for that region) and timeframe (the past year), and the system will update the visualization results in real time.

[0063] Market research companies may need to share analysis results with other departments or export visualizations into reports. Through the system's integrated interfaces with external systems, users can export visualizations directly to office software like Excel and PowerPoint, or share analysis results with the company's internal management system, improving work efficiency and information flow.

[0064] The data connection module can support multiple connection protocols, including ODBC, JDBC and REST API, to achieve connection with different data sources, and add support for emerging database technologies NoSQL databases and distributed file systems, so as to access the widest range of data sources.

[0065] In a specific embodiment, the system connects to a relational database via the ODBC protocol, another relational database via the JDBC protocol, and a cloud storage service via the REST API protocol. Furthermore, the system supports connections to MongoDB and the Hadoop distributed file system, enabling broad access to a variety of data sources. A large enterprise data center needs to integrate data from various sources, including traditional relational databases, emerging NoSQL databases, and distributed file systems.

[0066] The data connection module supports multiple connection protocols. For internal relational databases such as MySQL and Oracle, the system uses the ODBC (Open Database Connectivity) protocol. The ODBC protocol provides a standard way to access different types of databases. By configuring the corresponding data source name, username, and password, the system can establish connections with MySQL and Oracle databases and retrieve data from them.

[0067] For databases used by Java-developed applications, the system uses the JDBC (Java Database Connectivity) protocol to connect. JDBC is a standard API for interacting with databases in the Java language. By loading the appropriate JDBC driver, the system can establish a connection with the database and perform data reading and writing operations.

[0068] For some web service-based data sources, the system uses REST APIs (Representational State Transfer Application Programming Interfaces) to connect. For example, a business might obtain market data from a third-party data provider, which is often provided via REST APIs. The system retrieves the required data by sending HTTP requests to the corresponding API addresses.

[0069] Support has been added for emerging database technologies such as NoSQL databases and distributed file systems. For NoSQL databases such as MongoDB and Cassandra, the system establishes connections with them through the corresponding drivers and protocols. MongoDB is a document-based database. The system can connect to MongoDB databases and read document data using MongoDB's Java driver or Python driver. For distributed file systems such as Hadoop's HDFS (Hadoop Distributed File System), the system can connect to HDFS through HDFS's Java API or command-line tools to retrieve the file data stored there. In this way, the system can access the widest range of data sources, providing strong support for enterprise data integration and analysis.

[0070] The feedback learning mechanism regularly evaluates the accuracy of each data source and adjusts the weight based on the evaluation results. It combines artificial intelligence technology to automatically identify key factors affecting the accuracy of data sources by learning from large amounts of historical data, and formulates personalized evaluation criteria accordingly to achieve more accurate weight adjustments.

[0071] In a specific embodiment, a research institute uses this system to analyze data from different experimental platforms, including physics, chemistry, and biology. The accuracy of the data from each platform may be affected by various factors, such as the precision of the experimental equipment and the operator's skill level.

[0072] The feedback learning mechanism regularly evaluates the accuracy of each data source. The system collects data from each experimental platform over a period of time (e.g., one month) and compares it with known standard data or data from other reliable data sources. For example, for data from a physics experimental platform, the system compares the measured physical parameter values ​​with international standard physical constants and calculates the error rate. The accuracy of the data source is assessed based on the size of the error rate.

[0073] Integrating artificial intelligence technology, the system automatically identifies key factors influencing data source accuracy by studying large amounts of historical data. The system analyzes the relationship between historical data and factors such as the use time of experimental equipment, experimental personnel's operation records, and the temperature and humidity of the experimental environment. Using machine learning algorithms, such as decision trees or neural networks, the system identifies key factors that significantly impact data accuracy.

[0074] Based on this, personalized evaluation criteria are developed. For physics experiment platforms, if the data accuracy decreases with age, the system will adjust the evaluation criteria accordingly. When experimental equipment exceeds a certain age, the accuracy requirements for the platform's data will be appropriately lowered, and its weight will be adjusted accordingly.

[0075] Assume that the original weight of the physical experiment platform is , the original weight of the chemical experiment platform is , the original weight of the biological experiment platform is After evaluation, the data accuracy of the physics experiment platform is high, with a feedback score of 0.8; the data accuracy of the chemistry experiment platform is average, with a feedback score of 0.5; the data accuracy of the biology experiment platform is low, with a feedback score of 0.2. The learning rate is set to 0.1. According to the weight update formula = *(1+α*f), the new weight of the physical experiment platform = *(1+0.1*0.2); New weight of the chemical experiment platform = *(1+0.1*0.5); New weight of biological experiment platform = *(1+0.1*0.2). In this way, more precise weight adjustment can be achieved, improving the accuracy of data aggregation.

[0076] Special statement: The "embodiment" and the like described in this specification refer to the specific features, elements or characteristics described in conjunction with the embodiment included in the embodiment generally described in this application. The appearance of the same expression in multiple places in the specification does not limit the specific reference to the same embodiment. In other words, when describing a specific feature, element or characteristic in conjunction with any embodiment, it is claimed that such feature, element or characteristic can be realized in conjunction with other embodiments and is included in the scope of the claims for protection of the present invention; the embodiment describes the present invention with reference to multiple explanatory embodiments of the logical architecture and ideas of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art can design many other modifications and implementation methods within the framework of the technical solution of the present invention, and can make various non-essential variations and improvements to the key points of the technical solution. For those skilled in the art, other uses will also be obvious, and non-substantial changes or replacements that can be easily thought of will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. An intelligent visualization system for dynamic aggregation of big data, including an adaptive dynamic weighted aggregation algorithm, anomaly detection based on deep learning, a multi-level visualization framework, a feedback learning mechanism, a data source identification module, a data connection module, a data preprocessing module, and a data analysis module, characterized in that: The system can automatically identify and connect various types of data sources, and adaptively perform dynamic weighted aggregation calculations based on the weights of the data sources. If the weight of the data source Di is set to wi, the formula for dynamic weighted aggregation is: , in, Is the data source The historical variance reflects the stability of the data. ϵ is the weight adjustment constant, which can correct errors according to actual conditions and avoid zero division errors. The value of is the weight; by introducing historical variance to dynamically adjust the weight, the reliability of the data source can be better reflected. The smaller the variance, the greater the weight, and the greater the variance, the smaller the weight.

2. The intelligent visualization system for dynamic aggregation of big data according to claim 1, characterized in that: The deep learning-based anomaly detection uses a long short-term memory network to detect anomalies in time series data. The model input is the data of the past T time points, and the output is the predicted value at the next time point. The mean square error is used as the loss function: , in: is the actual value, It is the predicted value. By training the LSTM model, the system can identify abnormal fluctuations in the data, issue early warnings in time, and improve the accuracy of data analysis.

3. The intelligent visualization system for dynamic aggregation of big data according to claim 1, characterized in that: The multi-level visualization framework includes: a base layer that displays time series graphs of raw data; an analysis layer that displays aggregated data and calculates moving averages and standard deviation indicators; a decision layer that uses decision trees or random forest models to help users understand the potential impact of different choices and displays visualization results of different decision paths; and an interactive design layer that allows users to select a time range through a slider, and the system updates the visualization results in real time to enhance the user experience.

4. The intelligent visualization system for dynamic aggregation of big data according to claim 1, characterized in that: In the feedback learning mechanism, users can rate the accuracy and visualization effect of the data during use. The system adjusts the weights and model parameters based on user feedback. The feedback update formula sets the feedback score to f, and the weight update formula is: , Where: α is the learning rate, which controls the influence of feedback on the weight. is the new weight of the i-th data source, is the original weight of the i-th data source.

5. The intelligent visualization system for dynamic aggregation of big data according to claim 1, characterized in that: The data preprocessing module includes data cleaning and data standardization functions, and integrates data validation technology pattern matching and logic checking to further improve data quality. It can support real-time monitoring of data flow and immediate processing of new data entering the system to ensure the accuracy and timeliness of subsequent analysis.

6. The intelligent visualization system for dynamic aggregation of big data according to claim 1, characterized in that: The adaptive dynamic weighted aggregation algorithm adopts a method of dynamically adjusting weights and uses a machine learning model for dynamic updates to improve the accuracy of aggregation results. It adopts a reinforcement learning method to allow the system to self-learn and optimize in a constantly changing data environment, achieving rapid response and adjustment to emergencies or abnormal data. It optimizes data source weights based on the accuracy of historical data and real-time feedback, introduces deep learning technology, and automatically adjusts weight strategies by analyzing data trends over long time spans to adapt to changes in data source reliability in different time periods, thereby improving the overall accuracy of data aggregation.

7. The intelligent visualization system for dynamic aggregation of big data according to claim 1, characterized in that: The data analysis module includes statistical analysis to identify specific situations in the data and make trend forecasts. It integrates time series analysis and forecasting models to make more accurate predictions about future trends and provide customized early warning services to help users prepare countermeasures in advance.

8. The intelligent visualization system for dynamic aggregation of big data according to claim 1, characterized in that: The multi-level visualization framework provides a variety of visualization forms, including line charts, bar charts and heat maps. Users can choose different visualization forms according to their needs and support highly customized visual configurations. Users can adjust color schemes, chart styles, etc. to meet specific display needs and enhance the flexibility and aesthetics of data expression. Users can select different data sources and time ranges for dynamic analysis. It also provides an integration interface with external systems, allowing users to export visualization results directly to other platforms or share analysis results with other systems.

9. The intelligent visualization system for dynamic aggregation of big data according to claim 1, characterized in that: The data connection module can support multiple connection protocols, including ODBC, JDBC and REST API, to achieve connection with different data sources, and add support for emerging database technologies NoSQL databases and distributed file systems, so as to access the widest range of data sources.

10. The intelligent visualization system for dynamic aggregation of big data according to claim 1, characterized in that: The feedback learning mechanism regularly evaluates the accuracy of each data source and adjusts the weight based on the evaluation results. It combines artificial intelligence technology to automatically identify key factors affecting the accuracy of data sources by learning from large amounts of historical data, and formulates personalized evaluation criteria accordingly to achieve more accurate weight adjustments.

Images