Multi-source heterogeneous data aggregation system and method

Abstract

The invention provides a multi-source heterogeneous data aggregation system and method, and relates to the technical field of data processing. The multi-source heterogeneous data aggregation system comprises a data acquisition layer which is responsible for acquiring data from heterogeneous data sources of a power and environment data system and a cloud platform data system and comprises a plurality of relational databases and time sequence databases; the data processing layer is used for cleaning, converting and aggregating the collected data to ensure the quality and consistency of the data; the data storage layer is used for storing the processed data in a relational database to facilitate unified management and query of the data; and the application layer is used for providing data query, analysis and report generation functions for users to meet the requirements of different users. According to the invention, the power and environment of the data center and the heterogeneous data of the cloud platform can be efficiently collected to one platform, and unified management and analysis of the data are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a multi-source heterogeneous data aggregation system and method. Background Art

[0002] In today's highly digitalized era, data centers are the core hubs for information storage and processing, and their management and operation are becoming increasingly complex. Power and environmental data are mainly responsible for monitoring and managing the physical infrastructure of data centers, such as electricity, cooling, and space. Cloud platform data focuses on the management and scheduling of cloud computing, supercomputing, and intelligent computing resources. However, these two systems usually use different data formats, storage methods, and communication protocols, making it difficult to uniformly manage and analyze data.

[0003] Currently, there are two main methods for processing heterogeneous data: data conversion and data integration. Data conversion usually requires customized development for different data sources, which is costly and inefficient. Data integration requires the establishment of a complex data warehouse, which has high requirements for hardware and software.

[0004] In addition, existing data aggregation methods often have compatibility issues and low data transmission efficiency when dealing with various types of databases (relational databases, non-relational databases, and time series databases, etc.). For example, the query languages ​​and data structures of different databases vary greatly, making the data collection and conversion process complicated. At the same time, traditional data transmission methods may be limited by network bandwidth and latency, resulting in slow data aggregation. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a multi-source heterogeneous data aggregation system and method, which solves the problems raised by the background technology.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: A multi-source heterogeneous data aggregation system, comprising:

[0009] Data collection layer: responsible for collecting data from heterogeneous data sources of the power and environment data system and the cloud platform data system, including multiple relational databases and time series databases;

[0010] Data processing layer: cleans, converts and aggregates the collected data to ensure data quality and consistency;

[0011] Data storage layer: Store the processed data in a relational database to facilitate unified management and query of data;

[0012] Application layer: Provides users with data query, analysis and report generation functions to meet the needs of different users.

[0013] Preferably, during the data collection process, attention is paid to data transmission security, and the data source is authenticated and authorized to ensure that only legitimate users can access the data source.

[0014] Preferably, the data processing layer includes:

[0015] Data cleaning module: removes noise and errors in the data, specifically using data verification and data filtering methods for cleaning;

[0016] Data conversion module: convert data in different formats into a unified format;

[0017] Data aggregation module: aggregates and analyzes data to extract useful information and indicators, specifically using statistical analysis and machine learning methods for aggregation.

[0018] Preferably, during the data storage process, the appropriate table structure and index are determined according to the characteristics of the data and application requirements, and the data stored in the database is encrypted. At the same time, the database is backed up and optimized regularly to ensure the security and reliability of the data.

[0019] Preferably, the application layer includes:

[0020] Data query module: Users query data in the database through SQL query statements or visual interfaces, providing a variety of query conditions and filtering functions to help users quickly find the data they need;

[0021] Data analysis module: users conduct statistical analysis, trend analysis and correlation analysis on data to extract useful information and knowledge;

[0022] Report generation module: Users can generate various types of reports according to their needs, such as daily, weekly, and monthly reports. Report templates and generation tools are provided to facilitate users to quickly generate the required reports.

[0023] The present invention further discloses a multi-source heterogeneous data aggregation method, based on the multi-source heterogeneous data aggregation system as described above, comprising the following steps:

[0024] Step 1: Get the data source:

[0025] Acquire various types of data from power and environmental data systems, cloud platform data systems, and other related systems;

[0026] Step 2: Determine the data source type:

[0027] Classify the acquired data sources to clarify whether they are relational databases, time series databases, or other types of data sources;

[0028] Step 3: Collect data from different types of data sources:

[0029] Collect data from relational databases, time series databases, and other data sources;

[0030] Step 4: Data format unification and identity authentication and authorization:

[0031] Convert data from different data sources into a unified data format to facilitate subsequent processing and analysis; at the same time, strictly authenticate and authorize data sources through a secure key mechanism to ensure that only legitimate and authorized data sources can be accessed and used, thus ensuring the security and legitimacy of data;

[0032] Step 5: Send data to the message queue:

[0033] Send the data after format unification and identity authentication and authorization to the message queue, waiting for subsequent processing and consumption;

[0034] Step 6: Message queue data consumption and reception:

[0035] The aggregation platform consumes and receives data from the message queue, preparing for subsequent data processing operations;

[0036] Step 7: Data processing in memory:

[0037] Perform comprehensive data cleaning on the received data in memory to remove noise, duplicate data and invalid data;

[0038] Step 8: Data persistence storage and optimization:

[0039] The processed data is persistently stored in the database, and incremental / full backups are performed regularly to prevent data loss. At the same time, the database is optimized, including adjustment of the database structure, creation and optimization of indexes, to improve data storage efficiency and query performance.

[0040] Preferably, in step three, for a relational database, a connection is established using a corresponding database driver, and the required data is accurately obtained by writing SQL query statements; in the process of obtaining data, the performance of the database and the integrity of the data are considered to ensure that accurate and comprehensive data are obtained; after obtaining the data, it is preliminarily sorted and converted, some unnecessary fields are removed, or the data format is preliminarily adjusted to make it more in line with the requirements of subsequent processing.

[0041] Preferably, in step three, for the time series database: adopt a specific HTTP API connection method, make full use of the characteristics of the time series database, and extract valid time series data; in the data extraction process, pay attention to the timestamp processing of the data and the accuracy verification of the data to ensure that the acquired time series data has good quality and availability; after acquiring the data, perform necessary conversion and preprocessing according to the characteristics of the time series data to adapt to the subsequent processing flow.

[0042] Preferably, in step three, for other data sources: data is acquired through customized API interfaces, which are specially designed and developed according to the special needs and characteristics of the data sources; when acquiring data, the stability of the interface and the accuracy of the data must be ensured, and the acquired data must be converted into the necessary format and preliminarily processed so that it can be integrated with other types of data.

[0043] (III) Beneficial effects

[0044] The present invention provides a multi-source heterogeneous data aggregation system and method, which has the following beneficial effects:

[0045] 1. Efficiency: The combination of message queues and API interfaces achieves efficient data transmission and processing, improving the system's throughput and response speed.

[0046] 2. Compatibility: Special data collection and conversion modules are designed for different types of databases to improve compatibility with heterogeneous data sources.

[0047] 3. Scalability: The platform has good scalability and can easily add new data sources and data processing modules to meet the needs of different users.

[0048] 4. Flexibility: Users can choose different data sources and data processing methods according to their needs to achieve personalized data aggregation and management.

[0049] 5. Ease of use: The platform provides a user-friendly interface and rich functions, making it convenient for users to perform operations such as data query, analysis and report generation.

[0050] 6. Reliability: Use advanced technologies and architectures to ensure secure and reliable data transmission and storage. For example, using message queue middleware can ensure that data is not lost or transmitted repeatedly; backing up and optimizing the database can ensure data security and reliability. At the same time, data security is guaranteed by encrypting, desensitizing, and controlling access to the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the data aggregation system architecture of the present invention;

[0052] Figure 2 It is a flow chart of the data aggregation method of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Example:

[0055] An embodiment of the present invention provides a multi-source heterogeneous data aggregation system, including:

[0056] Data collection layer: responsible for collecting data from heterogeneous data sources of power and environment data systems and cloud platform data systems, including multiple relational databases and time series databases. A combination of message queues and API interfaces is used to achieve efficient data transmission. During the data collection process, attention is paid to data transmission security, and data sources are authenticated and authorized to ensure that only legitimate users can access the data source. Authentication and authorization can be performed using user names and passwords, digital certificates, etc. For different types of databases, use the corresponding database driver or data collection tool to connect to the data source and send the data to the message queue. For other types of data sources, such as file systems, network protocols, etc., data collection can be performed through customized API interfaces.

[0057] Data processing layer: Clean, convert and aggregate the collected data to ensure the quality and consistency of the data. Special data processing modules are designed for different types of databases. In the process of data processing, data processing security is considered, and the data is desensitized to protect the security of sensitive data. Sensitive data can be desensitized using data desensitization algorithms, such as replacement and encryption. The data cleaning module removes noise and errors in the data, such as duplicate data and invalid data. Data verification and data filtering can be used for cleaning. The data conversion module converts data in different formats into a unified format for subsequent processing and analysis. Appropriate conversion algorithms and tools can be selected according to the type and characteristics of the data. The data aggregation module aggregates and analyzes the data to extract useful information and indicators. Statistical analysis, machine learning and other methods can be used for aggregation.

[0058] Data storage layer: Store the processed data in a relational database to facilitate unified management and query of data. Improve the efficiency of data query by optimizing the database structure and index. Design a reasonable database table structure, including table name, field name, field type, etc. Determine the appropriate table structure and index according to the characteristics of the data and application requirements. Encrypt the data stored in the database to prevent illegal access to the data and ensure data storage security. Back up and optimize the database regularly to ensure data security and reliability. Database backup tools, index optimization tools, etc. can be used for backup and optimization.

[0059] Application layer: Provide users with functions such as data query, analysis and report generation to meet the needs of different users. During the data query and application process, user access control is performed to limit user access rights to the database to ensure data application security. Access control can be performed using methods such as database user management and permission management to ensure that only authorized users can access data in the database. Data query function users can query data in the database through SQL query statements or visual interfaces. Provide a variety of query conditions and filtering functions to facilitate users to quickly find the required data. Data analysis function users can perform statistical analysis, trend analysis, and correlation analysis on data to extract useful information and knowledge. Provide data analysis tools and algorithms to support users to perform customized analysis. Report generation function users can generate various types of reports according to their needs, such as daily, weekly, monthly, etc. Provide report templates and generation tools to facilitate users to quickly generate required reports.

[0060] Specifically, Figure 1 As shown, the embodiments of the present invention are described in detail below:

[0061] 1. Data Collection

[0062] 1.1 For relational databases:

[0063] (1.1.1) Determine the database connection parameters, including database address, port number, user name, password, etc. Configure the data collection tool so that it can connect to the target database.

[0064] (1.1.2) Use a suitable database driver, such as JDBC driver, to establish a connection with the database. Obtain the data in the required data table by executing SQL query statements. For example, for a MySQL database, you can use the "SELECT * FROM table_name" statement to obtain all data in the specified table.

[0065] (1.1.3) Perform preliminary processing on the acquired data, such as removing unnecessary fields and converting data types, so that it meets the requirements of the data aggregation platform. Then, convert the data into a unified data format, such as JSON format, to facilitate subsequent transmission and processing.

[0066] (1.1.4) Send the processed data to the message queue. Select the appropriate message queue middleware, create the corresponding topic, and send the data to the topic in the form of a message. Ensure the reliability and sequence of the message so that the subsequent processing links can correctly receive and process the data. Before sending the data, authenticate and authorize the data source to ensure that only legitimate users can access the data source.

[0067] 1.2 For time series databases:

[0068] (1.2.1) Analyze the characteristics and data structure of the time series database and determine the appropriate data collection method. For example, for the InfluxDB database, you can use the HTTP API it provides for data collection.

[0069] (1.2.2) Configure the data collection tool and set the connection parameters with the time series database, including database address, port number, user name, password, etc. At the same time, determine the data range and time interval to be collected.

[0070] (1.2.3) Use specialized time series data processing algorithms to convert and process the collected data, for example, timestamp processing, data aggregation, etc., to meet the requirements of the data aggregation platform.

[0071] (1.2.4) Convert the processed data into a unified data format and send it to the message queue. This is similar to the processing method of relational databases, ensuring the reliability and order of the data.

[0072] 1.3 For other types of data sources, such as file systems, network protocols, etc.:

[0073] (1.3.1) Develop customized API interfaces based on the specific types and characteristics of data sources. For example, for data in a file system, you can develop a file reading API that can read files in a specific format and convert the data into a unified data format.

[0074] (1.3.2) Configure the parameters of the API interface, including the path of the data source, file format, network protocol address and port number, etc. Ensure that the API interface can correctly access and read the data in the data source.

[0075] (1.3.3) Call the API interface to obtain data and perform necessary processing and conversion. After converting the data into a unified data format, send it to the message queue for subsequent processing. When calling the API interface, perform identity authentication and authorization to ensure that only legitimate users can access the data source.

[0076] 2. Data Processing

[0077] 2.1 Data cleaning:

[0078] (2.1.1) Read data from the message queue and use data cleaning algorithms to remove noise and errors in the data. For example, for duplicate data, you can compare key fields of the data, such as the ID field, to determine whether it is duplicated and remove the duplicate data. For invalid data, you can judge according to specific rules of the data, such as if the data value exceeds the reasonable range, and filter out the invalid data.

[0079] (2.2.2) Verify the data to ensure its integrity and accuracy. You can use data verification algorithms, such as checksums, hash values, etc. to verify the data. If the data verification fails, mark the data as invalid and handle it accordingly.

[0080] 2.2 Data conversion:

[0081] (2.2.1) Select appropriate conversion algorithms and tools based on the data formats of different data sources and the requirements of the data aggregation platform. For example, for data in JSON format, you can use a JSON parsing library to convert it to the data format in a relational database table. For data in XML format, you can use an XML parser to convert it to a unified data format.

[0082] (2.2.2) Convert the data to ensure that the data type meets the requirements of the data aggregation platform. For example, convert string data to numeric or date data.

[0083] 2.3 Data Aggregation:

[0084] (2.3.1) Use statistical analysis, machine learning and other methods to aggregate and analyze data. For example, you can calculate statistical indicators such as the average, sum, maximum, and minimum of the data. You can also use machine learning algorithms such as clustering algorithms and classification algorithms to analyze and process data and extract useful information and indicators.

[0085] (2.3.2) Store the aggregated and analyzed data in a temporary database or memory, awaiting further processing. You can choose a suitable temporary storage method, such as an in-memory database, temporary files, etc., to ensure data storage and reading efficiency. During data processing, desensitize sensitive data to protect data security.

[0086] 3. Data Storage

[0087] 3.1 Design a reasonable database table structure:

[0088] (3.1.1) Determine the name of the database table, field name, field type, etc. according to the characteristics of the data and application requirements. For example, for the collected power and environmental data, you can design a database table containing fields such as equipment information, performance indicators, and fault records.

[0089] (3.1.2) Select the appropriate index type and creation method to improve the efficiency of data query. For example, you can create a primary key index, unique index, common index, etc. according to the data query requirements. For fields that are frequently queried, you can create an index to speed up the query.

[0090] 3.2 Back up and optimize the database regularly:

[0091] (3.2.1) Select a suitable database backup tool. Configure the parameters of the backup tool, including the backup directory, backup frequency, etc. Perform backup operations regularly and back up the data in the database to the specified directory to prevent data loss.

[0092] (3.2.2) Optimize the database, such as optimizing table structure, index, query statements, etc. You can use database optimization tools to analyze the performance bottleneck of the database and make corresponding optimizations. For example, for tables that are frequently queried, you can partition the table to improve query efficiency. Encrypt the data stored in the database to ensure data security.

[0093] 4. Application Function

[0094] 4.1 Data Query:

[0095] (4.1.1) Develop a data query interface to support users to query data in the database using SQL query statements. Provide a friendly query interface so that users can easily enter query conditions and perform query operations. For example, you can use a web interface or command line tool to allow users to enter SQL query statements and return query results.

[0096] (4.1.2) Provide a visual interface to facilitate users to query and analyze data. You can use data visualization tools such as Echarts, D3.js, etc. to display the query results in the form of charts, allowing users to more intuitively understand the characteristics and trends of the data. During the data query process, user access control is performed to limit the user's access rights to the database.

[0097] 4.2 Data Analysis:

[0098] Develop data analysis tools to support users in statistical analysis, trend analysis, and correlation analysis of data. Provide a wealth of analysis functions and algorithms to enable users to analyze data according to their needs. For example, you can use statistical analysis tools to calculate statistical indicators such as the mean, standard deviation, and variance of data; use trend analysis tools to analyze the changing trends of data; and use correlation analysis tools to analyze the correlation between data.

[0099] 4.3 Report Generation:

[0100] Provide report templates and generation tools to support users to generate various types of reports according to their needs. Users can select appropriate report templates, such as daily, weekly, monthly, etc., and set the parameters and format of the report. Then, the system will automatically generate reports according to the user's settings, and provide download and print functions. In the process of report generation, ensure the security and accuracy of data.

[0101] like Figure 2 As shown, an embodiment of the present invention further provides a multi-source heterogeneous data aggregation method, based on the multi-source heterogeneous data aggregation system as described above, comprising the following steps:

[0102] Step 1: Get the data source:

[0103] Obtain various types of data from power and environmental data systems, cloud platform data systems and other related systems. These data come from a wide range of sources and in various formats.

[0104] Step 2: Determine the data source type:

[0105] Classify the acquired data sources to clarify whether they belong to relational databases (such as MySQL, Oracle, etc.), time series databases, or other types of data sources.

[0106] Step 3: Collect data from different types of data sources:

[0107] (1) For relational databases: Use the corresponding database driver to establish a connection and write SQL query statements to accurately obtain the required data. In the process of obtaining data, consider the performance of the database and the integrity of the data to ensure that accurate and comprehensive data is obtained. After obtaining the data, perform preliminary sorting and conversion on it, such as removing some unnecessary fields or making preliminary adjustments to the data format to make it more in line with the requirements of subsequent processing.

[0108] (2) For time series databases: Use a specific HTTP API connection method to make full use of the characteristics of the time series database to extract valid time series data. During the data extraction process, pay attention to the timestamp processing of the data and the accuracy verification of the data to ensure that the acquired time series data has good quality and usability. After acquiring the data, perform necessary conversion and preprocessing based on the characteristics of the time series data to adapt to the subsequent processing flow.

[0109] (3) For other data sources: Data is acquired through customized API interfaces, which are specially designed and developed based on the special needs and characteristics of the data source. When acquiring data, the stability of the interface and the accuracy of the data must be ensured, and the acquired data must be converted into the necessary format and processed initially so that it can be integrated with other types of data.

[0110] Step 4: Data format unification and identity authentication and authorization. Convert data from different data sources into a unified data format for subsequent processing and analysis. At the same time, strictly authenticate and authorize the data source through a secure key mechanism to ensure that only legitimate and authorized data sources can be accessed and used, thereby ensuring the security and legitimacy of the data.

[0111] Step 5: Send data to the message queue. Send the data after format unification and identity authentication and authorization to the message queue, waiting for subsequent processing and consumption. When sending data, ensure the integrity and accuracy of the data to avoid data loss or damage.

[0112] Step 6: Message queue data consumption and reception. The aggregation platform consumes and receives data from the message queue, preparing for subsequent data processing operations.

[0113] Step 7: Data processing in memory. Perform comprehensive data cleaning on the received data in memory to remove noise, duplicate data, and invalid data. Ensure the accuracy and integrity of the data through data verification. Then perform data conversion to convert data in different formats into a unified standard format for subsequent analysis and use. Then perform data aggregation and analysis, and use statistical analysis, machine learning and other methods to extract useful information and indicators. In this process, sensitive data is desensitized to protect data privacy and security.

[0114] Step 8: Data persistence and optimization. The processed data is persistently stored in the database, and incremental / full backups are performed regularly to prevent data loss. At the same time, the database is optimized, including database structure adjustment, index creation and optimization, etc., to improve data storage efficiency and query performance.

[0115] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-source heterogeneous data aggregation system, characterized by: include: Data collection layer: responsible for collecting data from heterogeneous data sources of the power and environment data system and the cloud platform data system, including multiple relational databases and time series databases; Data processing layer: cleans, converts and aggregates the collected data to ensure data quality and consistency; Data storage layer: Store the processed data in a relational database to facilitate unified management and query of data; Application layer: Provides users with data query, analysis and report generation functions to meet the needs of different users.

2. The multi-source heterogeneous data aggregation system according to claim 1, characterized in that: During the data collection process, attention is paid to data transmission security, and data sources are authenticated and authorized to ensure that only legitimate users can access the data source.

3. The multi-source heterogeneous data aggregation system according to claim 1, characterized in that: The data processing layer includes: Data cleaning module: removes noise and errors in the data, specifically using data verification and data filtering methods for cleaning; Data conversion module: convert data in different formats into a unified format; Data aggregation module: aggregates and analyzes data to extract useful information and indicators, specifically using statistical analysis and machine learning methods for aggregation.

4. The multi-source heterogeneous data aggregation system according to claim 1, characterized in that: During the data storage process, the appropriate table structure and index are determined based on the characteristics of the data and application requirements, and the data stored in the database is encrypted. At the same time, the database is backed up and optimized regularly to ensure the security and reliability of the data.

5. The multi-source heterogeneous data aggregation system according to claim 1, characterized in that: The application layer includes: Data query module: Users query data in the database through SQL query statements or visual interfaces, providing a variety of query conditions and filtering functions to help users quickly find the data they need; Data analysis module: users conduct statistical analysis, trend analysis and correlation analysis on data to extract useful information and knowledge; Report generation module: Users can generate various types of reports according to their needs, such as daily, weekly, and monthly reports. Report templates and generation tools are provided to facilitate users to quickly generate the required reports.

6. A multi-source heterogeneous data aggregation method, based on a multi-source heterogeneous data aggregation system according to any one of claims 1 to 5, characterized in that: The following steps are involved: Step 1: Get the data source: Acquire various types of data from power and environmental data systems, cloud platform data systems, and other related systems; Step 2: Determine the data source type: Classify the acquired data sources to clarify whether they are relational databases, time series databases, or other types of data sources; Step 3: Collect data from different types of data sources: Collect data from relational databases, time series databases, and other data sources; Step 4: Data format unification and identity authentication and authorization: Convert data from different data sources into a unified data format to facilitate subsequent processing and analysis; at the same time, strictly authenticate and authorize data sources through a secure key mechanism to ensure that only legitimate and authorized data sources can be accessed and used, thus ensuring the security and legitimacy of data; Step 5: Send data to the message queue: Send the data after format unification and identity authentication and authorization to the message queue, waiting for subsequent processing and consumption; Step 6: Message queue data consumption and reception: The aggregation platform consumes and receives data from the message queue, preparing for subsequent data processing operations; Step 7: Data processing in memory: Perform comprehensive data cleaning on the received data in memory to remove noise, duplicate data and invalid data; Step 8: Data persistence storage and optimization: The processed data is persistently stored in the database, and incremental / full backups are performed regularly to prevent data loss. At the same time, the database is optimized, including adjustment of the database structure, creation and optimization of indexes, to improve data storage efficiency and query performance.

7. A multi-source heterogeneous data aggregation method according to claim 6, characterized in that: In step three, for relational databases, use the corresponding database driver to establish a connection, and accurately obtain the required data by writing SQL query statements; in the process of obtaining data, consider the performance of the database and the integrity of the data to ensure that accurate and comprehensive data is obtained; after obtaining the data, perform preliminary sorting and conversion on it, remove some unnecessary fields or make preliminary adjustments to the data format to make it more in line with the requirements of subsequent processing.

8. The multi-source heterogeneous data aggregation method according to claim 6, characterized in that: In step three, for the time series database: use a specific HTTP API connection method to make full use of the characteristics of the time series database to extract valid time series data; in the data extraction process, pay attention to the data timestamp processing and data accuracy verification to ensure that the acquired time series data has good quality and availability; after acquiring the data, perform necessary conversion and preprocessing according to the characteristics of the time series data to adapt to the subsequent processing flow.

9. The multi-source heterogeneous data aggregation method according to claim 6, characterized in that: In step three, for other data sources: data is acquired through customized API interfaces. These API interfaces are specially designed and developed according to the special needs and characteristics of the data sources. When acquiring data, the stability of the interface and the accuracy of the data must be ensured, and the acquired data must be converted into the necessary format and processed initially so that it can be integrated with other types of data.

Images