Multi-source heterogeneous data processing method and system based on domestic intelligent chip

Abstract

The invention discloses a multi-source heterogeneous data processing method and system based on a domestic intelligent chip. The method comprises the steps of data collection, data preprocessing, intelligent analysis, data visualization and data storage and management. Compared with the prior art, the method has the advantages that the data collection capability is efficient; comprehensive data preprocessing is carried out; the intelligent analysis capability is strong; visual data visualization is realized; and safe data storage and management are realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and specifically to a multi-source heterogeneous data processing method and system based on domestic smart chips. Background Art

[0002] With the rapid development of information technology, data sources are becoming increasingly diverse, including databases, APIs, files, sensors and other types. These data are not only in different formats and complex structures, but also have a lot of noise, outliers and missing values. Traditional data processing methods are often unable to effectively cope with the challenges of such multi-source heterogeneous data, and there are problems such as low processing efficiency, low data quality, and poor adaptability of analysis models. At the same time, data security and privacy protection are increasingly becoming important links that cannot be ignored in the data processing process. Summary of the invention

[0003] The technical problems to be solved by the present invention are: 1) Low data collection efficiency: traditional methods are difficult to effectively integrate multiple data sources, and the data collection process is cumbersome and error-prone; 2) Insufficient data preprocessing: The cleaning, conversion and integration of raw data are not thorough enough, which affects the accuracy of subsequent analysis results; 3) Limited intelligent analysis capabilities: Existing analysis models are difficult to adapt to complex and changeable data environments, and it is difficult to discover hidden rules and trends in data; 4) Poor data visualization: The data presentation method is single, the user interaction experience is poor, and it is difficult to intuitively understand the meaning behind the data; 5) Insecure data storage and management: There are security risks in the storage and transmission of data, and there is a lack of effective data encryption and access control mechanisms.

[0004] In order to solve the above technical problems, the technical solution provided by the present invention is: a multi-source heterogeneous data processing method based on domestic smart chips, comprising the following steps:

[0005] S1. Data collection: First, clarify the purpose and requirements of data collection, determine what types of data need to be collected, and identify and determine the data sources, which include databases, APIs, files, and sensors; use SQL queries, API calls, and crawler plug-ins to collect data from various data sources;

[0006] S2. Data preprocessing: cleaning the collected data, including processing missing values, outliers, duplicate values, and correcting errors in the data. Converting data from one format or structure to another, including data type conversion, unit conversion, and data format adjustment, integrating data from different data sources, resolving data conflicts, and building a unified data view. Reducing the dimension and complexity of the data through feature selection and principal component analysis can improve the efficiency of subsequent processing.

[0007] S3, intelligent analysis, select the appropriate analysis model according to business needs and data characteristics, use the cleaned and integrated data to train the model, adjust the model parameters to optimize performance, use the trained model to conduct in-depth analysis of the pre-processed data, automatically discover the hidden rules, trends and associations in the data, evaluate the analysis results, and ensure the accuracy and reliability of the results;

[0008] S4. Data visualization: Use front-end technology or visualization libraries to build data visualization interfaces, present analysis results in the form of charts, reports, etc., including bar charts, line charts, and pie charts, and design user interaction functions so that users can easily view, filter, sort, and export data visualization results;

[0009] S5. Data storage and management: Store the processed data in designated targets, including data warehouses and cloud storage, provide data encryption and access control functions to ensure data security and privacy, regularly update data sources, and maintain and optimize stored data to ensure data accuracy and timeliness.

[0010] Correspondingly, this method is implemented with a supporting system, specifically a multi-source heterogeneous data processing system based on domestic smart chips, including a data acquisition and access module based on the system architecture, a data preprocessing module, an intelligent processing and analysis module, and a data output and management module. The data acquisition and access module includes a data source adaptation module and a data synchronization module. By defining a data source adapter, it realizes access and adaptation to different data sources to ensure smooth data collection. It supports both scheduled synchronization and real-time synchronization, and selects appropriate data synchronization strategies according to business needs. The data preprocessing module includes a data cleaning and conversion module and a data integration and standardization module. It ensures data quality by removing noise, outliers and redundant information in the data. The cleaned data is converted into a format suitable for subsequent processing. By defining data association rules, data from different sources are associated and integrated to build a complete data view; the intelligent processing and analysis module includes an intelligent analysis engine module and a heterogeneous data fusion module, which uses intelligent algorithms to conduct in-depth analysis of the pre-processed data, discover hidden rules, trends and associations in the data, and build analysis models according to business needs to achieve automated analysis and prediction of data; the data output and management module includes a data visualization module and a data storage and management module, which displays the analysis results to users in the form of charts, reports, etc., to help users quickly understand the meaning behind the data, store the processed data in the specified target, and provide data security management functions.

[0011] Compared with the prior art, the advantages of the present invention are: 1) Efficient data collection capability: by defining data source adapters and flexible ETL mechanisms, rapid access and preliminary processing of multi-source heterogeneous data can be achieved, thereby improving data collection efficiency; 2) Comprehensive data preprocessing: deep cleaning, conversion and integration of data to ensure data quality and lay a solid foundation for subsequent analysis; 3) Powerful intelligent analysis capability: based on domestically produced smart chips, integrating multiple intelligent algorithms to improve the adaptability and accuracy of analysis models and automatically discover hidden information in data; 4) Intuitive data visualization: building a user-friendly data visualization interface, providing rich charts and report forms, helping users quickly understand the meaning behind the data; 5) Secure data storage and management: providing security management functions such as data encryption and access control to ensure the security and privacy of data during storage and transmission.

[0012] Furthermore, the data source adaptation module is used to adapt and access data from different data sources, including relational databases, non-relational databases, file systems, real-time streaming data, Web data, API data, sensor data and social media data, and achieve initial data access and format unification through a flexible ETL mechanism.

[0013] Furthermore, the data synchronization module is responsible for data synchronization processing, and supports both scheduled synchronization and real-time synchronization. According to business needs, this module can ensure the accurate and timely transmission of data from the source to the target, and support the periodic writing of incremental data and full data.

[0014] Furthermore, the data cleaning and conversion module is responsible for dealing with noise, outliers, and missing value problems in the original data, as well as performing necessary conversions on the data according to business needs.

[0015] Furthermore, the data integration and standardization module is responsible for integrating data from different data sources and unifying them into a standardized data model.

[0016] Furthermore, the intelligent analysis engine module is based on domestically produced intelligent chips and integrates a variety of intelligent algorithms, including machine learning and deep learning, to conduct in-depth analysis of pre-processed data.

[0017] Furthermore, the heterogeneous data fusion module is used to deal with the fusion problem of multi-source heterogeneous data. By building a unified data model and fusion strategy, it realizes the seamless fusion of data of different structures and types, generates fused data results, and provides a basis for subsequent data applications.

[0018] Furthermore, the data visualization module can convert complex data analysis results into charts and reports by building a data visualization interface, helping users quickly understand the meaning behind the data.

[0019] Furthermore, the data storage and management module stores the processed data in designated targets, including data warehouses and cloud storage, and also provides data security management functions, including data encryption and access control, to ensure data security and privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is an architectural diagram of a multi-source heterogeneous data processing system based on domestic smart chips.

[0021] Figure 2 It is a schematic diagram of the architecture of the data collection and access module.

[0022] Figure 3 It is a schematic diagram of the architecture of the data preprocessing module.

[0023] Figure 4 It is a schematic diagram of the architecture of the intelligent processing and analysis module.

[0024] Figure 5 It is a schematic diagram of the architecture of the data output and management module. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0026] Implementation

[0027] like Figure 1 As shown in the figure, the specific workflow of this system is as follows:

[0028] 1. Data Collection

[0029] (1) Goal determination: First, clarify the purpose and requirements of data collection and determine what types of data need to be collected.

[0030] (2) Data source identification: Identify and determine the data source, which may include databases, APIs, files, sensors, etc.

[0031] (3) Data collection: Use appropriate technologies and methods (such as SQL queries, API calls, crawlers, etc.) to collect data from various data sources.

[0032] 2. Data Preprocessing

[0033] (1) Data cleaning: Clean the collected data, including processing missing values, outliers, duplicate values, and correcting errors in the data.

[0034] (2) Data conversion: Converting data from one format or structure to another format or structure as needed, including data type conversion, unit conversion, data format adjustment, etc.

[0035] (3) Data integration: Integrate data from different data sources, resolve data conflicts, and build a unified data view.

[0036] (4) Data reduction: Reduce the dimension and complexity of data through methods such as feature selection and principal component analysis, and improve the efficiency of subsequent processing.

[0037] 3. Intelligent Analysis

[0038] (1) Data modeling: Select appropriate analysis models based on business requirements and data characteristics, such as statistical models, machine learning models, etc.

[0039] (2) Model training: Use the cleaned and integrated data to train the model and adjust the model parameters to optimize performance.

[0040] (3) Automatic discovery: Use the trained model to conduct in-depth analysis of preprocessed data and automatically discover hidden patterns, trends, and associations in the data.

[0041] (4) Result evaluation: Evaluate the analysis results to ensure their accuracy and reliability.

[0042] 4. Data Visualization

[0043] (1) Interface construction: Use front-end technology or visualization library to build data visualization interface.

[0044] (2) Chart generation: The analysis results are presented in the form of charts, reports, etc., such as bar charts, line charts, pie charts, etc.

[0045] (3) Interaction design: Design user interaction functions to enable users to easily view, filter, sort, and export data visualization results.

[0046] 5. Data Storage and Management

[0047] (1) Data storage: Store the processed data in a specified target, such as a data warehouse, cloud storage, etc.

[0048] (2) Security management: Provides security management functions such as data encryption and access control to ensure data security and privacy.

[0049] (3) Data maintenance: Regularly update data sources, and maintain and optimize stored data to ensure data accuracy and timeliness.

[0050] Embodiment 2

[0051] Based on the first embodiment, Figures 1 to 5As shown in the figure, the data collection and access module adopts a modular design, which is easy to expand and customize according to actual needs. It supports access to multiple data sources, including relational databases, non-relational databases, file systems, real-time streaming data, etc. It obtains data from the data source through API interfaces or message queues, and converts the collected data into a unified format within the system for subsequent processing.

[0052] The data preprocessing process of the data preprocessing module is carried out according to the established process, including data cleaning, conversion, association and integration. Through automated tools and algorithms, manual intervention is reduced and processing efficiency is improved. Data interaction is carried out between the steps within the preprocessing module through function calls or data flow transmission, and the preprocessed data is output to the intelligent processing and analysis module through the interface.

[0053] The intelligent processing and analysis module integrates a variety of intelligent algorithms, such as machine learning and deep learning, for data analysis. Based on domestic intelligent chips, it provides efficient data processing capabilities. It receives the data output by the preprocessing module, calls the integrated intelligent algorithm during the intelligent processing process, and outputs the analysis results to the data output and management module through the interface.

[0054] The data output and management module provides an intuitive and easy-to-understand data visualization interface and supports a variety of data storage methods, such as data warehouses, cloud storage, etc. Receive the analysis results output by the intelligent processing and analysis module, display the analysis results to the user through the data visualization interface, and write the data to be saved to the specified storage medium.

[0055] Embodiment 3

[0056] Based on the first embodiment, Figure 1 and Figure 2As shown in the figure, the data source adaptation module realizes the preliminary access and format unification of data through the ETL mechanism. The algorithm steps include: 1) Define data source connection information: Identify the data source type (such as relational database, non-relational database, file, etc.), and configure the data source connection parameters (such as URL, username, password, port, etc.). 2) Extraction: Write or select the corresponding data extraction script / API call according to the data source type. Use SQL query (for relational database), NoSQL query (for non-relational database), file reading (for file system, etc.) to extract data. The extracted data can be full data or incremental data specified according to business needs. 3) Conversion: Design data conversion rules, including data type conversion, data cleaning (removing invalid values, outliers), data format adjustment, etc. Apply conversion rules to the extracted data to generate intermediate data formats. The conversion process may include data aggregation, splitting, association and other operations to meet the needs of subsequent processing or storage. 4) Loading: Load the converted data into the temporary storage area or data warehouse inside the system. The loading process may involve security measures such as data compression and encryption. After loading is completed, perform preliminary data verification to ensure the integrity and accuracy of the data.

[0057] The data synchronization module ensures accurate and timely data transmission and supports incremental and full writing. Its algorithm steps include: 1) Configure synchronization tasks: define the connection information of the source and target ends. Set the synchronization type (full synchronization, incremental synchronization). Configure the synchronization period (such as daily, hourly, etc.). 2) Incremental data identification: For incremental synchronization, it is necessary to define the incremental data identification mechanism. Common incremental identification methods include timestamp comparison, log analysis, change data capture (CDC), etc. 3) Data extraction and conversion: According to the synchronization type (full or incremental), perform the corresponding data extraction operation. Perform necessary conversion on the extracted data to meet the format requirements of the target end. 4) Data transmission: Use secure and reliable data transmission protocols (such as HTTPS, FTPS) to transfer data to the target end. Monitor the data transmission process to ensure the integrity and security of the data. 5) Data loading and verification: Load the data transmitted to the target end into the specified storage location. Perform data verification operations to ensure the accuracy and consistency of the data. 6) Logging and exception handling: Record key information during the synchronization process, including success, failure, exceptions, etc. Handle abnormal situations, such as retry mechanism, alarm notification, etc.

[0058] Embodiment 4

[0059] Based on the first embodiment, Figure 1 and 3As shown, the algorithm steps of the data cleaning and conversion module include: 1) Data cleaning: Identify and process missing values ​​(such as filling, deleting or interpolating). Identify and remove outliers (based on statistical methods, domain knowledge or business rules). Correct errors in the data (such as spelling errors, format errors, etc.) and remove duplicate records. 2) Data conversion: data type conversion (such as converting strings to numeric types). Data format adjustment (such as date format unification, text encoding conversion). Data aggregation and splitting (aggregate or decompose data according to business needs). Data normalization / standardization (such as scaling numerical data to a specific range or distribution).

[0060] The algorithm steps of the data integration and standardization module include: 1) Data integration: Identify and associate relevant data from different data sources. Resolve data conflicts (such as different attribute values ​​for the same entity in different data sources). Build a data warehouse or data lake to support complex data query and analysis. 2) Data standardization: Define a unified data model, including data table structure, field naming rules, data types, etc. Map the cleaned and converted data to a unified data model. Implement data quality checks to ensure data accuracy and consistency.

[0061] Embodiment 5

[0062] Based on the first embodiment, Figure 1 and 4 As shown in the figure, the analysis process of the intelligent analysis engine module extracts or creates meaningful features from the preprocessed data, which can reflect the essential attributes and potential associations of the data. It involves steps such as data cleaning, dimensionality reduction, and feature selection to improve the accuracy and efficiency of subsequent analysis. Select appropriate machine learning or deep learning algorithms according to the analysis objectives and data types. Use the preprocessed data to train the model, and improve the model's prediction and generalization capabilities by adjusting parameters and optimizing algorithms. Use the trained model to perform pattern recognition on the data to discover the laws and trends hidden in the data. Through cluster analysis, association rule mining, time series analysis and other methods, reveal the correlation and dynamic changes between data. Organize the analysis results into easy-to-understand reports, including data visualization charts, trend analysis, correlation analysis and other contents. The report should highlight key findings and provide data-supported decision-making recommendations.

[0063] The algorithm steps of the heterogeneous data fusion module include: 1) Data integration: Use data integration technology (such as ETL, ELT, etc.) to integrate and unify data from different data sources. Clean and convert data to eliminate data quality issues and achieve data compatibility and availability. 2) Data mapping and conversion: Design data mapping rules to map data of different structures and types into a unified data model. Use data conversion technology (such as data format conversion, data type conversion, data unit conversion, etc.) to convert the mapped data to ensure data consistency and accuracy. 3) Data fusion: Merge and integrate the converted data according to a certain fusion strategy to generate fused data results. Fusion strategies may include weighted average, data fusion algorithms (such as Kalman filtering, particle filtering, etc.), etc. The specific choice depends on the characteristics of the data and analysis requirements. The following is a partial code demonstration:

[0064] import pandas as pd

[0065] #Assume that df1 and df2 are datasets from different data sources with different structures and types

[0066] #Here we use the pandas library to simulate the data fusion process

[0067] #Sample data: df1 (structured data)

[0068] data1={'ID':[1,2,3],'Value1':[10,20,30]}

[0069] df1 = pd.DataFrame(data1)

[0070] #Sample data: df2 (semi-structured data, converted to structured form)

[0071] data2=[{'ID':4,'Value2':40},{'ID':5,'Value2':50}]

[0072] df2 = pd.DataFrame(data2)

[0073] #Data fusion: Merge based on ID field

[0074] df_merged=pd.merge(df1,df2,on='ID',how='outer')

[0075] #If some fields need mathematical operations or conversions, they can be performed in this step

[0076] #For example, calculate the average of the Value field in two data sets

[0077] df_merged['Average_Value']=(df_merged['Value1']+df_merged['Value2']) / 2

[0078] # Output the fused data results

[0079] print(df_merged)

[0080] Embodiment 6

[0081] Based on the first embodiment, Figure 1 and 5 As shown in the figure, the algorithm steps of the data visualization module include: 1) Selecting the chart type: Select the appropriate chart type according to the purpose of data analysis and the characteristics of the results. 2) Data mapping: Map the analysis results to the various elements of the chart, such as the X-axis, Y-axis, color, size, etc. 3) Interface construction: Use front-end technology (such as HTML / CSS / JavaScript) or visualization libraries (such as D3.js, ECharts, Highcharts, etc.) to build a visualization interface. The following is a partial code display:

[0082]

[0083]

[0084] The algorithm steps of the data storage and management module include: 1) Data storage: select a suitable storage system (such as relational database, NoSQL database, distributed file system, etc.) according to the needs, and design the data storage structure. 2) Data encryption: use encryption algorithms (such as AES, RSA, etc.) to encrypt sensitive data. 3) Access control: implement role-based access control (RBAC), attribute-based access control (ABAC) and other strategies to ensure that only authorized users can access the data. The following is a partial code demonstration:

[0085] --Suppose there is a table called Sales, which contains fields such as ID, Product, Quantity, Amount, etc.

[0086] --Insert a sales record

[0087] INSERT INTO Sales(Product,Quantity,Amount)

[0088] VALUES('shirt',5,100);

[0089] --Note: In actual applications, you may also need to encrypt and store sensitive fields (such as Amount)

[0090] --Encrypted storage is usually not handled directly at the database level, but is encrypted at the application level before storage

[0091] --Encryption example (pseudocode)

[0092] encryptedAmount = encrypt(100, secretKey); -- Assume encrypt is an encryption function

[0093] --Then insert the encryptedAmount as the value of Amount into the Sales table

[0094] The above shows and describes the basic principles and main features of the present invention and the advantages of the invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-source heterogeneous data processing method based on domestic smart chips, characterized in that The following steps are involved: S1. Data collection: First, clarify the purpose and needs of data collection, determine what types of data need to be collected, and identify and determine the data sources, which include databases, APIs, files, and sensors; Use SQL queries, API calls, and crawler plugins to collect data from various data sources; S2. Data preprocessing: cleaning the collected data, including processing missing values, outliers, duplicate values, and correcting errors in the data. Converting data from one format or structure to another, including data type conversion, unit conversion, and data format adjustment, integrating data from different data sources, resolving data conflicts, and building a unified data view. Reducing the dimension and complexity of the data through feature selection and principal component analysis can improve the efficiency of subsequent processing. S3, intelligent analysis, select the appropriate analysis model according to business needs and data characteristics, use the cleaned and integrated data to train the model, adjust the model parameters to optimize performance, use the trained model to conduct in-depth analysis of the pre-processed data, automatically discover the hidden rules, trends and associations in the data, evaluate the analysis results, and ensure the accuracy and reliability of the results; S4. Data visualization: Use front-end technology or visualization libraries to build data visualization interfaces, present analysis results in the form of charts, reports, etc., including bar charts, line charts, and pie charts, and design user interaction functions so that users can easily view, filter, sort, and export data visualization results; S5. Data storage and management: Store the processed data in designated targets, including data warehouses and cloud storage, provide data encryption and access control functions to ensure data security and privacy, regularly update data sources, and maintain and optimize stored data to ensure data accuracy and timeliness.

2. A multi-source heterogeneous data processing system based on domestic smart chips, including a data acquisition and access module, a data preprocessing module, an intelligent processing and analysis module, and a data output and management module based on the system architecture, characterized in that: The data collection and access module includes a data source adaptation module and a data synchronization module. By defining a data source adapter, it realizes access and adaptation to different data sources to ensure smooth data collection. It supports both scheduled synchronization and real-time synchronization, and selects appropriate data synchronization strategies according to business needs. The data preprocessing module includes a data cleaning and conversion module and a data integration and standardization module. It ensures data quality by removing noise, outliers and redundant information in the data, converts the cleaned data into a format suitable for subsequent processing, and associates and integrates data from different sources by defining data association rules to build a complete data view. The intelligent processing and analysis module includes an intelligent analysis engine module and a heterogeneous data fusion module, which uses intelligent algorithms to conduct in-depth analysis of pre-processed data, discover hidden rules, trends and associations in the data, build analysis models according to business needs, and realize automatic analysis and prediction of data; the data output and management module includes a data visualization module and a data storage and management module, which displays the analysis results to users in the form of charts, reports, etc., to help users quickly understand the meaning behind the data, store the processed data in the specified target, and provide data security management functions.

3. According to claim 2, a multi-source heterogeneous data processing system based on domestic smart chips is characterized by: The data source adaptation module is used to adapt and access data from different data sources, including relational databases, non-relational databases, file systems, real-time streaming data, Web data, API data, sensor data and social media data, and achieves preliminary data access and format unification through a flexible ETL mechanism.

4. According to claim 2, a multi-source heterogeneous data processing system based on domestic smart chips is characterized by: The data synchronization module is responsible for data synchronization processing and supports both scheduled synchronization and real-time synchronization. According to business needs, the module can ensure accurate and timely transmission of data from the source to the target, and support periodic writing of incremental data and full data.

5. According to claim 2, a multi-source heterogeneous data processing system based on domestic smart chips is characterized by: The data cleaning and conversion module is responsible for dealing with noise, outliers, and missing value problems in the original data, and performing necessary conversions on the data according to business needs.

6. According to claim 2, a multi-source heterogeneous data processing system based on domestic smart chips is characterized by: The data integration and standardization module is responsible for integrating data from different data sources and unifying them into a standardized data model.

7. According to claim 2, a multi-source heterogeneous data processing system based on domestic smart chips is characterized by: The intelligent analysis engine module is based on domestically produced intelligent chips and integrates multiple intelligent algorithms, including machine learning and deep learning, to conduct in-depth analysis of pre-processed data.

8. According to claim 2, a multi-source heterogeneous data processing system based on domestic smart chips is characterized by: The heterogeneous data fusion module is used to handle the fusion problem of multi-source heterogeneous data. By building a unified data model and fusion strategy, it realizes the seamless fusion of data of different structures and types, generates fused data results, and provides a basis for subsequent data applications.

9. According to claim 2, a multi-source heterogeneous data processing system based on domestic smart chips is characterized by: The data visualization module constructs a data visualization interface, which can convert complex data analysis results into charts and reports to help users quickly understand the meaning behind the data.

10. The multi-source heterogeneous data processing system based on domestic smart chips according to claim 2 is characterized in that: The data storage and management module stores the processed data in a designated target, including a data warehouse and cloud storage, and also provides data security management functions, including data encryption and access control, to ensure data security and privacy.

Images