Multi-data-source data processing method and system

Abstract

The invention relates to a multi-data-source data processing method and system, and relates to the technical field of data processing. During data collection, data sources are determined according to business targets, internal and external databases are integrated, quality is evaluated, and connection is established. The data processing includes extraction, cleaning, conversion, fusion and storage. The cleaning adopts an algorithm to optimize a missing value to approach a real value; the conversion meets business requirements; and fusing to form a unified data set. According to data verification, multi-aspect characteristics are verified by setting a difference range and utilizing optimized data. The system has a powerful monitoring optimization function, and optimized data are used for real-time monitoring adjustment, early warning and decision support. According to the method, the traditional problems are effectively solved, the data processing efficiency, quality and reliability are improved, complex business requirements are met, accurate data positioning, stable connection, high-quality processing and reliable verification can be achieved, and powerful guarantee is provided for cross-data-source data analysis and decision making.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for processing data from multiple data sources. Background Art

[0002] With the rapid development of information technology, enterprises and organizations need to process data from multiple different data sources in their daily operations. These data sources may include various internal business systems, such as enterprise resource planning (ERP) systems, customer relationship management (CRM) systems, etc., as well as external public data resources, data provided by partners, etc. However, data processing from multiple data sources faces many challenges.

[0003] Traditional data processing methods often have the following problems when facing multiple data sources: It is difficult to determine the data source, and it is difficult to accurately match business needs to achieve comprehensive collection; the connection is complex and unstable, and the differences in different data sources make the connection difficult and prone to failure; data quality is difficult to ensure, and there is a lack of effective processing methods in terms of accuracy, completeness, consistency and timeliness; fourth, data processing efficiency is low, and the tools and algorithms in each link are backward and cannot respond quickly to business changes. These problems make the existing technology unable to meet the needs of increasingly complex business and data environments.

[0004] In summary, the existing multi-data source data processing technology cannot meet the increasingly complex business needs and data environment, and an innovative multi-data source data processing method and system are urgently needed to solve these problems. Summary of the invention

[0005] The multi-data source data processing system aims to efficiently integrate, process and verify data from different data sources to meet the needs of various business analysis and decision-making. The system covers the entire process from data collection, processing to verification to ensure the quality, consistency and timeliness of the data. In view of the shortcomings of the prior art, the technical solution adopted by the present invention to solve its technical problems is: a multi-data source data processing system, including: The data collection module determines the data source, identifies the required data type based on the core business objectives and key indicators, identifies the key factors that affect the achievement of business objectives, and then defines the data indicators. Inventory the internal data sources, sort out the existing information systems and storage locations, and establish an internal database; at the same time, explore external data sources, retrieve industry-related public data resources, establish an external database, and integrate the two into a comprehensive database. Verify data accuracy, select samples and compare them with reference data to clarify errors; evaluate data integrity, analyze sample data missing and completeness; analyze data timeliness, and establish a historical span database that meets update frequency and time delay requirements.

[0006] Establish connections, analyze data source structures, understand the data storage architecture, format, and organization of internal and external databases, and determine the data parts that need to be focused on and connected first. According to the type and characteristics of the data source, select appropriate connection technologies and tools, such as database connection interfaces, API calls, or data integration platforms, and configure the corresponding connection parameters. Perform connection tests to verify data accuracy and completeness, check connection stability and data transmission fluency, and establish connection monitoring to monitor connection status and data quality indicators in real time and set warning thresholds.

[0007] Data processing module, data extraction, determine the extraction rules according to business needs, clarify the extracted data fields, records and time range, select appropriate data extraction tools, extract data from the data source and temporarily store it in the intermediate storage area. Data cleaning, remove duplicate data, analyze the distribution of missing values, and use innovative algorithms to optimize the missing values. By building an objective function and updating the weight coefficient using the gradient descent method, the optimized data is infinitely close to the true value. Data conversion, according to data analysis and storage needs, convert data types and specific character encoding formats, and calculate and derive new fields based on the original data. Data fusion, merge data with related relationships, establish data associations through common fields, obtain a complete data view, and integrate the merged and associated data to eliminate inconsistencies and conflicting data to form a unified data set. Data storage, select storage media and database systems based on factors such as data volume, access frequency and data type, design a suitable data storage structure, and load the processed and fused data into the database using batch loading or real-time loading.

[0008] Data verification module sets a reasonable range of differences. According to business characteristics, comprehensive databases, and historical span databases, set maximum and minimum thresholds to determine the reasonable range of differences. Accuracy verification uses optimized data to evaluate the accuracy of data after processing. When it is found that the difference between the estimated value and the reference value exceeds the reasonable range, locate the area where data errors may exist. Integrity verification uses optimized data to fill missing values, count the number of statistical records and check the filling of required fields, analyze the data missing patterns and root causes, and ensure the integrity of the data in structure and content, especially in the data fusion scenario of multiple data sources, to ensure the integrity of the fused data. Consistency verification uses optimized data to coordinate and unify the data differences between different data sources, adjust the differences of the same entity in different data sources according to business rules, and improve the consistency level of data. Timeliness verification plays the role of optimized data in real-time data processing and updating, ensures the temporal continuity and timeliness of data, and ensures the integrity and availability of historical data in the historical data analysis scenario.

[0009] System monitoring and optimization module, monitoring indicator system, incorporating optimized data into the monitoring indicator system, real-time monitoring of the difference between actual data and optimized data, so as to timely discover problems and adjust data processing algorithms and parameters. Real-time monitoring function, using optimized data to monitor data consistency and synchronization between different data sources, quickly identify data differences and synchronization problems, and ensure data accuracy. Early warning capability, when the difference between optimized data and actual data exceeds the preset threshold or the number of missing values ​​of a data source suddenly increases, the system can quickly issue an alarm to achieve early detection and resolution of potential problems. Decision support, combining optimized data with real-time data, providing real-time information support for data analysis and decision-making, insight into trends and changes in the data processing process, realizing data analysis and decision-making across data sources, and dynamically optimizing the data processing process based on monitoring data and optimized data.

[0010] A multi-data source data processing method is used to process a multi-data source data processing system, comprising the following steps: Data collection: identify data sources and establish connections; Data processing: extracting, cleaning, converting, fusing and storing the data collected from the data center; Data Verification: Perform quality verification on processed data and monitor the data processing process in real time.

[0011] In the data collection, the method for determining the data source is as follows: Determine data requirements, determine the required data types based on business core goals and key indicators, find out the key factors that affect the realization of business goals, and define data indicators based on business goals and key factors; Identify potential data sources. First, conduct an inventory of internal data sources, sort out existing internal information systems and storage locations, and establish an internal database. Then explore external data sources, retrieve industry-related public data resources, establish an external database, and integrate the internal and external databases to obtain a comprehensive database. To evaluate the quality of data sources, first verify the data accuracy, select a part of data samples, compare and verify with the reference data in the known comprehensive database, clarify the data errors in the data source, and obtain error information. The error information is used as a reference for evaluating data integrity. Then evaluate the data integrity, analyze the missing data of the sample data, and calculate the proportion of missing values ​​of each sample data to understand the completeness of each sample data in different data sources. Finally, analyze the timeliness of the data and establish a historical span database that meets the update frequency and time delay according to actual needs; The method of establishing a connection includes the following steps: Analyze the data source structure, the data storage architecture, data format and data organization of the internal and external databases. For the internal database, understand the data storage logic of each information system, including the table structure, field type and association relationship. For the external database, analyze the file format of the public data resources and the hierarchical structure of the data. According to the error information, missing values ​​and data timeliness characteristics obtained in the data source quality assessment stage, determine which data parts need to be focused on and the priority to establish connections; Select connection technology. Select connection technology and tools according to the type and characteristics of the data source. Connection methods include database connection interface, API call and data integration platform. For internal database, if it is a homogeneous database system, you can directly use the connection components that come with the database for efficient connection. If it is a heterogeneous database, you need to use middleware. For external database, configure the corresponding connection parameters and authentication information according to the access method provided by it. Configure connection parameters. For the selected connection technology, configure detailed connection parameters. For database connection, set the database server address, port number, user name, password, and the specific database name to be connected. For API connection, fill in the API key, requested URL address, and request parameter format. According to the data timeliness requirements, set the connection update frequency parameters; Test the connection. After configuring the connection parameters, perform a connection test, select data samples, read data from the data source and verify the accuracy and completeness of the data, check whether the connection is stable and whether the data transmission is smooth, obtain the test results, and compare the test results with the data characteristics of the previous stage of evaluating the quality of the data source; Establish connection monitoring, monitor the connection status, data flow and data quality indicators in real time, set warning thresholds, and issue an alarm when the connection is abnormal or the data quality drops to the warning threshold, notify relevant personnel to handle it, and regularly maintain and optimize the connection, and adjust the connection parameters according to changes in business needs and updates to data sources.

[0012] The data processing comprises the following steps: Data extraction: determine the extraction rules. According to the core business objectives and key indicator requirements, clarify which data fields and data records to extract from the data source and the time range of extraction. According to the type and connection method of the data source, select the data extraction tool. According to the extraction rules and data extraction tools, extract the required data from the data source and temporarily store it in an intermediate storage area for subsequent processing; Data cleaning, removing duplicate data, identifying and deleting duplicate data records by comparing key features of the data, and then analyzing the distribution of missing values ​​in the data. For missing values ​​in important fields, the following methods can be used: First, define Represents sample points The probability of missing values, represents the sum of the probabilities of missing values ​​of all sample points, Represents the total number of missing sample points, then:

[0013] in, is an indicator function, when When there is value, ,when When missing, ; set up They represent the weight coefficients of the three methods: interpolation, model-based prediction, and marking as unknown, and , represents the estimated value of the missing value obtained by interpolation, represents the estimate of the missing value obtained through model-based prediction, Indicates the characteristic tag value when the tag is unknown, Represents optimized data, for sample points The total estimated value of , the specific calculation formula is as follows:

[0014] Construct an objective function based on all missing sample points , , here It represents the actual value when it has a value, and is equivalent to an abstract representation of the true value to be estimated during the optimization process when it is missing. right To find partial derivatives, , denoted as ; right To find partial derivatives, , denoted as ; right To find partial derivatives, , denoted as ; Then update the weight coefficients according to the update rule of the gradient descent method: Suppose a learning rate is , in each iteration, the update formula of the weight coefficient is as follows: ; ; ; Repeat the above steps until the objective function Converge and get optimized data, which is infinitely close to the true value; The data processing further comprises the following steps: Data conversion: convert the data type according to the needs of data analysis and storage; Data conversion: convert data in some specific character encoding formats into a unified encoding format; Data calculation and derivation: Calculate and derive new fields based on the original data according to business needs.

[0015] Data fusion combines data from different data sources but with related relationships. Data association establishes the relationship between data and associates data in different data tables through common fields to obtain a complete data view. Data integration: Integrate the merged and associated data to eliminate inconsistencies and conflicting data to obtain a unified and consistent data set; Data storage, select storage media and database, select storage media and database system based on data volume, access frequency, and data type factors; Design data storage structure: Design database table structure, data warehouse dimensional model, and NoSQL database document structure based on data characteristics and analysis requirements; Data loading: Load the processed and integrated data into the database according to the designed storage structure, which can be done in batch loading or real-time loading.

[0016] The data verification comprises the following steps: Set a reasonable range of differences: Set the maximum and minimum thresholds based on business characteristics, comprehensive databases, and historical span databases. The reasonable range of differences is between the maximum and minimum thresholds. Accuracy verification: The optimization data assumes the function of accurately evaluating the accuracy status of data after processing. When it is found that the difference between the estimated value and the reference value exceeds the reasonable range, it quickly guides relevant personnel to locate the area where data errors may exist and mark it as a potential grid. The specific marking method is as follows: Optimize the data to calculate various statistical indicators based on it in the mathematical operation verification phase of the data, and compare them with the expected indicators based on accurate data to verify the rationality of the data in mathematical operation logic. If significant differences are found, it means that there may be errors or outliers in the data; Integrity verification: Optimized data has the function of filling in missing data values. By participating in statistical data record quantity and checking the filling status of required fields, combined with its reasonable inference ability, it can more accurately judge the integrity status of data. At the same time, based on its distribution law analysis function, it can also deeply analyze the patterns and root causes of missing data, provide key decision-making basis for improving data collection and processing processes, and ensure the integrity of data in structure and content.

[0017] In the scenario of multi-data source data fusion, data optimization plays a key role in ensuring the integrity of the fused data. When data from different data sources are integrated, it can reasonably infer and supplement the missing data in a data source based on the relevant clues and reference information of other data sources, so that the overall fused data remains consistent and complete between various data sources, meeting the strict requirements of complex business analysis for data integrity.

[0018] Consistency verification: Optimize data to coordinate and unify the data inconsistency between different data sources. According to the data context and business rules, adjust the differences of the same entity in different data sources, including attribute value differences and missing differences, to achieve a higher level of consistency between different data sources, and provide a reliable data foundation for cross-data source data analysis and business decision-making; For consistency verification of time series data in data processing from multiple data sources, data optimization plays the role of filling in missing time point data, so that time series data from different data sources can be aligned and continuous in terms of time nodes and data values, thereby improving the accuracy of consistency verification, ensuring the consistency and availability of time series data in a multi-data source environment, and providing strong support for business analysis based on the time dimension.

[0019] Timeliness verification: Optimize data to ensure the time continuity and timeliness of data during real-time processing and updating. When data from a certain data source is missing or delayed, the gaps can be filled in time to keep the data coherent in the time dimension, meeting the strict requirements of real-time analysis and decision-making on data timeliness.

[0020] In the historical data analysis scenario, accurate optimization data plays the role of ensuring the integrity and availability of historical data. By reasonably supplementing the missing values ​​in historical data, the time span of historical data is made more complete, providing a reliable data foundation for accurate trend analysis and prediction, improving the reliability of timeliness verification in historical data, and ensuring that business decisions based on historical data are more accurate and forward-looking.

[0021] The optimized data is included in the monitoring indicator system as a key indicator, and is used to measure the quality and accuracy of missing value estimation during data processing. By real-time monitoring of the difference between actual data and optimized data, problems in the data processing process can be discovered in a timely manner, and according to the size and trend of the difference, relevant personnel can be guided to adjust the data processing algorithm and parameters to optimize the data processing effect.

[0022] In the real-time monitoring of data processing from multiple data sources, optimized data plays a role in monitoring the consistency and synchronization of data between different data sources. By comparing the data features of the same data points in different data sources, data differences and synchronization issues between data sources can be quickly identified, ensuring the consistency and accuracy of data in a multi-data source environment, and providing a reliable data foundation for comprehensive data analysis and application.

[0023] Improved early warning capability: When the data quality deteriorates or anomalies occur, the optimized data serves as an important reference to trigger early warnings. If the difference between the optimized data and the actual data exceeds the preset threshold, or the number of missing values ​​in a data source suddenly increases, the system can quickly issue an alarm based on the changes and notify relevant personnel to handle the situation, thereby achieving early detection and resolution of potential problems in the data processing process, effectively avoiding the adverse impact of the expansion of problems on the business.

[0024] In the complex environment of multi-data source data processing, real-time monitoring of optimization data has the function of discovering the source of potential system failures or data quality problems. By monitoring the change trend and abnormal fluctuations of optimization data, problems can be quickly located, such as data acquisition equipment failure, network transmission problems or data processing algorithm defects, providing key clues for timely repair and optimization measures, and improving the stability and reliability of the system.

[0025] Decision support: In the real-time monitoring process, the optimization data is combined with the real-time generated data to provide real-time information support for data analysis and decision-making. Through real-time analysis of data sources, we can gain insight into trends and changes in the data processing process, timely discover abnormal situations and potential opportunities in the business, and provide timely and accurate basis for the adjustment and optimization of business decisions.

[0026] In real-time monitoring of data processing from multiple data sources, optimized data helps to achieve data analysis and decision-making across data sources. Through comprehensive analysis of optimized data, we can gain an in-depth understanding of the associations and changing trends between different market data, and provide comprehensive and accurate information support for key decisions such as risk assessment and market forecasting in complex business scenarios. At the same time, we can also dynamically optimize the data processing process based on real-time monitoring data and optimized data guidance, thereby improving the efficiency and quality of data processing.

[0027] The beneficial effects of the present invention are as follows: First, in the data collection stage, we accurately locate the data source based on the core business objectives and key indicators, and comprehensively integrate internal and external data resources to build a comprehensive database, laying a solid foundation for subsequent work.

[0028] Secondly, in terms of connection establishment, by analyzing the data source structure in detail, adapting the best connection technology and tools and optimizing parameters, and using connection testing and monitoring to ensure connection stability and smooth data transmission, the efficiency and reliability of data acquisition are greatly improved.

[0029] Furthermore, the quality of data processing has been greatly improved. When cleaning data, algorithms are used to optimize the processing of missing values ​​to make them closer to the true value; data conversion can flexibly meet various business needs; data fusion eliminates inconsistencies and forms a unified data set.

[0030] Then, data verification is reliable. Reasonably set the difference range, use optimized data to comprehensively verify the accuracy, completeness, consistency and timeliness of the data, quickly locate and solve problems, and provide accurate basis for decision-making.

[0031] Finally, the system monitoring and optimization functions are powerful. Incorporating optimized data into the monitoring system for real-time monitoring and adjustment to ensure data processing results; real-time monitoring of data consistency and synchronization to promptly resolve discrepancies; having early warning capabilities to prevent problems from expanding; combining optimized data with real-time data to support decision-making and dynamically optimize processing procedures to comprehensively improve data processing efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a system flow chart of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0034] Example 1

[0035] See also Figure 1 The present invention provides a technical solution: a method for processing data from multiple data sources, comprising the following steps: Data collection: identify data sources and establish connections; Data processing: extracting, cleaning, converting, fusing and storing the data collected from the data center; Data Verification: Perform quality verification on processed data and monitor the data processing process in real time.

[0036] During data collection, the method for determining the data source is as follows: Determine data requirements, determine the required data types based on business core goals and key indicators, find out the key factors that affect the realization of business goals, and define data indicators based on business goals and key factors; Identify potential data sources. First, conduct an inventory of internal data sources, sort out existing internal information systems and storage locations, and establish an internal database. Then explore external data sources, retrieve industry-related public data resources, establish an external database, and integrate the internal and external databases to obtain a comprehensive database. To evaluate the quality of data sources, first verify the data accuracy, select a part of data samples, compare and verify with the reference data in the known comprehensive database, clarify the data errors in the data source, and obtain error information. The error information is used as a reference for evaluating data integrity. Then evaluate the data integrity, analyze the missing data of the sample data, and calculate the proportion of missing values ​​of each sample data to understand the completeness of each sample data in different data sources. Finally, analyze the timeliness of the data and establish a historical span database that meets the update frequency and time delay according to actual needs; The method of establishing a connection includes the following steps: Analyze the data source structure, the data storage architecture, data format and data organization of the internal and external databases. For the internal database, understand the data storage logic of each information system, including the table structure, field type and association relationship. For the external database, analyze the file format of the public data resources and the hierarchical structure of the data. According to the error information, missing values ​​and data timeliness characteristics obtained in the data source quality assessment stage, determine which data parts need to be focused on and the priority to establish connections; Select connection technology. Select connection technology and tools according to the type and characteristics of the data source. Connection methods include database connection interface, API call and data integration platform. For internal database, if it is a homogeneous database system, you can directly use the connection components that come with the database for efficient connection. If it is a heterogeneous database, you need to use middleware. For external database, configure the corresponding connection parameters and authentication information according to the access method provided by it. Configure connection parameters. For the selected connection technology, configure detailed connection parameters. For database connection, set the database server address, port number, user name, password, and the specific database name to be connected. For API connection, fill in the API key, requested URL address, and request parameter format, etc. Set the connection update frequency parameters according to the data timeliness requirements; Test the connection. After configuring the connection parameters, perform a connection test, select data samples, read data from the data source and verify the accuracy and completeness of the data, check whether the connection is stable and whether the data transmission is smooth, obtain the test results, and compare the test results with the data characteristics of the previous stage of evaluating the quality of the data source; Establish connection monitoring, monitor the connection status, data flow and data quality indicators in real time, set warning thresholds, and issue an alarm when the connection is abnormal or the data quality drops to the warning threshold, notify relevant personnel to handle it, and regularly maintain and optimize the connection, and adjust the connection parameters according to changes in business needs and updates to data sources.

[0037] Data processing includes the following steps: Data extraction: determine the extraction rules. According to the core business objectives and key indicator requirements, clarify which data fields and data records to extract from the data source and the time range of extraction. According to the type and connection method of the data source, select the data extraction tool. According to the extraction rules and data extraction tools, extract the required data from the data source and temporarily store it in an intermediate storage area for subsequent processing; Data cleaning, removing duplicate data, identifying and deleting duplicate data records by comparing key features of the data, and then analyzing the distribution of missing values ​​in the data. For missing values ​​in important fields, the following methods can be used: First, define Represents sample points The probability of missing values, represents the sum of the probabilities of missing values ​​of all sample points, Represents the total number of missing sample points, then:

[0038] in, is an indicator function, when When there is value, ,when When missing, ; set up They represent the weight coefficients of the three methods: interpolation, model-based prediction, and marking as unknown, and , represents the estimated value of the missing value obtained by interpolation, represents the estimate of the missing value obtained through model-based prediction, Indicates the characteristic tag value when the tag is unknown, Represents optimized data, for sample points The total estimated value of , the specific calculation formula is as follows:

[0039] Construct an objective function based on all missing sample points , , here It represents the actual value when it has a value, and is equivalent to an abstract representation of the true value to be estimated during the optimization process when it is missing. right To find partial derivatives, , denoted as ; right To find partial derivatives, , denoted as ; right To find partial derivatives, , denoted as ; Then update the weight coefficients according to the update rule of the gradient descent method: Suppose a learning rate is , in each iteration, the update formula of the weight coefficient is as follows: ; ; ; Repeat the above steps until the objective function Converge and get optimized data, which is infinitely close to the true value; Data processing also includes the following steps: Data conversion: convert the data type according to the needs of data analysis and storage; Data conversion: convert data in some specific character encoding formats into a unified encoding format; Data calculation and derivation: Calculate and derive new fields based on the original data according to business needs.

[0040] Data fusion combines data from different data sources but with related relationships. Data association establishes the relationship between data and associates data in different data tables through common fields to obtain a complete data view. Data integration: Integrate the merged and associated data to eliminate inconsistencies and conflicting data to obtain a unified and consistent data set; Data storage, select storage media and database, select storage media and database system based on data volume, access frequency, and data type factors; Design data storage structure: Design database table structure, data warehouse dimensional model, and NoSQL database document structure based on data characteristics and analysis requirements; Data loading: Load the processed and integrated data into the database according to the designed storage structure, which can be done in batch loading or real-time loading.

[0041] Data validation includes the following steps: Set a reasonable range of differences: Set the maximum and minimum thresholds based on business characteristics, comprehensive databases, and historical span databases. The reasonable range of differences is between the maximum and minimum thresholds. Accuracy verification: The optimization data assumes the function of accurately evaluating the accuracy status of data after processing. When it is found that the difference between the estimated value and the reference value exceeds the reasonable range, it quickly guides relevant personnel to locate the area where data errors may exist and mark it as a potential grid. The specific marking method is as follows: Optimize the data to calculate various statistical indicators based on it in the mathematical operation verification phase of the data, and compare them with the expected indicators based on accurate data to verify the rationality of the data in mathematical operation logic. If significant differences are found, it means that there may be errors or outliers in the data; Integrity verification: Optimized data has the function of filling in missing data values. By participating in statistical data record quantity and checking the filling status of required fields, combined with its reasonable inference ability, it can more accurately judge the integrity status of data. At the same time, based on its distribution law analysis function, it can also deeply analyze the patterns and root causes of missing data, provide key decision-making basis for improving data collection and processing processes, and ensure the integrity of data in structure and content.

[0042] In the scenario of multi-data source data fusion, data optimization plays a key role in ensuring the integrity of the fused data. When data from different data sources are integrated, it can reasonably infer and supplement the missing data in a data source based on the relevant clues and reference information of other data sources, so that the overall fused data remains consistent and complete between various data sources, meeting the strict requirements of complex business analysis for data integrity.

[0043] Consistency verification: Optimize data to coordinate and unify the data inconsistency between different data sources. According to the data context and business rules, adjust the differences of the same entity in different data sources, including attribute value differences and missing differences, to achieve a higher level of consistency between different data sources, and provide a reliable data foundation for cross-data source data analysis and business decision-making; For consistency verification of time series data in data processing from multiple data sources, data optimization plays the role of filling in missing time point data, so that time series data from different data sources can be aligned and continuous in terms of time nodes and data values, thereby improving the accuracy of consistency verification, ensuring the consistency and availability of time series data in a multi-data source environment, and providing strong support for business analysis based on the time dimension.

[0044] Timeliness verification: Optimize data to ensure the time continuity and timeliness of data during real-time processing and updating. When data from a certain data source is missing or delayed, the gaps can be filled in time to keep the data coherent in the time dimension, meeting the strict requirements of real-time analysis and decision-making on data timeliness.

[0045] In the historical data analysis scenario, accurate optimization data plays the role of ensuring the integrity and availability of historical data. By reasonably supplementing the missing values ​​in historical data, the time span of historical data is made more complete, providing a reliable data foundation for accurate trend analysis and prediction, improving the reliability of timeliness verification in historical data, and ensuring that business decisions based on historical data are more accurate and forward-looking.

[0046] Optimized data is included in the monitoring indicator system as a key indicator, and it is responsible for measuring the quality and accuracy of missing value estimation during data processing. By real-time monitoring of the difference between actual data and optimized data, problems in the data processing process can be discovered in a timely manner, and according to the size and trend of the difference, relevant personnel can be guided to adjust the data processing algorithm and parameters to optimize the data processing effect.

[0047] In the real-time monitoring of data processing from multiple data sources, optimized data plays a role in monitoring the consistency and synchronization of data between different data sources. By comparing the data features of the same data points in different data sources, data differences and synchronization issues between data sources can be quickly identified, ensuring the consistency and accuracy of data in a multi-data source environment, and providing a reliable data foundation for comprehensive data analysis and application. Improved early warning capability: When the data quality deteriorates or anomalies occur, the optimized data serves as an important reference to trigger early warnings. If the difference between the optimized data and the actual data exceeds the preset threshold, or the number of missing values ​​in a data source suddenly increases, the system can quickly issue an alarm based on the changes and notify relevant personnel to handle the situation, thereby achieving early detection and resolution of potential problems in the data processing process, effectively avoiding the adverse impact of the expansion of problems on the business.

[0048] In a complex environment with multiple data sources, real-time monitoring of optimized data has the function of discovering the source of potential system failures or data quality problems. By monitoring the changing trends and abnormal fluctuations of optimized data, problems can be quickly located, such as data acquisition equipment failures, network transmission problems, or data processing algorithm defects, providing key clues for timely repair and optimization measures, and improving system stability and reliability. Decision support: In the real-time monitoring process, the optimization data is combined with the real-time generated data to provide real-time information support for data analysis and decision-making. Through real-time analysis of data sources, we can gain insight into trends and changes in the data processing process, timely discover abnormal situations and potential opportunities in the business, and provide timely and accurate basis for the adjustment and optimization of business decisions.

[0049] In real-time monitoring of data processing from multiple data sources, optimized data helps to achieve data analysis and decision-making across data sources. Through comprehensive analysis of optimized data, we can gain an in-depth understanding of the associations and changing trends between different market data, and provide comprehensive and accurate information support for key decisions such as risk assessment and market forecasting in complex business scenarios. At the same time, we can also dynamically optimize the data processing process based on real-time monitoring data and optimized data guidance, thereby improving the efficiency and quality of data processing.

[0050] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention are implemented according to the conventional means in the field unless otherwise specified and limited.

Claims

1. A multi-data source data processing system, characterized in that: include: Data collection module, which determines the data type according to the core business objectives and data indicators. The data collection module takes inventory of internal data sources, sorts out information systems and storage locations, and establishes an internal database; at the same time, it explores external data sources, retrieves industry public data resources, analyzes data timeliness, and establishes a historical span database with update frequency and time delay requirements; A data processing module, which is used for data extraction, data cleaning, data conversion, data fusion and data storage; Data verification module, which includes setting a reasonable range of differences, accuracy verification, completeness verification, consistency verification and timeliness verification; The monitoring and optimization module includes a monitoring indicator system, real-time monitoring function, early warning capability and decision support.

2. A multi-data source data processing method, used for processing a multi-data source data processing system according to claim 1, characterized in that: The following steps are involved: Step 1: Data collection: determine the data source and establish a connection; Step 2: Data processing: extracting, cleaning, converting, integrating and storing the data collected from the data; Step 3: Data Verification: Verify the quality of the processed data and monitor the data processing process in real time.

3. The method for processing data from multiple data sources according to claim 2, characterized in that: In the data collection, the method for determining the data source is as follows: Step 1: Determine data requirements. According to the core business objectives and data indicators, determine the data type, find out the data factors that affect the realization of business objectives, and define data indicators based on business objectives and data factors; Step 2: Identify potential data sources. First, inventory internal data sources, sort out internal information systems and storage locations, and establish an internal database. Then explore external data sources, retrieve industry public data resources, establish an external database, and integrate the internal and external databases to obtain a comprehensive database. Step 3: Evaluate the quality of data sources. First, verify data accuracy. Select a part of data samples and compare and verify with the reference data in the comprehensive database to clarify the data errors in the data source and obtain error information. The error information is used as a reference for evaluating data integrity. Then evaluate data integrity, analyze the missing data of sample data, calculate the proportion of missing values ​​of each sample data, understand the completeness of each sample data in different data sources, and finally analyze the timeliness of data. According to actual needs, establish a historical span database that meets the update frequency and time delay. The method of establishing a connection includes the following steps: Step 1: Analyze the data source structure, analyze the data storage architecture, data format and data organization of the internal and external databases. For the internal database, understand the data storage logic of each information system, including the table structure, field type and association relationship. For the external database, analyze the file format of the public data resources and the hierarchical structure of the data. According to the error information, missing value situation and data timeliness characteristics obtained in the stage of evaluating the quality of the data source, determine which data parts need to be focused on and the priority to establish connections; Step 2: Select connection technology. Select connection technology and tools according to the type and characteristics of the data source. Connection methods include database connection interface, API call and data integration platform. For internal databases, if it is a homogeneous database system, you can directly use the connection components that come with the database to connect. If it is a heterogeneous database, you need to use middleware. For external databases, configure the corresponding connection parameters and authentication information according to the access method provided by it. Step 4: Configure connection parameters. For the selected connection technology, configure detailed connection parameters. For database connection, set the database server address, port number, user name, password, and the specific database name to be connected. For API connection, fill in the API key, requested URL address, and request parameter format. Set the connection update frequency parameters according to the data timeliness requirements. Step 5: Test the connection. After configuring the connection parameters, perform a connection test, select data samples, read data from the data source and verify the accuracy and completeness of the data, check whether the connection is stable and whether the data transmission is smooth, obtain the test results, and compare the test results with the data characteristics in the previous stage of evaluating the quality of the data source; Step six: Establish connection monitoring, monitor the connection status, data flow, and data quality indicators in real time, set warning thresholds, and issue an alarm when the connection is abnormal or the data quality drops to the warning threshold to notify relevant personnel to handle it. At the same time, regularly maintain and optimize the connection, and adjust the connection parameters according to changes in business needs and updates to data sources.

4. The method for processing data from multiple data sources according to claim 3, characterized in that: The data processing comprises the following steps: Step 1: Data extraction. Determine the extraction rules. According to the core business objectives and key indicator requirements, clarify which data fields and data records to extract from the data source and the time range of extraction. Select data extraction tools based on the type and connection method of the data source. According to the extraction rules and data extraction tools, extract the required data from the data source and temporarily store it in an intermediate storage area for subsequent processing. Step 2: Data cleaning and removal of duplicate data. By comparing the key features of the data, duplicate data records are identified and deleted. Then the distribution of missing values ​​in the data is analyzed. For missing values ​​in important fields, the following methods can be used: First, define Represents sample points The probability of missing values, represents the sum of the probabilities of missing values ​​of all sample points, Represents the total number of missing sample points, then: ； in, is an indicator function, when When there is value, ,when When missing, ; set up They represent the weight coefficients of the three methods: interpolation, model-based prediction, and marking as unknown, and , represents the estimated value of the missing value obtained by interpolation, represents the estimate of the missing value obtained through model-based prediction, Indicates the characteristic tag value when the tag is unknown, Represents optimized data, for sample points The total estimated value of , the specific calculation formula is as follows: ； Construct an objective function based on all missing sample points , , here It represents the actual value when it has a value, and is equivalent to an abstract representation of the true value to be estimated during the optimization process when it is missing. right To find partial derivatives, , denoted as ; right To find partial derivatives, , denoted as ; right To find partial derivatives, , denoted as ; Then update the weight coefficients according to the update rule of the gradient descent method: Suppose a learning rate is , in each iteration, the update formula of the weight coefficient is as follows: ； ； ； Repeat the above steps until the objective function Convergence, get the optimized data, the optimized data is infinitely close to the true value.

5. A method for processing data from multiple data sources according to claim 4, characterized in that: The data processing further comprises the following steps: Step 1: Data conversion: convert the data type according to the needs of data analysis and storage; Step 2: Data conversion: convert data in some specific character encoding formats into a unified encoding format; Step 3: Data calculation and derivation: Calculate and derive new fields based on the original data according to business needs; Step 4: Data fusion, which combines data from different data sources but with associations. Data association establishes associations between data, and associates data in different data tables through common fields to obtain a complete data view. Step 5: Data integration: Integrate the merged and associated data to eliminate inconsistencies and conflicting data to obtain a unified and consistent data set; Step 6: Data storage: Select storage media and databases based on data volume, access frequency, and data type. Step 7: Design the data storage structure. According to the characteristics of the data and analysis requirements, design the database table structure, the dimensional model of the data warehouse, and the document structure of the NoSQL database. Step 8, data loading: load the processed and integrated data into the database according to the designed storage structure, using batch loading and real-time loading methods.

6. A method for processing data from multiple data sources according to claim 5, characterized in that: The data verification comprises the following steps: Step 1: Set a reasonable range of differences: According to business characteristics, comprehensive databases, and historical span databases, set the maximum and minimum thresholds. The reasonable range of differences is between the maximum and minimum thresholds. When it is found that the difference between the estimated value and the reference value exceeds the reasonable range, quickly guide relevant personnel to locate areas where data errors may exist and mark them as potential grids. The specific marking method is as follows: Optimize the data to calculate various statistical indicators based on it in the mathematical operation verification phase of the data, and compare them with the expected indicators based on accurate data to verify the rationality of the data in mathematical operation logic. If significant differences are found, it means that there may be errors or outliers in the data; Step 2: Completeness verification: Optimize the data by checking the number of statistical data records and the filling status of the required fields to judge the completeness of the data; Step 3: Consistency Verification: Optimize data. According to the context and business rules of the data, adjust the differences of the same entity in different data sources, including differences in attribute values ​​and missing conditions. For consistency verification of time series data in multi-data source data processing, optimize the data so that the time series data from different data sources are aligned and continuous in terms of time nodes and data values. Step 4: Timeliness verification: When data from a certain data source is missing or delayed, optimize the data to fill the gaps in a timely manner.

7. A method for processing data from multiple data sources according to claim 6, characterized in that: The optimized data discovers problems in the data processing process by real-time monitoring of the differences between actual data and optimized data, and guides relevant personnel to adjust data processing algorithms and parameters according to the size and changing trends of the differences. In real-time monitoring of multi-data source data processing, the optimized data quickly identifies data differences and synchronization problems between data sources by comparing data features of the same data points in different data sources. When the data quality of the optimized data deteriorates or an abnormality occurs, if the difference between the optimized data and the actual data exceeds a preset threshold, or the number of missing values ​​in a certain data source suddenly increases, the system can quickly issue an alarm based on the changes and notify relevant personnel to handle it. For the complex environment of multi-data source data processing, the optimized data locates the problem by monitoring the changing trends and abnormal fluctuations of the data sources.

8. A method for processing data from multiple data sources according to claim 7, characterized in that: In the real-time monitoring process, the optimization data is combined with the real-time generated data. Through real-time analysis of the data source, the trends and changes in the data processing process can be observed. In the real-time monitoring of multi-data source data processing, through comprehensive analysis of the optimization data, the data processing process is dynamically optimized based on the real-time monitoring data and optimization data.

Images