Heterogeneous data source fusion analysis system

Abstract

The invention relates to the technical field of data processing, and discloses a heterogeneous data source fusion analysis system. The system comprises a data integration module, a feature analysis module and a fusion evaluation module. The data integration module obtains data quality parameters of heterogeneous data source distributed storage nodes, generates an evaluation value through multi-dimensional analysis, judges data fusion reliability and generates a priority instruction. The feature analysis module is used for extracting abnormal time nodes according to the high-priority instruction, performing association analysis to generate a data feature rule value, judging whether the abnormality is periodic or not, and generating a rule adjustment instruction; and the fusion evaluation module determines an abnormal influence time period according to the rule adjustment instruction, matches the abnormal influence time period with the current fusion time node, dynamically adjusts the fusion process and generates an instruction. The system can effectively evaluate the data quality, mine the data rule, dynamically adjust the fusion process and improve the accuracy and reliability of heterogeneous data source fusion analysis, and is widely applicable to various data processing scenes.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a heterogeneous data source fusion analysis system. Background Art

[0002] In today's digital age, data has become a critical basis for decision-making in businesses and organizations. With the rapid development of information technology, data sources are becoming increasingly diverse, and heterogeneous data sources are becoming the norm. Heterogeneous data sources refer to data sources that differ in data structure, storage methods, data formats, and communication protocols, such as relational databases, non-relational databases, file systems, and sensor data. While this diversity provides rich material for data analysis, it also presents numerous challenges.

[0003] The data quality of heterogeneous data sources varies. Data consistency and integrity across different data sources are difficult to ensure. In some distributed systems, data may be lost, duplicated, or erroneous during transmission or storage, leading to data consistency issues. For example, on e-commerce platforms, order data may differ across different database nodes, resulting in inconsistent order information and hindering accurate analysis of sales data. Furthermore, some data sources may not be able to provide the required data for various reasons, making data missing common. This significantly hinders the accuracy and completeness of data analysis. In financial risk assessment scenarios, if some client financial data is missing, risk assessment results will be biased and risk prediction will be inaccurate.

[0004] Fusion of heterogeneous data sources is challenging. Differences in data structure and format require complex data conversion and preprocessing. For example, the fusion process for structured relational database data and unstructured text data must address issues such as data schema mismatches and inconsistent data types. Furthermore, the update frequencies and timestamp formats of different data sources vary, making it difficult to ensure data timeliness and accuracy during data fusion. In the IoT environment, data collected by multiple sensors is difficult to synchronize, and fusion can easily lead to time series confusion, hindering accurate analysis of physical phenomena.

[0005] Traditional data fusion and analysis methods often lack dynamic adjustment capabilities. When faced with data quality anomalies, they are unable to promptly detect and implement effective countermeasures, resulting in skewed fusion results. Once data quality issues arise, the entire data analysis process may need to be re-run, consuming significant time and resources. For example, in intelligent transportation systems, if road sensor data exhibits anomalies, traditional methods cannot promptly adjust the fusion process, resulting in erroneous traffic flow analysis results and, in turn, impacting traffic scheduling decisions.

[0006] The rapid growth of data volumes has also placed higher demands on the fusion and analysis of heterogeneous data sources. This massive amount of data not only increases the pressure on data storage and transmission, but also exponentially increases the complexity of data processing. Traditional analysis systems struggle to integrate and analyze large amounts of heterogeneous data in a short period of time, failing to meet real-time requirements. In social media data analysis, the massive amounts of text, images, and other data generated every second require timely processing to extract valuable information, but traditional systems struggle to meet this requirement. Summary of the Invention

[0007] The purpose of the present invention is to provide a heterogeneous data source fusion analysis system to solve the problems raised in the above background technology.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a heterogeneous data source fusion analysis system, the system comprising:

[0009] Data integration module: obtains data quality parameters of distributed storage nodes associated with heterogeneous data sources over multiple data collection cycles, generates data quality assessment values ​​based on multi-dimensional analysis of the data quality parameters, determines the data fusion reliability of the distributed storage nodes based on the data quality assessment values, and generates priority instructions, wherein the instructions include high-priority adjustment instructions and low-priority adjustment instructions;

[0010] comparing the data quality assessment value with the data quality assessment threshold;

[0011] If the data quality assessment value is greater than the data quality assessment threshold, a high priority adjustment instruction is generated;

[0012] If the data quality assessment value is less than or equal to the data quality assessment threshold, a low priority adjustment instruction is generated;

[0013] Feature Analysis Module: Based on high-priority adjustment instructions, it extracts the time nodes where data quality anomalies occur in distributed storage nodes within multiple data collection cycles. It then performs correlation analysis based on the time nodes and data quality parameters to generate data feature regularity values. Based on these values, it determines whether the data quality anomaly exhibits periodic characteristics. If so, it generates regularity adjustment instructions.

[0014] The time point of data quality anomaly includes the specific time point when the anomaly occurs;

[0015] Fusion evaluation module: Based on regular adjustment instructions, it determines the abnormal impact period within the data fusion cycle and matches it with the current fusion time node. According to the matching results, it dynamically adjusts the fusion process of heterogeneous data sources and generates adjustment instructions, among which the adjustment instructions include pause fusion instructions and continue fusion instructions.

[0016] Preferably, the data quality parameters include a data consistency index and a data integrity index;

[0017] Based on independent analysis of data consistency indicators and data integrity indicators, a consistency assessment value and an integrity assessment value are generated respectively;

[0018] The consistency assessment value and the integrity assessment value are weighted and fused to generate a data quality assessment value.

[0019] Preferably, the data consistency indicators of the distributed storage nodes in multiple data collection cycles are integrated and the mean is calculated to generate a consistency mean, the consistency mean is proportionally processed with the total duration of the data collection cycle to obtain the consistency frequency, the consistency frequency is normalized with the consistency frequency threshold to generate a consistency evaluation value.

[0020] Preferably, the data integrity indicators of the distributed storage nodes in multiple data collection cycles are accumulated to generate an integrity sum, the integrity sum is proportional to the number of data collection cycles to obtain an integrity mean, the integrity mean is normalized with the total duration of the data collection cycle to generate an integrity assessment value.

[0021] Preferably, the data characteristic regularity value is generated in the following manner:

[0022] Generate data stability performance value and data change performance value;

[0023] The data stability performance value and the data change performance value are linearly combined through preset weight coefficients to generate data characteristic regularity values.

[0024] Preferably, the data stability performance value is generated in the following manner:

[0025] Screen historical data collection periods where both data consistency and data integrity indicators are within normal ranges, mark them as stable periods, and serialize and mark the abnormal data quality time points in each stable period in chronological order;

[0026] Calculate the time interval deviation of adjacent time points based on the serialized marked time points;

[0027] Compare the time interval deviation with the preset interval threshold. If the deviation is less than or equal to the threshold, it is marked as a stable interval.

[0028] The ratio of the number of stable intervals to the total number of intervals is counted. If the ratio exceeds a preset ratio threshold, the stable period is marked as a comprehensive stable period.

[0029] The ratio of the number of comprehensive stable periods to the total number of historical data collection periods is counted to generate the data stability performance value.

[0030] Preferably, the data change performance value is generated in the following manner:

[0031] Extract the interval time of adjacent data quality abnormal time points within all comprehensive stable periods to form an interval time set;

[0032] Calculate the variance of each group of interval time in the interval time set and take the average value to generate the variance mean, which is marked as the data change performance value.

[0033] Preferably, the data characteristic regularity value is compared with the characteristic regularity threshold;

[0034] If the data characteristic regularity value is greater than the characteristic regularity threshold, a regularity adjustment instruction is generated;

[0035] If the data characteristic regularity value is less than or equal to the characteristic regularity threshold, no action is triggered.

[0036] Preferably, the abnormal impact period within the data fusion cycle is determined as follows:

[0037] Extract the start and end time points of the data fusion cycle and mark them on the time axis;

[0038] Obtain the most recent abnormal data quality time point before the start time point of the data fusion cycle and use it as the benchmark point;

[0039] Calculate the average of the interval time of historical data quality anomalies, generate the interval benchmark value, take the benchmark point as the starting point, and mark the predicted anomaly starting points on the time axis in sequence according to the interval benchmark value;

[0040] The preset abnormality duration is superimposed after each predicted abnormality starting point to generate a predicted abnormality ending point;

[0041] The period between the predicted anomaly start point and the corresponding predicted anomaly end point is marked as the predicted anomaly period, and the overlapping part of the time range of the data fusion cycle and the predicted anomaly period is marked as the anomaly impact period.

[0042] Preferably, when executing the data fusion task, the current fusion time node is obtained and matched with the abnormal impact period;

[0043] If the current fusion time node is within the abnormal impact period, a pause fusion instruction is generated;

[0044] If the current fusion time node is outside the abnormal impact period, a matching instruction is generated;

[0045] Based on the instruction to be matched, locate the predicted anomaly starting point closest to the current fusion time node on the time axis and calculate the interval between the two;

[0046] If the interval duration is greater than or equal to the preset safety threshold, a continue fusion instruction is generated;

[0047] If the interval duration is less than the preset safety threshold, a pause fusion instruction is generated.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] At the data quality assessment level, the system obtains data quality parameters, such as data consistency and data integrity indicators, from distributed storage nodes associated with heterogeneous data sources over multiple data collection cycles and performs a multi-dimensional analysis. Consistency and integrity assessment values ​​are generated separately, and then weighted and integrated to derive a data quality assessment. This scientific assessment method accurately measures data quality. Compared to traditional, simple judgment methods, it fully considers the diverse characteristics of data and avoids the one-sidedness of single-metric assessments. Taking data processing in the financial industry as an example, in the integrated analysis of customer asset data and transaction data, accurate quality assessment can ensure data accuracy, reduce risk assessment errors caused by data errors, provide a reliable basis for financial institutions' decision-making, and mitigate potential economic losses.

[0050] Based on the data quality assessment results, the system generates priority instructions. When the data quality assessment value exceeds the data quality assessment threshold, a high-priority adjustment instruction is generated; when it is less than or equal to the threshold, a low-priority adjustment instruction is generated. This enables the system to promptly adopt different processing strategies based on the data quality status and rationally allocate system resources. When processing large amounts of product data on e-commerce platforms, high-quality data is prioritized for integration analysis, quickly outputting accurate sales trend reports and helping merchants adjust their business strategies in a timely manner. Low-quality data is processed when resources are relatively idle, improving the overall operational efficiency of the system.

[0051] The feature analysis module further unlocks the value of data. Based on high-priority adjustment instructions, it extracts the time nodes of data quality anomalies, performs correlation analysis based on these time nodes and data quality parameters, generates data feature regularity values, and determines whether the data quality anomalies exhibit periodic characteristics. If so, it generates regularity adjustment instructions. This function provides a deeper understanding of the inherent patterns in the data and identifies potential problems in advance. In power system monitoring, by analyzing the periodic characteristics of power data quality anomalies, the time of equipment failure can be predicted, maintenance work can be scheduled in advance, and power outages caused by equipment failures can be reduced, ensuring the stability of the power supply and reducing maintenance costs.

[0052] The fusion assessment module adjusts instructions based on regular patterns, identifies periods of abnormal impact within the data fusion cycle, matches these periods with the current fusion time point, and dynamically adjusts the fusion process for heterogeneous data sources, generating instructions to pause or resume fusion. This dynamic adjustment mechanism effectively avoids fusion operations during periods of data quality anomalies, ensuring the accuracy of fusion results. In meteorological data fusion analysis, when meteorological sensor data exhibits anomalies, the system suspends fusion to prevent erroneous data from being incorporated into the analysis results, ensuring the reliability of weather forecasts and providing accurate weather information for people's daily lives.

[0053] The methods for generating data stability and data variation performance values ​​characterize data from different perspectives. The data stability performance value is derived by screening stable periods and analyzing time interval deviations, reflecting data stability. The data variation performance value is generated by calculating the mean of the variance of intervals and reflects data variability. The linear combination of the two generates a data characteristic regularity value, providing a comprehensive basis for identifying abnormal characteristics in data quality. In logistics data processing, this characteristic analysis can help companies better understand the laws of logistics distribution, promptly identify abnormal fluctuations, optimize logistics routes and delivery plans, improve logistics efficiency, and reduce operating costs.

[0054] The method for determining the abnormal impact period within the data fusion cycle comprehensively considers factors such as the average interval between historical data quality anomalies, making the determination of abnormal periods more scientific and reasonable. In practical applications, when executing data fusion tasks, the system can accurately make decisions based on the matching of the current fusion time node and the abnormal impact period, improving the system's adaptability and reliability. In industrial production process monitoring, this function ensures the accuracy of production data fusion analysis, promptly detects abnormalities in the production process, ensures smooth production, and improves product quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a working principle diagram of the heterogeneous data source fusion analysis system according to the present invention;

[0056] Figure 2 Schematic diagram of how data quality assessment values ​​are generated;

[0057] Figure 3 A diagram showing the working principle for generating stable performance values ​​for data;

[0058] Figure 4 A diagram showing the working principle of the generated regularity adjustment instructions.

[0059] Figure 5 Diagram of the working principle for determining the abnormal impact period within the data fusion cycle. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] See also Figure 1-Figure 5 The present invention provides a heterogeneous data source fusion analysis system, and the specific implementation steps are as follows:

[0062] During system operation, the data integration module continuously obtains data quality parameters from distributed storage nodes associated with heterogeneous data sources over multiple data collection cycles. These data quality parameters play a key role in assessing data quality. After obtaining the data quality parameters, the data integration module performs a multi-dimensional analysis, comprehensively considering factors from each dimension to generate a data quality assessment value that accurately reflects the data quality status. Based on this data quality assessment value, the data integration module determines the data fusion reliability of the distributed storage nodes. Specifically, the data quality assessment value is compared with a pre-set data quality assessment threshold. If the data quality assessment value exceeds the data quality assessment threshold, indicating good data quality and high fusion reliability, a high-priority adjustment instruction is generated. Conversely, if the data quality assessment value is less than or equal to the data quality assessment threshold, a low-priority adjustment instruction is generated. These instructions provide a basis for subsequent system data processing priorities.

[0063] The feature analysis module begins operation after receiving a high-priority adjustment instruction. It extracts the time points at which data quality anomalies occurred on distributed storage nodes across multiple data collection cycles, pinpointing the exact moment the anomaly occurred. It then performs correlation analysis based on these time points and data quality parameters. By analyzing and processing large amounts of data, it generates data feature regularity values. Once these values ​​are determined, the feature analysis module determines whether the data quality anomaly exhibits periodic characteristics. If the data feature regularity values ​​indicate periodicity, it generates regularity adjustment instructions, providing a basis for the subsequent fusion evaluation module to adjust the fusion process.

[0064] After receiving the regularity adjustment instruction, the fusion assessment module determines the abnormal impact period within the data fusion cycle. It matches this abnormal impact period with the current fusion time node and dynamically adjusts the fusion process of heterogeneous data sources based on the matching results. If the matching result shows that the current fusion time node is within the abnormal impact period, a pause fusion instruction is generated to avoid fusion operations during the data quality abnormality period and ensure the quality of the fused data. If the current fusion time node is outside the abnormal impact period, the module further determines whether fusion can continue and generates a continue fusion instruction or other related instructions based on the different situations.

[0065] The technical solution of the present invention is further described in detail below with reference to specific embodiments.

[0066] Example 1:

[0067] In the system, data quality parameters include data consistency indicators and data integrity indicators. For data consistency indicators, the system integrates the data consistency indicators of distributed storage nodes over multiple data collection cycles. The data consistency indicators within each cycle are aggregated and their average is calculated to obtain the mean consistency value. This mean consistency value reflects the average level of data consistency over multiple cycles. Next, the mean consistency value is proportional to the total duration of the data collection cycle. This proportionality process allows us to understand the frequency of data consistency changes throughout the entire data collection period and thus determine the consistency frequency. The consistency frequency is then normalized with the consistency frequency threshold. Normalization is performed to unify data of different magnitudes onto a standard scale for easier comparison and analysis. This process ultimately generates a consistency assessment value.

[0068] For data integrity metrics, the system accumulates the data integrity metrics of distributed storage nodes across multiple data collection cycles to obtain a total integrity score. This total is then proportional to the number of data collection cycles to calculate the average level of data integrity within each cycle, resulting in a mean integrity score. Finally, the mean integrity score is normalized by the total duration of the data collection cycle to generate a completeness assessment value.

[0069] When generating data quality assessments, the system combines the consistency and integrity assessments with a weighted approach. Based on actual needs and the importance of the data, the consistency and integrity assessments are assigned different weights. This weighted calculation comprehensively considers both data consistency and integrity, ultimately generating a data quality assessment that comprehensively reflects data quality. This approach to generating assessments enables a more scientific and accurate assessment of data quality, providing a reliable basis for subsequent data processing.

[0070] Consider a medical data management scenario involving multiple different types of medical devices. These devices act as heterogeneous data sources and store data in distributed storage nodes. Each device collects data at a fixed interval, for example, every 10 minutes. This 10-minute period constitutes a data collection cycle.

[0071] In this scenario, data consistency metrics primarily measure whether data collected by different medical devices is logically and numerically consistent. For example, multiple devices measuring a patient's vital signs should produce similar heart rate data for the same patient. Significant deviations indicate a data consistency issue. Data integrity metrics focus on data completeness. For example, in a patient's medical record, all required fields must be complete; missing fields indicate insufficient data integrity.

[0072] After the system begins running, over a period of time (including multiple data collection cycles), the system integrates relevant information about the data consistency of different devices within each cycle. For example, over 10 data collection cycles (i.e., 100 minutes), the consistency of heart rate data from each device is recorded. The mean of the heart rate data consistency index for these 10 cycles is calculated. Assuming that these index values ​​are 85, 88, 90, 86, 87, 89, 91, 88, 87, and 86, respectively, the sum is divided by the number of cycles, 10, to obtain a mean consistency value of 87.7.

[0073] Next, the consistency mean of 87.7 is proportional to the total duration of the data collection cycle of 100 minutes. Here, a reasonable calculation method can be used (the specific calculation method depends on the system settings and the formula is not involved for the time being) to obtain a value that reflects the frequency of consistency changes, namely the consistency frequency. Assume that the consistency frequency obtained is 0.877. Afterwards, the consistency frequency of 0.877 is normalized with the pre-set consistency frequency threshold (assuming it is 0.8). The normalization process will measure the consistency frequency and threshold under the same standard according to the established rules of the system, and finally generate a consistency evaluation value. For example, after a series of processing, the consistency evaluation value obtained is 0.9.

[0074] For the data integrity indicator, similarly, over these 10 data collection cycles, the values ​​related to medical data integrity are accumulated for each cycle. Assuming the data integrity indicator values ​​for each cycle are 90, 88, 92, 89, 91, 90, 87, 88, 90, and 93, these values ​​are added together to obtain a total integrity score of 898. This total, 898, is then proportionally calculated by the number of data collection cycles (10), yielding a mean integrity score of 89.8. Finally, this mean integrity score of 89.8 is normalized by the total data collection cycle duration of 100 minutes to generate a completeness score. Assume that the final completeness score is 0.89.

[0075] When generating data quality assessment values, the system assigns different weights to consistency assessment values ​​and integrity assessment values ​​based on actual conditions. For example, based on the characteristics and importance of medical data, data consistency is considered relatively more important, so a weight of 0.6 is assigned to the consistency assessment value and a weight of 0.4 is assigned to the integrity assessment value. Through weighted calculation (no formula involved, just a conceptual explanation), the consistency assessment value of 0.9 is multiplied by 0.6, and the integrity assessment value of 0.89 is multiplied by 0.4, and then the two are added together to obtain a data quality assessment value of 0.9×0.6+0.89×0.4=0.54+0.356=0.896. This data quality assessment value comprehensively considers data consistency and integrity, can more comprehensively reflect the data quality status, and provide a reliable basis for subsequent data processing and analysis.

[0076] Example 2:

[0077] The process of generating the data stability performance value is complex and rigorous. The system selects historical data collection periods where both data consistency and data integrity indicators are within normal ranges and marks these periods as stable periods. This step identifies periods of relatively stable data quality, providing a reliable data foundation for subsequent analysis. Next, the system chronologically marks the time points of data quality anomalies within each stable period to facilitate analysis of their relationships. Based on these marked time points, the time interval deviation between adjacent time points is calculated. This calculation reveals the variation in the time intervals between data quality anomalies. The time interval deviation is then compared with a preset interval threshold. If the deviation is less than or equal to the threshold, the interval is relatively stable and is marked as a stable interval. The system then calculates the ratio of the number of stable intervals to the total number of intervals. If this ratio exceeds the preset threshold, the intervals of data quality anomalies within this stable period are relatively stable, and the stable period is marked as a comprehensive stable period. Finally, the ratio of the number of comprehensive stable periods to the total number of historical data collection periods is calculated; this ratio represents the data stability performance value. The data stability performance value reflects the stability of data over time and provides an important reference for analyzing data characteristics.

[0078] Consider an e-commerce data processing system with multiple distributed storage nodes storing sales data from different stores, with each data collection cycle lasting one hour. Data consistency measures the logical and numerical consistency of various sales data (such as order volume and sales revenue) collected by different stores at the same time. Data integrity measures the completeness of required information (such as product name, sales quantity, and customer information) within store sales data.

[0079] Over a period of time, the system holds sales data from multiple historical data collection cycles. First, the system screens for historical data collection cycles whose data consistency and integrity indicators are within normal ranges. For example, after checking, it is found that within the past 24 data collection cycles (i.e., 24 hours), 15 cycles have data consistency and integrity indicators that meet normal standards. These 15 cycles are marked as stable.

[0080] For each stable period, the system serializes and marks the time points of data quality anomalies in chronological order. For example, if two data quality anomalies occur within a stable period (the 10th hour of the collection cycle), the first at 10:15 and the second at 10:45, then 10:15 is marked as anomaly time point 1, and 10:45 is marked as anomaly time point 2.

[0081] The time interval deviation between adjacent time points is calculated based on the serialized marked time points. In this example, the time interval between abnormal time points 1 and 2 is 30 minutes. If the system's preset interval threshold is 40 minutes and 30 minutes is less than 40 minutes, then this interval is marked as a stable interval.

[0082] The system calculates the ratio of stable intervals to the total number of intervals. Within these 15 stable periods, there are 20 abnormal time point intervals, 16 of which are below the preset interval threshold. Therefore, the ratio of stable intervals to the total number of intervals is 16 ÷ 20 = 0.8. If the preset ratio threshold is 0.7, and 0.8 is greater than 0.7, then this stable period is marked as a comprehensive stable period.

[0083] The data stability performance value is calculated by calculating the ratio of the number of comprehensive stable periods to the total number of historical data collection periods. Of the 24 historical data collection periods, 12 are marked as comprehensive stable periods, so the data stability performance value is 12 ÷ 24 = 0.5. This data stability performance value reflects the stability of the intervals between data quality anomalies in e-commerce sales data over a longer period of time, providing an important reference for subsequent analysis of data characteristics and patterns.

[0084] Example 3:

[0085] The data change performance value is generated based on data from all comprehensive stable periods. The system extracts the intervals between adjacent data quality anomaly points within each comprehensive stable period and collects these intervals to form an interval set. This set contains multiple interval data, reflecting the variation in the intervals between data quality anomaly points within the comprehensive stable period. Next, the variance of each interval in the interval set is calculated and averaged. Variance measures the degree of data dispersion. By averaging the variances, a value reflecting the fluctuation of interval changes is obtained, which is labeled the data change performance value. The data change performance value supplements information about data characteristic patterns from another perspective and, together with the data stability performance value, provides more comprehensive data support for generating the data characteristic pattern value. When generating the data characteristic pattern value, the system linearly combines the data stability performance value and the data change performance value using preset weighting coefficients. The weighting coefficients for the data stability performance value and the data change performance value are pre-set based on the actual data characteristics and analysis requirements. Through the linear combination calculation, the two values ​​are combined according to the weighting to ultimately generate a data characteristic pattern value that accurately reflects the data characteristic pattern.

[0086] Taking a city traffic flow monitoring system as an example, there are many sensor nodes distributed in the system, which are used to collect traffic flow data on different road sections. These sensor nodes constitute a heterogeneous data source, and data collection is performed every 15 minutes. This 15-minute period is a data collection cycle.

[0087] In the process described in this embodiment, the system first determines all comprehensive stability periods according to the method of embodiment 2. Assume that after analysis, 30 comprehensive stability periods are determined in the monitoring data of a certain day (a total of 96 data collection periods).

[0088] The system extracts the intervals between adjacent data quality anomalies within these 30 comprehensive stable periods to form an interval time set. For example, in one comprehensive stable period, the first data quality anomaly occurred at 10:15 and the second at 10:45, with an interval of 30 minutes. In another comprehensive stable period, the interval between two consecutive data quality anomalies was 25 minutes, and so on. All such intervals are collected to form a set containing a large number of interval time data.

[0089] The system calculates the variance of each interval in the interval time set and takes the average. Variance is a statistic used to measure the degree of dispersion in a set of data, reflecting the fluctuations in these intervals. Assuming that after complex calculations (the specific formulas are not involved here), the final average variance is 15 (for example data only). This value is marked as the data variation performance value. It reflects the fluctuation of the intervals between data quality anomalies within the overall stability period.

[0090] When generating the data characteristic regularity value, the system will linearly combine the data stability performance value and the data change performance value through a preset weight coefficient. Assume that according to the characteristics and analysis requirements of the city's traffic flow data, the weight coefficient of the data stability performance value is pre-set to 0.6, and the weight coefficient of the data change performance value is pre-set to 0.4. If the data stability performance value obtained by Example 2 is 0.7, then according to the calculation method of the linear combination (no formula is involved, only a conceptual explanation), the data stability performance value 0.7 is multiplied by 0.6, and the data change performance value 15 is multiplied by 0.4, and then the two are added, that is, 0.7×0.6+15×0.4=0.42+6=6.42, and the final data characteristic regularity value generated is 6.42. This data characteristic regularity value comprehensively considers the stability and variability of the data, and can more comprehensively and accurately reflect the characteristic regularity of the abnormal quality of urban traffic flow data, providing important data support for subsequent traffic flow analysis and system decision-making.

[0091] Example 4:

[0092] When determining the abnormal impact period within the data fusion cycle, the system first extracts the start and end time points of the data fusion cycle and clearly marks them on the timeline. This clearly defines the time range of the data fusion cycle and provides a basis for subsequent analysis. Next, the most recent data quality abnormality before the start time point of the data fusion cycle is obtained and used as the reference point. This reference point is a key reference for subsequent prediction of the abnormal time.

[0093] Next, the mean of the intervals between historical data quality anomalies is calculated to generate a baseline interval value. By analyzing historical data, an average anomaly interval is obtained, which serves as a basis for predicting future anomaly times. Starting from the baseline point, predicted anomaly start points are sequentially marked on the timeline according to the baseline interval value. Based on the average interval, the time when future data quality anomalies may occur is estimated on the timeline. A preset anomaly duration is added to each predicted anomaly start point to generate a predicted anomaly end point. The preset anomaly duration is set based on experience or historical data and is used to determine how long the anomaly may persist. Finally, the period between the predicted anomaly start point and the corresponding predicted anomaly end point is marked as the predicted anomaly period. The overlapping portion of the data fusion cycle's time range and the predicted anomaly period is then marked as the anomaly impact period. This approach accurately identifies the time period within the data fusion cycle that may be affected by data quality anomalies, providing an accurate basis for subsequent adjustments to the fusion process.

[0094] Consider a financial transaction data processing system that integrates transaction data from multiple financial institutions. This data is stored in distributed storage nodes, and the data integration cycle is set to be daily (from 00:00 to 23:59) to generate a daily comprehensive transaction report.

[0095] The system first extracts the start time (00:00) and end time (23:59) of the data fusion cycle and clearly marks them on the time axis. This determines the time range involved in the data fusion operation on that day.

[0096] The system needs to obtain the time point of the most recent data quality anomaly before the start time of the data fusion cycle. Assuming that a data quality anomaly occurred at 22:30 yesterday (the day before the data fusion cycle), then this 22:30 is used as the reference point.

[0097] The system calculates the average interval between historical data quality anomalies to generate the interval baseline. The system collects data quality anomaly time points over a period of time (for example, the past 30 days) and calculates the intervals between adjacent anomaly time points. Assuming these intervals are 12 hours, 8 hours, 10 hours, and so on, the system performs a statistical calculation (the specific calculation process does not involve formulas) to obtain an average interval of 10 hours. This 10-hour interval is the interval baseline.

[0098] Starting at 22:30, the predicted anomaly start points are marked on the timeline at 10-hour intervals. The first predicted anomaly start point is 22:30 plus 10 hours, which is 8:30 today. The second predicted anomaly start point is 8:30 plus 10 hours, which is 18:30 (time conversion is required because it exceeds 24 hours).

[0099] The predicted anomaly end point is generated by adding the preset anomaly duration to each predicted anomaly start point. Assuming the preset anomaly duration is 2 hours, the predicted anomaly end point for the first predicted anomaly start point of 8:30 is 8:30 plus 2 hours, which is 10:30. The predicted anomaly end point for the second predicted anomaly start point of 18:30 is 20:30.

[0100] The period between the predicted anomaly start point and the corresponding predicted anomaly end point is marked as the predicted anomaly period, that is, 8:30-10:30 and 18:30-20:30 are predicted anomaly periods. Then, the overlapping part of the time range of the data fusion cycle (00:00-23:59 on the same day) and the predicted anomaly period is marked as the anomaly impact period. In this example, 8:30-10:30 and 18:30-20:30 are both within the data fusion cycle of the day, so these two periods are the anomaly impact periods. In this way, the system can accurately determine the time period within the data fusion cycle that may be affected by data quality anomalies, provide an accurate basis for subsequent adjustments to the fusion process, and ensure the quality of the generated daily comprehensive transaction report data is reliable.

[0101] Example 5:

[0102] When the system executes a data fusion task, it obtains the current fusion time node. This time node is the key basis for determining whether to continue the fusion operation. The system matches the current fusion time node with the anomaly impact period. If the current fusion time node falls within the anomaly impact period, data fusion at this time may be affected by data quality anomalies. To ensure the quality of the fused data, the system generates a pause fusion instruction, pausing the data fusion operation. If the current fusion time node falls outside the anomaly impact period, the system generates a pending match instruction. Based on this pending match instruction, the system locates the predicted anomaly onset point closest to the current fusion time node on the timeline and calculates the interval between the two points. If the interval is greater than or equal to a preset safety threshold, the fusion operation at the current fusion time node will not be affected by data quality anomalies for a period of time. The system generates a continue fusion instruction, allowing the data fusion operation to proceed. Conversely, if the interval is less than the preset safety threshold, the current fusion time node is close to the possible anomaly time. To avoid risks, the system generates a pause fusion instruction, pausing the data fusion operation. Through this matching and judgment mechanism, the system can dynamically adjust the fusion process of heterogeneous data sources based on actual conditions, effectively improving the quality and reliability of data fusion.

[0103] Consider a data fusion system for an online education platform. This system integrates data from multiple sources, such as student learning progress data and course feedback data, stored on different distributed storage nodes. The data fusion cycle is set to occur every hour to generate real-time learning analysis reports.

[0104] When the system executes a data fusion task, it obtains the current fusion time node. For example, the current fusion time node is 14:20.

[0105] The system will match the current fusion time node with the previously determined abnormal impact period. Assume that through the method of Example 4, 14:00-14:30 is determined to be the abnormal impact period. Because 14:20 is within the abnormal impact period of 14:00-14:30, the system will generate a pause fusion instruction to stop the current data fusion operation to avoid being affected by data quality anomalies and ensure the accuracy of the learning situation analysis report.

[0106] If the current fusion time node is outside the anomaly impact period, for example, the current fusion time node is 3:00 PM, but the anomaly impact period is 2:00 PM-2:30 PM, the system generates a pending match instruction. Based on this pending match instruction, the system locates the predicted anomaly start point closest to the current fusion time node on the timeline. Assume that based on previous calculations and predictions, the predicted anomaly start point closest to 3:00 PM is 4:00 PM.

[0107] The system then calculates the interval between the two. The formula used here is: Interval = Predicted Anomaly Start Time - Current Fusion Time Node Time, where "Predicted Anomaly Start Time" represents the predicted anomaly start moment closest to the current fusion time node, and "Current Fusion Time Node Time" represents the time point obtained when the system performs the data fusion task. In this example, 16:00 is converted to minutes as 16 × 60 = 960 minutes, and 15:00 is converted to minutes as 15 × 60 = 900 minutes, resulting in an interval of 960 - 900 = 60 minutes.

[0108] Assuming the preset safety threshold is 30 minutes, since 60 minutes is greater than 30 minutes, the system will generate a continue fusion instruction, allowing the data fusion operation to continue at 15:00, so that accurate learning situation analysis reports can be generated in a timely manner to provide support for teaching decisions.

[0109] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0110] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A heterogeneous data source fusion analysis system, characterized by: include: Data integration module: obtains data quality parameters of distributed storage nodes associated with heterogeneous data sources over multiple data collection cycles, generates data quality assessment values ​​based on multi-dimensional analysis of the data quality parameters, determines the data fusion reliability of the distributed storage nodes based on the data quality assessment values, and generates priority instructions, wherein the instructions include high-priority adjustment instructions and low-priority adjustment instructions; comparing the data quality assessment value with the data quality assessment threshold; If the data quality assessment value is greater than the data quality assessment threshold, a high priority adjustment instruction is generated; If the data quality assessment value is less than or equal to the data quality assessment threshold, a low priority adjustment instruction is generated; Feature Analysis Module: Based on high-priority adjustment instructions, it extracts the time nodes where data quality anomalies occur in distributed storage nodes within multiple data collection cycles. It then performs correlation analysis based on the time nodes and data quality parameters to generate data feature regularity values. Based on these values, it determines whether the data quality anomaly exhibits periodic characteristics. If so, it generates regularity adjustment instructions. The time point of data quality anomaly includes the specific time point when the anomaly occurs; Fusion evaluation module: Based on regular adjustment instructions, it determines the abnormal impact period within the data fusion cycle and matches it with the current fusion time node. According to the matching results, it dynamically adjusts the fusion process of heterogeneous data sources and generates adjustment instructions, among which the adjustment instructions include pause fusion instructions and continue fusion instructions.

2. The heterogeneous data source fusion analysis system according to claim 1, characterized in that: Data quality parameters include data consistency indicators and data integrity indicators; Based on independent analysis of data consistency indicators and data integrity indicators, a consistency assessment value and an integrity assessment value are generated respectively; The consistency assessment value and the integrity assessment value are weighted and fused to generate a data quality assessment value.

3. The heterogeneous data source fusion analysis system according to claim 2, characterized in that: The data consistency indicators of distributed storage nodes in multiple data collection cycles are integrated and the average is calculated to generate a consistency mean. The consistency mean is proportional to the total duration of the data collection cycle to obtain the consistency frequency. The consistency frequency is normalized with the consistency frequency threshold to generate a consistency evaluation value.

4. The heterogeneous data source fusion analysis system according to claim 2, characterized in that: The data integrity indicators of distributed storage nodes in multiple data collection cycles are accumulated to generate the integrity sum, the integrity sum is proportional to the number of data collection cycles to obtain the integrity mean, and the integrity mean is normalized with the total duration of the data collection cycle to generate the integrity assessment value.

5. The heterogeneous data source fusion analysis system according to claim 1, characterized in that: The data characteristic regularity value is generated as follows: Generate data stability performance value and data change performance value; The data stability performance value and the data change performance value are linearly combined through preset weight coefficients to generate data characteristic regularity values.

6. The heterogeneous data source fusion analysis system according to claim 5, characterized in that: The method for generating data stability performance value is: Screen historical data collection periods where both data consistency and data integrity indicators are within normal ranges, mark them as stable periods, and serialize and mark the abnormal data quality time points in each stable period in chronological order; Calculate the time interval deviation of adjacent time points based on the serialized marked time points; Compare the time interval deviation with the preset interval threshold. If the deviation is less than or equal to the threshold, it is marked as a stable interval. The ratio of the number of stable intervals to the total number of intervals is counted. If the ratio exceeds a preset ratio threshold, the stable period is marked as a comprehensive stable period. The ratio of the number of comprehensive stable periods to the total number of historical data collection periods is counted to generate the data stability performance value.

7. The heterogeneous data source fusion analysis system according to claim 5, characterized in that: The data change performance value is generated as follows: Extract the interval time of adjacent data quality abnormal time points within all comprehensive stable periods to form an interval time set; Calculate the variance of each group of interval time in the interval time set and take the average value to generate the variance mean, which is marked as the data change performance value.

8. The heterogeneous data source fusion analysis system according to claim 5, characterized in that: Compare the data characteristic regularity value with the characteristic regularity threshold; If the data characteristic regularity value is greater than the characteristic regularity threshold, a regularity adjustment instruction is generated; If the data characteristic regularity value is less than or equal to the characteristic regularity threshold, no action is triggered.

9. The heterogeneous data source fusion analysis system according to claim 1, characterized in that: The abnormal impact period within the data fusion cycle is determined as follows: Extract the start and end time points of the data fusion cycle and mark them on the time axis; Obtain the most recent abnormal data quality time point before the start time point of the data fusion cycle and use it as the benchmark point; Calculate the average of the interval time of historical data quality anomalies, generate the interval benchmark value, take the benchmark point as the starting point, and mark the predicted anomaly starting points on the time axis in sequence according to the interval benchmark value; The preset abnormality duration is superimposed after each predicted abnormality starting point to generate a predicted abnormality ending point; The period between the predicted anomaly start point and the corresponding predicted anomaly end point is marked as the predicted anomaly period, and the overlapping part of the time range of the data fusion cycle and the predicted anomaly period is marked as the anomaly impact period.

10. The heterogeneous data source fusion analysis system according to claim 9, characterized in that: When executing the data fusion task, obtain the current fusion time node and match it with the abnormal impact period; If the current fusion time node is within the abnormal impact period, a pause fusion instruction is generated; If the current fusion time node is outside the abnormal impact period, a matching instruction is generated; Based on the instruction to be matched, locate the predicted anomaly starting point closest to the current fusion time node on the time axis and calculate the interval between the two; If the interval duration is greater than or equal to the preset safety threshold, a continue fusion instruction is generated; If the interval duration is less than the preset safety threshold, a pause fusion instruction is generated.

Images