Financial heterogeneous data real-time cleaning and fusion method

Abstract

The invention discloses a financial heterogeneous data real-time cleaning and fusion method, and the method comprises the following steps: S1, obtaining multi-source, multi-time-dimension and multi-granularity financial heterogeneous original data, building a data access channel, and executing format standardization processing; and S2, constructing a cross-source, cross-time and cross-granularity financial index consistency constraint space, dynamically learning systematic deviation distribution of each data source in different market states based on the constraint space, and generating an implicit consistency deviation field. According to the financial heterogeneous data real-time cleaning and fusion method, a cross-source, cross-time and cross-granularity financial index consistency constraint space is constructed, systematic deviation distribution of data sources in different market states is dynamically learned based on the space, and an implicit consistency deviation field is generated; the problem of consistency among multi-source heterogeneous data is effectively solved, and the accuracy and reliability of the data are improved.

Description

Technical Field

[0001] This invention relates to the field of financial data processing, and more specifically, to a method for real-time cleaning and fusion of heterogeneous financial data. Background Technology

[0002] In the financial sector, with the rapid development of information technology and the increasing complexity of financial markets, financial data exhibits characteristics of multiple sources, multiple time dimensions, and multiple granularities. Data sources are extensive, including securities trading systems, futures market data systems, bank wealth management platforms, financial information terminals, and third-party data service providers; time dimensions are diverse, covering different time scales such as high-frequency tick-level, millisecond-level sampling, minute-level, hourly-level, daily-level, weekly-level, and monthly-level; data granularity also varies, and data types are rich, including structured transaction data, order queue data, account holding data, semi-structured financial statement summary data, industry research report keyword data, and unstructured news text data, policy interpretation audio transcription data, etc.

[0003] However, due to differences in data collection methods, storage formats, and calculation standards among different data sources, these heterogeneous financial data exhibit inconsistencies in format, accuracy, and semantics. Directly using this data for analysis and decision-making may introduce biases, affecting the accuracy and reliability of the results. Furthermore, the dynamic changes in financial markets necessitate timely data processing; real-time acquisition and processing of this data are crucial for timely understanding of market dynamics and making accurate decisions.

[0004] Existing financial data processing methods often struggle to simultaneously meet the real-time cleaning and fusion requirements of heterogeneous data from multiple sources, across multiple time dimensions, and at multiple granularities. Some methods may focus only on a single data source or a specific type of data, failing to handle the complex and diverse heterogeneous financial data; others may lack effective constraints on data consistency during processing, resulting in biases in the cleaned data; still others are inadequate in terms of real-time performance, failing to meet the demands of rapidly changing financial markets. Therefore, a method is needed that can efficiently, accurately, and in real-time clean and fuse heterogeneous financial data. Summary of the Invention

[0005] The purpose of this invention is to provide a method for real-time cleaning and fusion of heterogeneous financial data, which solves the problem that existing financial data processing methods often cannot simultaneously meet the requirements for real-time cleaning and fusion of heterogeneous data from multiple sources, multiple time dimensions, and multiple granularities.

[0006] This invention achieves the above objective through the following technical solution: a method for real-time cleaning and fusion of heterogeneous financial data, the method comprising the following steps: S1. Acquire heterogeneous raw financial data from multiple sources, across multiple time dimensions, and at multiple granularities, establish data access channels, and perform format standardization processing; S2. Construct a consistency constraint space for financial indicators across sources, time periods, and granularities. Based on this constraint space, dynamically learn the systematic deviation distribution of each data source under different market conditions to generate an implicit consistency deviation field. S3. During the real-time data inflow phase, calculate the offset vector of a single financial data item in the implicit consistency deviation field in real time. S4. Perform continuous and adaptive correction on a single piece of financial data according to the offset vector to complete real-time data cleaning; S5. Perform feature alignment and fusion processing on the cleaned heterogeneous financial data to output standardized fused data.

[0007] Furthermore, step S1 includes the following steps: The system synchronously collects heterogeneous raw financial data from securities trading systems, futures market data systems, bank wealth management platforms, financial information terminals, and third-party data service providers through a distributed high-concurrency data access interface. The data covers stocks, bonds, funds, and derivatives-related data. The data types include structured transaction data, order queue data, account position data, semi-structured financial statement summary data, industry research report keyword data, and unstructured news text data and policy interpretation audio transcription data. The data granularity covers different time scales, including high-frequency tick-level, millisecond-level sampling, minute-level, hour-level, daily-level, weekly-level, and monthly-level.

[0008] Furthermore, in step S1, the format standardization process specifically includes: By using regular expression matching and protocol parsing technology, non-standard timestamps from different data sources are uniformly converted into the ISO 8601 standard time format; The precision of various numerical data is normalized and uniformly converted into a floating-point representation with 6 decimal places, while also handling abnormal format characters in the data; For unstructured text data, entity recognition, keyword extraction, and semantic structuring techniques from natural language processing are used to extract core financial information and convert it into structured data in key-value pair form, ultimately forming a standardized raw data set.

[0009] Furthermore, step S2 includes the following steps: Before constructing a consistency constraint space for financial indicators, a set of core financial indicators with clear business meaning and cross-data source comparability is selected based on the core business scenario requirements of the financial market. Core financial metrics include daily returns, trading volume growth rate, annualized volatility, price-to-earnings ratio, and price-to-book ratio for stocks; yield to maturity, duration, and convexity for bonds; and net asset value growth rate, Sharpe ratio, and maximum drawdown for funds.

[0010] Furthermore, step S2 includes the following steps: The consistency constraint space of financial indicators has a three-dimensional structure, including the source constraint dimension, the time constraint dimension, and the granularity constraint dimension. Constraint rules are established in three dimensions: data source, time dimension, and granularity level. The corresponding constraint thresholds for each dimension are determined through historical data statistical analysis to ensure the rationality and adaptability of the constraint rules.

[0011] Furthermore, step S2 includes the following steps: When generating the implicit consistency deviation field, a time-series attention mechanism is used to capture the long-term and short-term time-series dependencies of the data based on a historical standardized dataset. A Gaussian mixture model is used to model the deviation distribution of each data source, and the systematic deviation distribution of different data sources under different market conditions is dynamically learned. Then, the discrete systematic deviation distribution is mapped into a continuous and smooth implicit consistency deviation field through the kernel density estimation method.

[0012] Furthermore, in step S3, the calculation of the offset vector includes the following steps: Extract observations of core financial indicators from single standardized data points that flow in real time; Based on the implicit consistency deviation field, query the deviation distribution parameters corresponding to this data; The expected value of each core indicator is calculated by combining the constraint rules in the consistency constraint space, and then the observation bias of each core indicator is obtained. The observation bias is normalized to construct an offset vector, and the magnitude of the offset vector comprehensively reflects the overall offset degree of a single data point.

[0013] Furthermore, step S4 includes the following steps: Adaptive correction is achieved by constructing a dynamic adaptive correction function, which dynamically adjusts the correction intensity based on the magnitude of the offset vector. During the correction process, the hyperbolic tangent function is used to adjust the correction coefficients to avoid discretized outlier data removal operations and maintain the temporal continuity and integrity of the data. The corrected data needs to be substituted into a three-dimensional consistency constraint space for full-dimensional consistency verification.

[0014] Furthermore, consistency verification includes: Source constraint verification, time constraint verification, granularity constraint verification; If the corrected data does not meet some of the constraints, the smoothing coefficient of the correction function is dynamically adjusted based on the degree of constraint deviation, and iterative correction is performed until the data meets all constraints or reaches the preset maximum number of correction iterations.

[0015] Furthermore, in step S5, the feature alignment process includes the following steps: Establish globally unified feature dimensions based on core financial indicator types, with each feature dimension clearly defining its data type, calculation standard, accuracy requirements, and business implications; The fusion process adopts a weighted fusion strategy based on data source credibility. The data source credibility weight is calculated by combining the variance of the systematic deviation distribution over the past 180 days and the stability coefficient of the data source over the past 90 days, and the fused index value is obtained by weighted average method. The fused data is processed to standardize the format, generating standardized fused data containing complete source traceability information, supporting real-time querying and batch export in multiple formats.

[0016] The beneficial effects of this invention are as follows: 1. This invention can acquire and process heterogeneous raw financial data from multiple sources, including securities trading systems, futures market data systems, bank wealth management platforms, financial information terminals, and third-party data service providers. The data covers multiple fields such as stocks, bonds, funds, and derivatives, and includes structured, semi-structured, and unstructured data. The data granularity covers different time scales, enabling comprehensive processing of complex and diverse data in the financial field.

[0017] 2. By constructing a consistency constraint space for financial indicators across sources, time periods, and granularities, and dynamically learning the systematic deviation distribution of each data source under different market conditions based on this space, an implicit consistency deviation field is generated. During the real-time data inflow stage, the offset vector of a single financial data point is calculated in real time, and the data is continuously and adaptively corrected based on the offset vector. This effectively solves the consistency problem between multi-source heterogeneous data and improves the accuracy and reliability of the data.

[0018] 3. This invention performs real-time calculation and correction during the real-time data inflow stage, which can process newly arriving data in a timely manner and meet the timeliness requirements of the financial market. Through a dynamic adaptive correction function, the correction intensity is dynamically adjusted according to the magnitude of the offset vector, avoiding discretized abnormal data removal operations, maintaining the temporal continuity and integrity of the data, and providing strong support for real-time analysis and decision-making.

[0019] 4. When performing feature alignment and fusion processing on the cleaned heterogeneous financial data, a globally unified feature dimension is established according to the core financial indicator types. The data types, calculation standards, accuracy requirements, and business meanings are clearly defined to ensure the consistency and standardization of the fused data. A weighted fusion strategy based on data source credibility is adopted. The data source credibility weight is calculated by comprehensively considering the variance of the systematic deviation distribution over the past 180 days and the stability coefficient of the data source over the past 90 days. The fused indicator value is obtained by weighted averaging, which improves the quality and credibility of the fused data.

[0020] 5. The generated standardized fusion data contains complete source traceability information, supports real-time querying and batch export in multiple formats, making it convenient for users to trace and analyze the data, and improving the usability and practicality of the data. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating the overall method of the present invention; Figure 2 This is a flowchart of the three-dimensional constrained space structure of the present invention; Figure 3 This is a flowchart of the adaptive correction process of the present invention. Detailed Implementation

[0022] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0023] Example 1: Please see Figure 1-3 This invention provides a technical solution: a method for real-time cleaning and fusion of heterogeneous financial data, the method comprising: S1. Acquire heterogeneous raw financial data from multiple sources, across multiple time dimensions, and at multiple granularities, establish data access channels, and perform format standardization processing; Among these, "multiple sources" refers to financial data originating from various channels, such as stock exchanges, banking systems, financial information platforms, and third-party data providers. Data from different sources may vary in data structure, quality, and update frequency. "Multiple time dimensions" encompasses data across different time spans and granularities, such as data at different time frequencies (minute, hour, day, week, month, etc.) and different time ranges (historical and real-time data, etc.). "Multiple granularities" indicates varying levels of detail; for example, stock data may include company-wide data (total market capitalization, total share capital, etc.) or data broken down to specific business segments or products. Transaction data may include individual transaction data or transaction data aggregated at certain time intervals. "Heterogeneous financial raw data" refers to data from various sources... Financial data from the same source but with different structures, formats, and semantics differ in data type, representation, and meaning. Examples include structured tabular data, semi-structured log data, and unstructured text data. Data access channels are used to transmit heterogeneous raw financial data from different sources to the data processing system. These channels may include network interfaces, data interfaces, message queues, and other forms to ensure stable and efficient data flow into the system. Format standardization converts heterogeneous raw financial data from different sources and in different formats into a unified, standardized format for subsequent data processing and analysis. S2. Construct a consistency constraint space for financial indicators across sources, time periods, and granularities. Based on the constraint space, dynamically learn the systematic deviation distribution of each data source under different market conditions to generate an implicit consistency deviation field. Among these, "cross-source, cross-time, and cross-granularity" emphasizes that the constructed financial indicator consistency constraint space must comprehensively consider data from different sources, time dimensions, and granularities, breaking down these boundaries between data to achieve comprehensive constraints and consistency processing. The financial indicator consistency constraint space is an abstract mathematical space or conceptual model used to define and measure the consistency relationship between different financial indicators. Within this space, various rules and conditions can be set to ensure that financial indicators from different sources, time periods, and granularities remain logically and numerically consistent. Systematic deviation distribution refers to the distribution of systematic differences between data from different data sources and their true or reference values ​​under different market conditions. This deviation may be caused by various factors such as data collection methods, data processing procedures, and market characteristics. The implicit consistency deviation field is a conceptual field dynamically learned based on the financial indicator consistency constraint space and systematic deviation distribution. It describes the potential consistency deviations between data sources under different market conditions. It can be understood as a virtual field containing various deviation information, used for subsequent correction of data deviations. S3. During the real-time data inflow phase, calculate the offset vector of a single financial data point in the implicit consistency deviation field in real time. The real-time data inflow stage refers to the continuous inflow of financial data into the data processing system, emphasizing the real-time and dynamic nature of the data and requiring the system to process newly arriving data promptly. The offset vector, in the implicit consistency deviation field, is a vector used to represent the degree and direction of deviation of a single piece of financial data from its consistency state. By calculating the offset vector, the deviation of the data can be quantified, providing a basis for subsequent corrections. S4. Perform continuous and adaptive correction on a single financial data point based on the offset vector to complete real-time data cleaning. Among them, continuous and adaptive correction are defined as follows: continuous correction means that the correction of financial data is a continuous process, not a discrete, one-off operation. As new data continues to flow in and market conditions change, the correction process continues to ensure the accuracy and consistency of the data; adaptive correction means that the correction method can automatically adjust the correction parameters and strategies according to different market conditions, data source characteristics, and changes in the implicit consistency deviation field, so as to effectively correct data deviations under different conditions; real-time data cleaning is to promptly correct individual financial data with deviations during the real-time data inflow process, remove errors, noise, and inconsistent information from the data, ensure data quality and usability, and provide reliable data support for subsequent data analysis and decision-making. S5. Perform feature alignment and fusion processing on the cleaned heterogeneous financial data to output standardized fused data; Feature alignment is crucial because heterogeneous financial data originates from diverse sources, possessing varying feature representations and semantics. The goal of feature alignment is to match and unify relevant features from different data sources, ensuring comparability and operability within the same semantic framework. For example, aligning features representing stock prices from different data sources guarantees they represent the same concepts and calculation methods. Fusion processing integrates and merges the cleaned, feature-aligned heterogeneous financial data, generating a comprehensive and standardized dataset. Fusion processing considers factors such as the weights and correlations of different data sources, fully leveraging their strengths and enhancing their information content and value. Standardized fused data, obtained after feature alignment and fusion processing, possesses a unified format, structure, and semantics. This facilitates subsequent analysis, modeling, and application, providing a consistent and reliable data foundation for various business operations and decision-making in the financial sector.

[0024] It should be noted that during use, the acquisition and standardized processing of data from multiple sources, dimensions, and granularities can comprehensively collect information and unify formats, laying the foundation for subsequent processing. It constructs a consistency constraint space for financial indicators and generates an implicit consistency deviation field, which can accurately grasp the deviation patterns of different data sources under different market conditions, providing a scientific basis for data correction. When data flows in in real time, it calculates the offset vector and performs continuous and adaptive correction, which can eliminate data deviations in a timely manner, ensuring data timeliness and accuracy. After cleaning, the data is feature aligned and fused to output standardized fused data, eliminating data heterogeneity and making the data more consistent and comparable, which facilitates subsequent analysis and applications. It can better support financial decision-making, risk assessment and other business operations, and improve the efficiency and value of financial data processing.

[0025] In one embodiment, acquiring heterogeneous raw financial data from multiple sources, across multiple time dimensions, and at multiple granularities, establishing a data access channel, and performing format standardization processing includes: Through distributed high-concurrency data access interfaces, heterogeneous raw financial data is collected synchronously from multiple channels such as securities trading systems, futures market data systems, bank wealth management platforms, financial information terminals, and third-party data service providers. This covers data related to diverse financial products such as stocks, bonds, funds, and derivatives. The data types specifically include structured transaction data, order queue data, and account holding data; semi-structured financial report summary data and industry research report keyword data; as well as unstructured news text data and policy interpretation audio transcription data. The data granularity comprehensively covers different time scales such as high-frequency tick level, millisecond level sampling, minute level, hour level, daily level, weekly level, and monthly level. Perform full-process format parsing and standardization processing on the collected heterogeneous raw financial data: By using regular expression matching and protocol parsing technology, non-standard timestamps from different data sources, such as Unix timestamps and string-formatted times, are uniformly converted into the ISO 8601 standard time format. It performs precision normalization on various numerical data, including prices, trading volume, open interest, and returns, and converts them into floating-point representations with 6 decimal places. It also handles abnormal format characters in the data, such as comma-separated thousands separators and Chinese numerals. For unstructured text data, entity recognition, keyword extraction, and semantic structuring techniques from natural language processing are used to extract core financial information such as company names, financial indicator values, and policy impact areas, and convert them into structured data in key-value pair form. The final standardized raw data set is formed:

[0026] in, This represents the total amount of data after standardization. Indicates the first Each piece of standardized raw data contains a unique source identifier. Source identification Used to trace the data source and standard timestamp. Clear granularity levels and complete core field data Core field data It covers key information such as the financial product identifier, indicator type, and specific value corresponding to the data.

[0027] This design utilizes distributed, high-concurrency interfaces to collect data of different types and granularities from diverse financial products through multiple channels. It then performs format parsing and standardization to form a standardized raw data set, comprehensively covering various types of financial data and meeting the multi-dimensional data needs of financial businesses. The distributed, high-concurrency interfaces can efficiently process large amounts of data, ensuring timely data collection. Format standardization eliminates differences in time, numerical format, and text structure, making the data consistent and comparable. This lays a solid foundation for subsequent data processing and analysis, facilitates unified management and application, improves data processing efficiency, and reduces errors and confusion caused by inconsistent data formats.

[0028] In one embodiment, a consistency constraint space for financial indicators is constructed across sources, time periods, and granularities. Based on this constraint space, the systematic deviation distribution of each data source under different market conditions is dynamically learned to generate an implicit consistency deviation field, including: Based on the core business needs of the financial market, such as quantitative trading, risk pricing, and asset allocation, a set of core financial indicators with clear business implications and cross-data source comparability is selected:

[0029] in, This indicates the number of core financial indicators, which specifically include key indicators such as daily returns, trading volume growth rate, annualized volatility, price-to-earnings ratio, and price-to-book ratio for stocks; yield to maturity, duration, and convexity for bonds; and net asset value growth rate, Sharpe ratio, and maximum drawdown for funds. Constructing a three-dimensional consistency constraint space for financial indicators:

[0030] Constraint rules are established for the three core dimensions of data source, time, and granularity level, and the threshold rules for each dimension are as follows: (Source constraint dimension): Calculated by Pearson correlation coefficient Correlation coefficient with Spearman's rank The average of the two values ​​is taken as the comprehensive correlation strength of the same core indicator between different data sources. The calculation formula is as follows:

[0031] Correlation threshold Adaptive rule determination is adopted: Based on 365 days of historical multi-source data, the comprehensive correlation coefficient distribution of all data source pairs is calculated. The 75th quantile of the distribution is taken as the initial threshold, and then dynamically updated every 90 days according to the correlation distribution of newly added data, with the update increment not exceeding the initial threshold. To ensure threshold stability; for any two data sources and If their comprehensive correlation coefficient of the same core indicator If the two are determined to be a strongly correlated source pair, the indicator data between the strongly correlated source pairs should meet the consistency constraint, and the deviation should be controlled within the preset range. (Time Constraint Dimension): Taking full account of the time-series continuity of financial market data, based on three years of historical market state data (including bull markets, bear markets, and sideways markets), the rate of change distribution of each core indicator in adjacent time windows under different market states is statistically analyzed, and the 95th quantile of each distribution is calculated as the threshold for the rate of change of the indicator under the corresponding market state. , Represents the market state, that is, for any consecutive points in time. and , For the corresponding granularity time interval, the core metrics The rate of change must satisfy:

[0032] Among them, the yield indicator in a bull market The initial value is set to 0.2, 0.25 for bear markets, and 0.15 for sideways markets; the volume indicator in bull markets... The initial value is set to 0.5, 0.6 for bear markets, and 0.4 for sideways markets. All thresholds are dynamically calibrated quarterly based on market fluctuations. Values ​​exceeding the corresponding thresholds are considered abnormal fluctuations in the time series. (Granularity constraint dimension): Establishing mathematical aggregation mapping relationships between data of different granularities. To achieve consistent aggregation computation from high-granularity data to low-granularity data, and to set an aggregation consistency threshold. The aggregation consistency threshold is the error threshold between the aggregated data and the directly collected low-granularity data. Rule of thumb: Calculate the absolute error between high-granularity aggregated data and directly collected low-granularity data over the past 180 days. The 90th quantile of the error distribution is taken as ,and The error should not exceed 5% of the average of directly collected data. For example, when aggregating minute-level data into hourly-level data, price indicators should use a time-weighted average method, cumulative indicators such as trading volume and turnover should use a summation method, and statistical indicators such as volatility should use a rolling calculation method to ensure that the error between the aggregated low-granularity data and the directly collected low-granularity data does not exceed 5%. ; Based on historical standardized datasets:

[0033] The total historical data volume must include at least three years of complete data. A temporal attention mechanism is employed to capture the long-term and short-term temporal dependencies of the data. A Gaussian mixture model is combined to model the bias distribution of each data source, dynamically learning the different data sources under different market conditions. Distribution of systematic deviations Market conditions Market conditions are categorized into bull markets, bear markets, and sideways markets, using indicators such as the magnitude of market rises and falls and volatility levels. Indicates data source In market conditions The actual deviation value of a certain core indicator is given below. This distribution can accurately depict the systematic deviation pattern of the data source under a specific market environment. The kernel density estimation method is used to map the learned discrete systematic deviation distribution into a continuous and smooth implicit consistency deviation field. ,in This is a three-dimensional feature vector of the data, where the three-dimensional feature vectors correspond to the data source, timestamp, and granularity level, respectively. As the core mapping function, it can output the deviation distribution parameters corresponding to the feature vector. ,in This represents the mean deviation of the data under this feature combination. This represents the standard deviation of the deviation, which can be used to characterize the reasonable deviation range of any data in the current financial environment in real time.

[0034] This design constructs a three-dimensional financial indicator consistency constraint space, dynamically learns the systematic deviation distribution to generate an implicit consistency deviation field, and comprehensively constrains financial indicators from three key dimensions: source, time, and granularity. This ensures data consistency and rationality. By calculating correlation coefficients and setting adaptive thresholds, it can accurately determine the correlation between data sources, effectively identify abnormal data, determine the indicator change rate threshold by considering market conditions and time series characteristics, and establish aggregation mapping relationships and consistency thresholds. This can accurately capture the temporal and granular change patterns of data, dynamically learn the deviation distribution and generate a deviation field, and reflect the reasonable deviation range of data in a specific environment in real time, providing a scientific basis for data cleaning.

[0035] In one embodiment, during the real-time data inflow phase, the offset vector of a single financial data point in the implicit consistency deviation field is calculated in real time, including: For a single piece of standardized data that flows in in real time through the data access channel:

[0036] Based on its core field data Based on preset indicator extraction rules, the observed values ​​of various core financial indicators corresponding to this data are accurately extracted. ,in Indicates the first Real-time observations of key indicators; Based on the constructed implicit consistency deviation field Input the three-dimensional feature vector of the current data:

[0037] By quickly searching and interpolating, the deviation distribution parameters corresponding to the data can be obtained in real time. ,in This is the vector of the mean deviations of each core indicator. This is the vector of standard deviations of each core indicator; For each core metric Based on consistency constraint space Based on the constraints in the data, combined with historical data from the same source, data from strongly correlated sources from the same period, and data from adjacent time windows, a weighted average method is used to calculate the expected value of the indicator in the current scenario. The weighting rule is as follows: weights of historical data from the same source over the past 30 days. Strongly correlated sources have weights on data from the same period. adjacent time window data weights ,and If there are at least two strongly correlated source pairs, then the average of the data from each strongly correlated source is taken before assigning the value. Weights; and then the observation bias of each core indicator is calculated. , This intuitively reflects the degree of deviation between real-time observations and reasonable expected values; Based on the observation bias of each core indicator Based on this, combined with the corresponding standard deviation Perform normalization to construct the offset vector:

[0038] Each component of the offset vector represents the standardized offset of the corresponding index observation relative to the bias field. The magnitude of the offset vector is:

[0039] It comprehensively reflects the overall deviation of a single data point; the larger the modulus, the more significant the deviation of the data from the reasonable range.

[0040] This design allows for real-time calculation of the offset vector of a single financial data point within the implicit consistency bias field. It precisely quantifies the degree of deviation of a single data point from a reasonable range. By extracting core indicator observations and quickly querying the deviation distribution parameters using the implicit consistency bias field, it combines various data to calculate the expected value, thereby obtaining the observed bias. Normalizing the observed bias constructs an offset vector that comprehensively considers the deviation of each indicator. The magnitude of the offset vector directly reflects the overall degree of data deviation, providing a clear direction and quantitative indicator for subsequent data correction. This helps improve the accuracy and relevance of data cleaning, ensuring data quality.

[0041] In one embodiment, continuous and adaptive correction is performed on a single piece of financial data based on the offset vector to complete real-time data cleaning, including: Based on the calculated offset vector Construct a dynamic adaptive correction function This function can dynamically adjust the correction strength according to the degree of data offset, ensuring the continuity and rationality of the correction process. The specific expression is as follows:

[0042] in, It is a hyperbolic tangent function, and its output range is within... Between these values, the correction coefficients are dynamically adjusted when the offset vector magnitude is [value missing]. When the modulus is small, the data deviation is small and the correction strength is weak; when the modulus is large, the data deviation is large and the correction strength is correspondingly stronger. This is the local minimum constant, which takes the value 1e-8. It is used to avoid the case where the denominator is zero when the magnitude of the offset vector is 0, thus ensuring the stability of the function. Core field data obtained through the correction function Re-extract the corrected values ​​of each core indicator. And substitute it into the three-dimensional consistency constraint space Perform full-dimensional consistency verification: Source constraint verification: Check whether the overall correlation between the corrected data and the strongly correlated sources on the contemporaneous data is satisfied. ; Time constraint verification: Check whether the rate of change of the index between the corrected data and the data in adjacent time windows meets the requirements. , This reflects the current market situation. Granularity constraint verification: If the data is high-granularity, check whether the error between the aggregated low-granularity data and the directly collected low-granularity data does not exceed the specified value. ; If the corrected data meets all constraints, it is directly marked as valid cleaned data; if it does not meet some constraints, it is judged based on the degree of constraint deviation. Dynamically adjust the smoothing coefficient of the correction function The rules have been adjusted to That is, the greater the degree of deviation, The larger the value, the greater the maximum value, not exceeding 2; re-sort Substituting into the correction function, the correction function is modified as follows:

[0043] Perform iterative correction until the data meets all constraints or reaches the preset maximum number of correction iterations. The maximum number of correction iterations is set to 3 by default and can be adjusted from 1 to 5 according to the real-time requirements of the business. Throughout the entire correction process, the data is continuously adjusted without discretization of outlier data removal, thus preserving the temporal continuity and integrity of the data to the greatest extent possible and avoiding information loss due to data removal.

[0044] This design performs continuous and adaptive correction on individual financial data points based on the offset vector to complete real-time cleaning. The correction process is continuous and adaptive, dynamically adjusting the correction intensity according to the degree of data offset to avoid over-correction or under-correction. The dynamic adaptive correction function combined with the hyperbolic tangent function ensures that the correction intensity is reasonably matched with the degree of offset. Through full-dimensional consistency verification, it is ensured that the corrected data meets various constraints. If not, the smoothing coefficient is dynamically adjusted for iterative correction to maximize the preservation of the temporal continuity and integrity of the data. This design can effectively improve the quality and efficiency of data cleaning, reduce human intervention, and make the cleaned data more in line with the actual needs of financial business.

[0045] In one embodiment, the cleaned heterogeneous financial data undergoes feature alignment and fusion processing to output standardized fused data, including: For the effective dataset obtained after real-time cleaning:

[0046] Perform unified feature alignment processing based on core financial indicator types to establish a globally unified feature dimension. ,in This represents the total number of features after fusion. Each feature dimension clearly defines the data type, calculation standard, accuracy requirements, and business meaning. A weighted fusion strategy based on data source credibility is adopted to perform fusion calculations on multi-source cleaned data of the same financial product, the same timestamp, and the same indicator: Based on the variance of the systematic bias distribution of each data source over the past 180 days Calculate the initial credibility weight of the data source:

[0047] To account for the number of data sources participating in the integration, a data source stability coefficient is introduced. Data source stability coefficient Calculated based on the availability and latency of data sources over the past 90 days.

[0048] Availability = Normal data output duration / Total duration; Latency rate = Actual latency duration / Allowable latency duration; Reliability weight:

[0049] All weights are normalized to ensure Then, the merged index value is calculated using the weighted average method:

[0050] in For data source The index values ​​after cleaning; The merged indicator data undergoes final format standardization processing to unify the data storage format, encoding method, and accuracy requirements, generating standardized merged data:

[0051] in, This represents the total amount of data after fusion. Each piece of fused data contains complete source traceability information. The source traceability information records the identifiers of all data sources participating in the fusion, standard timestamps, granularity levels, and core indicator data after fusion. It also supports real-time querying with a response time of no more than 100 milliseconds and batch export. Batch export supports multiple formats such as CSV, Parquet, and JSON to meet the data usage needs of different business scenarios.

[0052] This design performs feature alignment and fusion processing on cleaned heterogeneous financial data to output standardized fused data. Feature alignment establishes a globally unified feature dimension, ensuring that data from different sources have the same semantics and structure, facilitating subsequent analysis and application. The weighted fusion strategy based on data source credibility comprehensively considers the deviation distribution and stability of data sources, enabling more reasonable fusion of multi-source data and improving data accuracy and reliability. The final standardized format processing, comprehensive source traceability information, and support for real-time querying and batch export of multiple formats meet the data needs of different business scenarios, improve data availability and sharing, and provide stronger support for financial decision-making.

[0053] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0054] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for real-time cleaning and fusion of heterogeneous financial data, characterized in that, The method includes the following steps: S1. Acquire heterogeneous raw financial data from multiple sources, across multiple time dimensions, and at multiple granularities, establish data access channels, and perform format standardization processing; S2. Construct a consistency constraint space for financial indicators across sources, time periods, and granularities. Based on this constraint space, dynamically learn the systematic deviation distribution of each data source under different market conditions to generate an implicit consistency deviation field. S3. During the real-time data inflow phase, calculate the offset vector of a single financial data item in the implicit consistency deviation field in real time. S4. Perform continuous and adaptive correction on a single piece of financial data according to the offset vector to complete real-time data cleaning; S5. Perform feature alignment and fusion processing on the cleaned heterogeneous financial data to output standardized fused data.

2. The method for real-time cleaning and fusion of heterogeneous financial data according to claim 1, characterized in that, Step S1 includes the following steps: The system synchronously collects heterogeneous raw financial data from securities trading systems, futures market data systems, bank wealth management platforms, financial information terminals, and third-party data service providers through a distributed high-concurrency data access interface. The data covers stocks, bonds, funds, and derivatives-related data. The data types include structured transaction data, order queue data, account position data, semi-structured financial statement summary data, industry research report keyword data, and unstructured news text data and policy interpretation audio transcription data. The data granularity covers different time scales, including high-frequency tick-level, millisecond-level sampling, minute-level, hour-level, daily-level, weekly-level, and monthly-level.

3. The method for real-time cleaning and fusion of heterogeneous financial data according to claim 2, characterized in that, In step S1, the format standardization process specifically includes: By using regular expression matching and protocol parsing technology, non-standard timestamps from different data sources are uniformly converted into the ISO8601 standard time format; The precision of various numerical data is normalized and uniformly converted into a floating-point representation with 6 decimal places, while also handling abnormal format characters in the data; For unstructured text data, entity recognition, keyword extraction, and semantic structuring techniques from natural language processing are used to extract core financial information and convert it into structured data in key-value pair form, ultimately forming a standardized raw data set.

4. The method for real-time cleaning and fusion of heterogeneous financial data according to claim 1, characterized in that, Step S2 includes the following steps: Before constructing a consistency constraint space for financial indicators, a set of core financial indicators with clear business meaning and cross-data source comparability is selected based on the core business scenario requirements of the financial market. Core financial metrics include daily returns, trading volume growth rate, annualized volatility, price-to-earnings ratio, and price-to-book ratio for stocks; yield to maturity, duration, and convexity for bonds; and net asset value growth rate, Sharpe ratio, and maximum drawdown for funds.

5. The method for real-time cleaning and fusion of heterogeneous financial data according to claim 4, characterized in that, Step S2 includes the following steps: The consistency constraint space of financial indicators has a three-dimensional structure, including the source constraint dimension, the time constraint dimension, and the granularity constraint dimension. Constraint rules are established in three dimensions: data source, time dimension, and granularity level. The corresponding constraint thresholds for each dimension are determined through historical data statistical analysis to ensure the rationality and adaptability of the constraint rules.

6. The method for real-time cleaning and fusion of heterogeneous financial data according to claim 5, characterized in that, Step S2 includes the following steps: When generating the implicit consistency deviation field, a time-series attention mechanism is used to capture the long-term and short-term time-series dependencies of the data based on a historical standardized dataset. A Gaussian mixture model is used to model the deviation distribution of each data source, and the systematic deviation distribution of different data sources under different market conditions is dynamically learned. Then, the discrete systematic deviation distribution is mapped into a continuous and smooth implicit consistency deviation field through the kernel density estimation method.

7. The method for real-time cleaning and fusion of heterogeneous financial data according to claim 1, characterized in that, Step S3, when calculating the offset vector, includes the following steps: Extract observations of core financial indicators from single standardized data points that flow in real time; Based on the implicit consistency deviation field, query the deviation distribution parameters corresponding to this data; The expected value of each core indicator is calculated by combining the constraint rules in the consistency constraint space, and then the observation bias of each core indicator is obtained. The observation bias is normalized to construct an offset vector, and the magnitude of the offset vector comprehensively reflects the overall offset degree of a single data point.

8. The method for real-time cleaning and fusion of heterogeneous financial data according to claim 1, characterized in that, Step S4 includes the following steps: Adaptive correction is achieved by constructing a dynamic adaptive correction function, which dynamically adjusts the correction intensity based on the magnitude of the offset vector. During the correction process, the hyperbolic tangent function is used to adjust the correction coefficients to avoid discretized outlier data removal operations and maintain the temporal continuity and integrity of the data. The corrected data needs to be substituted into a three-dimensional consistency constraint space for full-dimensional consistency verification.

9. The method for real-time cleaning and fusion of heterogeneous financial data according to claim 8, characterized in that, Consistency verification includes: Source constraint verification, time constraint verification, granularity constraint verification; If the corrected data does not meet some of the constraints, the smoothing coefficient of the correction function is dynamically adjusted based on the degree of constraint deviation, and iterative correction is performed until the data meets all constraints or reaches the preset maximum number of correction iterations.

10. The method for real-time cleaning and fusion of heterogeneous financial data according to claim 1, characterized in that, In step S5, the feature alignment process includes the following steps: Establish globally unified feature dimensions based on core financial indicator types, with each feature dimension clearly defining its data type, calculation standard, accuracy requirements, and business implications; The fusion process adopts a weighted fusion strategy based on data source credibility. The data source credibility weight is calculated by combining the variance of the systematic deviation distribution over the past 180 days and the stability coefficient of the data source over the past 90 days, and the fused index value is obtained by weighted average method. The fused data is processed to standardize the format, generating standardized fused data containing complete source traceability information, supporting real-time querying and batch export in multiple formats.

Images