Real-time processing methods based on multi-source heterogeneous data

Abstract

This invention discloses a real-time processing method based on multi-source heterogeneous data, belonging to the field of data analysis and processing technology. It uses a UTC benchmark and adjustable gain, power-law normalization to normalize multi-source timestamps while retaining metadata information. In streaming processing, it achieves reordering of out-of-order data and marking of expired data by setting buffer queues and watermark thresholds. When null fields or cross-source conflicts are detected, imputation and conflict handling are performed, outputting complete data after consistency correction. Subsequently, external high-precision references or cross-correlation functions are used to further fine-tune the timestamps at the millisecond level; if the adjustment exceeds the safety boundary, it is marked as suspicious. Finally, the corrected data is distributed to multiple nodes according to a sharding mapping strategy, and fault-tolerant consensus and difference repair are triggered under fault duration determination, thereby maintaining high-precision time-series alignment and high availability in high-concurrency scenarios, avoiding data loss or accuracy degradation caused by time-series inconsistencies and node failures. It can be widely applied to high-frequency analysis.

Description

Technical Field

[0001] This invention relates to the field of data analysis and processing technology, specifically to a real-time processing method based on multi-source heterogeneous data. Background Technology

[0002] In today's era of highly integrated data and real-time information exchange, a vast amount of heterogeneous data from various industries and fields is characterized by its massive scale, diverse structures, and high update frequency. This is particularly true in financial and industrial scenarios, where data sources may include trading systems, IoT devices, policy announcements, news feeds, and more. Due to significant differences in data precision, time zones, and release times across these channels, failure to establish a unified time stamp and multi-source alignment mechanism at the initial stage often leads to severe issues of out-of-order processing and missing data in subsequent real-time decision analysis. Simultaneously, to meet the demands of high-frequency processing at the second or even sub-millisecond level, the system not only needs to manage and tolerate large-scale concurrent data in a streaming manner but also needs to ensure alignment with external high-precision signals (such as synchronous clocks or high-frequency trading benchmarks). If effective caching and watermarking strategies are lacking, or if distributed fault tolerance is not possible in the event of failures, key indicators (such as industry return on investment or other core business parameters) are easily distorted, thus affecting the timeliness and accuracy of decision-making. Therefore, building a real-time data processing solution that can achieve a precise and unified time base under multi-source heterogeneous inputs and has high availability capabilities in streaming processing, data interpolation, high-precision alignment and distributed fault tolerance has become a key technology focus in fields such as financial transactions, industrial automation and the Internet of Things.

[0003] Chinese Patent No. CN117171534B discloses a method, system, device, and medium for acquiring multi-source heterogeneous data from CNC machine tools, belonging to the field of data acquisition technology. The method includes: identifying the data types of the multi-source heterogeneous data to be acquired and generating corresponding acquisition tasks based on the data types; executing the acquisition tasks, acquiring data from heterogeneous data sources, parsing the communication protocols used by the heterogeneous data sources, and extracting their valid data; performing dimensionality reduction processing on the valid data from the heterogeneous data sources, transforming it into interpretable low-dimensional data from the heterogeneous data sources; performing data synchronization processing on the low-dimensional data from the heterogeneous data sources; and after processing, sending the data to the data receiving end and performing data consistency verification. By optimizing the structural design and parsing different protocols, the efficiency and accuracy of data transmission can be improved, and the real-time performance and synchronization of multi-source heterogeneous data can be ensured.

[0004] However, considering the above practical application scenarios and the above existing technologies:

[0005] One of the core technical challenges in real-time processing and analysis of multi-source heterogeneous data is how to further refine and reliably distribute data that has undergone initial alignment and interpolation under different network environments and high concurrency conditions. This ensures that the data maintains a unified time base and high-quality data output even in the event of node failures or load surges. For example, if a system has completed second-level or millisecond-level time alignment of multi-source heterogeneous data, but is deployed in a distributed cluster without a multi-node fault-tolerant mechanism and a clock synchronization consensus algorithm between nodes, it will fall into a predicament of increased latency, data loss, or cross-node timestamp divergence when individual nodes fail or network jitter occurs. This prevents the high-precision alignment achieved with significant computing resources from maintaining stability in a large-scale environment. This problem is particularly prominent in the financial sector: in high-frequency trading, whether it is real-time calculation of return on investment or risk control warning, it is necessary to maintain cross-node data consistency at sub-millisecond time intervals. Once a node fails and there is no compensation mechanism, the overall computational accuracy and availability will be severely impacted, making it difficult to meet the stringent requirements of actual production scenarios.

[0006] Therefore, the present invention provides a real-time processing method based on multi-source heterogeneous data. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] To address the shortcomings of existing technologies, this invention provides a real-time processing method based on multi-source heterogeneous data. It normalizes multi-source timestamps using a UTC benchmark and adjustable gain, and power exponent, while preserving metadata information. In streaming processing, it reorders out-of-order data and marks expired data by setting buffer queues and watermark thresholds. When null fields or cross-source conflicts are detected, it performs interpolation and conflict handling, outputting complete data after consistency correction. Subsequently, it further fine-tunes the timestamps at the millisecond level using an external high-precision reference or cross-correlation function; if the adjustment exceeds the safety boundary, it is marked as suspicious. Finally, the corrected data is distributed to multiple nodes according to a sharding mapping strategy, and fault-tolerant consensus and difference patching are triggered under fault duration determination. This maintains high-precision time alignment and high availability in high-concurrency scenarios, avoiding data loss or accuracy degradation caused by time inconsistencies and node failures, thus solving the technical problems mentioned in the background.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, the present invention provides the following technical solution: a real-time processing method based on multi-source heterogeneous data, comprising,

[0011] Upon receiving multi-source heterogeneous data, based on the unique identifier assigned to each data source The time zone and precision are initially normalized using UTC reference, adjustable gain coefficient, and power exponent, along with local or server timestamp information. The converted time is then identified as a standard timestamp. At the same time, supplement the necessary metadata information;

[0012] When a standard timestamp is detected After the marked data officially enters the streaming processing engine, a buffer queue is created for it and the tolerance latency range is determined based on the watermark threshold. Within a given window, out-of-order records are reordered and expired, and the completed synchronization batches are marked with batch tags.

[0013] When null fields or cross-source data conflicts are found in the sorted batch, the missing values ​​are calculated for the missing parts according to the imputation method, and the weight of abnormal records is reduced according to the conflict rules. The complete time series data after consistency correction is output, while the imputation mark is retained for reference when fine-tuning the timestamps later.

[0014] After receiving the interpolated data without serious contradictions, the system uses an external high-precision reference signal or the cross-correlation function of the internal trigger sequence, combined with adjustable parameters, to fine-tune the second-level or millisecond-level offset and record the adjustment amount. If the correction magnitude exceeds the safety boundary, it is marked as fine-tuned but suspicious, and the final output is a more refined time series that can meet the requirements of high-frequency or low-tolerance scenarios.

[0015] After completing the fine-grained timestamp alignment, the corrected data is distributed to multiple nodes according to the sharding mapping strategy, and the sharding status is maintained for each node. Once a node failure or abnormal load is detected and exceeds the fault judgment threshold, the fault-tolerant consensus mechanism is triggered to perform shard migration and difference patching, and to continuously maintain the highly available real-time data output after alignment in the multi-node environment.

[0016] Preferably, for each data source Assign a unique identifier Establish metadata in the database or cache ,include:

[0017] The geographic location or time zone label of the data source, a description of the time precision of the data reported by the data source; if the source does not have a precise timestamp locally, the local time must be specified here; combine the above information with... Stored in the index table in the form of;

[0018] Preferred, based on the registered , Information, for normal reporting of local time Based on the time zone difference or precision indication, a preliminary conversion is performed, and the result is mapped onto a unified reference axis for benchmarking to obtain a preliminary standard timestamp. Replace the local timestamp or placeholder timestamp in the original record with it;

[0019] Preferably, the data records that have been standardized will have updated standard timestamps. ,Will , ) are written sequentially to the index record table, where This is a reference to a specific data record; if several data entries from the same or multiple sources are detected under the same standard timestamp, a batch marker is added to the index entry;

[0020] Preferably, for each record that has completed the first step of processing. Establish a unified streaming access point and temporarily store it in a data structure called buffer queue, which can be divided into single queue mode or multi queue mode according to business needs.

[0021] Set a capacity limit for the buffer queue. This capacity limit This can be determined based on available memory or estimated traffic;

[0022] Introducing an adaptive control function During operation, based on the current queue length The deviation from the expected capacity automatically makes expansion or reduction decisions, which can be defined as follows:

[0023]

[0024] In the formula: Indicates time The number of records in the queue at that time. This represents the capacity that the queue is expected to maintain under ideal load. and To optimize the parameters, It is a power exponent;

[0025] Preferably, a watermarking function is defined within the real-time stream processing framework. This is used to represent an estimate of the time of the maximum event that has been reached, and it is extended to include a non-linear margin. Format:

[0026]

[0027] In the formula: This represents the maximum value of the standard timestamps of all records in the current buffer queue. For buffer queue, A custom time delay margin function, whose input is the current time. and allowable delay error This is to reserve some waiting space for high-latency data;

[0028]

[0029] in, The gain coefficient is adjustable. Indicates the start time or a certain reference time; watermark threshold. It can tolerate latency fluctuations more in the later stages of operation, preventing some data from being prematurely judged as late due to occasional delays;

[0030] When retrieving data records At that time, its standard timestamp is compared in real time. With the current watermark ;

[0031] like If the record is not found in the metadata, it is considered late data, a delay flag is added to it, and the delay flag is added to the metadata. Mark in; if If the arrival time is normal, it will be considered as a normal arrival and will not be treated as late.

[0032] The output includes the updated queue status, including the recorded delay flag and watermark value. wait;

[0033] Preferably, based on the queue Information, for all normal records, is formatted using standard timestamps. Sort in ascending order and package into logical batch units for unified output:

[0034]

[0035] After sorting is completed and the output conditions for a certain batch are met, missing data imputation and consistency correction are performed on that batch.

[0036] For records where the delay flag is true, the following two strategies can be used:

[0037] Delayed output strategy: Store it temporarily in the late arrival area and wait for a certain period of time, then determine whether it will arrive in the next watermark period and reorder it. If it still cannot catch up with the next batch in the delay period, add an expired mark to the metadata.

[0038] Directly mark and discard: If such late data is determined to have exceeded the tolerable time window. If the time is too long, mark it as discarded and stop sending it to subsequent steps to save computing resources, but retain only the most concise information, such as... This is for subsequent log analysis or data tracing.

[0039] After each reordering and batch cell generation is completed, an internal event is triggered to inform subsequent processes that missing data imputation and other operations can be performed; late or discarded data is also recorded in the flow monitoring daily.

[0040] Preferably, each record is retrieved from the batch data batch unit or the delayed waiting area; for data that has been discarded or expired, only the minimum reference information is retained; for data that is output normally or delayed, the required key fields are checked one by one.

[0041] If a record is found , , The absence of a numeric field or its value being marked as null is considered as imputable missing data. Similarly, if a data source receives no data at all during a certain time period, it is also classified as imputable missing data. This is based on the data source metadata. By attaching this category of information to the record, missing information can be categorized into critical missing information and general missing information.

[0042] Preferably, for generally missing data, a low-cost method is used for rapid data imputation. This includes previous value extension and interval interpolation. For critical missing fields, machine learning or rule-based algorithms can be introduced to perform high-precision prediction of missing values.

[0043] If external features can be obtained, they can be incorporated into kernel functions or combined with regression models to further improve prediction accuracy. After interpolation is completed, a filled mark is added to the corresponding record, and the specific interpolation method is written into the metadata. If interpolation fails, an unfilled mark is added and null values ​​are retained in the result set.

[0044] If a record has both a delayed flag and a null field, it is considered missing data and can be entered into the imputation algorithm; if the record is missing, no imputation process is triggered. This allows for the creation of a priority table for all flags, indicating their compatibility or exclusion relationships, thus avoiding conflicts.

[0045] Preferably, if there are logical conflicts between values ​​from different data sources within the same or adjacent time windows, they should be identified and adjusted.

[0046] Based on metadata The predefined constraints or association rules are used to verify each record with imputed or original values. If a conflict is found, it is marked as follows: minor inconsistency and major inconsistency;

[0047] If a record is marked as severely inconsistent, a penalty coefficient can be applied to its value in subsequent calculations. The adjusted results are formed. :

[0048]

[0049] in, As a safety baseline value;

[0050] For minor inconsistencies, monitoring or logging can be performed without immediately lowering the weight. The final validity or correction value of each record will be updated, and conflict markers or adjustment information will be added to the metadata.

[0051] Preferably, combined with metadata The recorded time precision information is used to detect the existence of a data source. If the actual acquisition accuracy is lower than required, and a data source is marked as having millisecond precision, but requires microsecond-level alignment in actual applications, and the precision does not meet the requirements, then an accuracy alarm or a low-precision marker will be added at this stage.

[0052] If an external signal is available, the time of that external signal is recorded as... ; can be combined with one's own time at this time For comparison, if the deviation between the two exceeds the preset deviation threshold... This will trigger a fine-calibration algorithm at the next moment, providing a basis for subsequent fine-tuning; within the preset deviation threshold If the value is within a certain range, no significant calibration is performed to save costs or avoid interference; if the external signal is unavailable or unstable, this step is skipped and subsequent calibration is performed only based on the internal correlation analysis.

[0053] Preferably, different data sources within the same time window or the same event trigger are cross-evaluated, and the correlation is calculated using labeled data to construct a metric function with an integral kernel. ,set up To align the evaluation time window, the following definition is made:

[0054]

[0055] In the formula: and For timestamps that are not yet refined. This is a kernel function used to characterize the contribution of timestamp differences to alignment. For the weight function;

[0056] If a high-precision reference is available, the current clock is first calibrated based on the external signal, and then the cross-correlation calculation of the local drift between different data sources is performed using the above formula. If the external signal is missing or unstable, the calculation is performed directly based on the metric function. To estimate the offset between sources;

[0057] The calculation results are stored in the deviation mapping table, recording each pair of data sources. optimal lag The deviation related to the external reference is denoted as Based on all optimal hysteresis values ​​in the offset mapping table Find the synchronization order that best aligns the vast majority of sources globally;

[0058] A synchronization path is determined by the minimum spanning tree or minimum loop coverage method, and the timestamps of each source are finely adjusted in sequence to avoid new conflicts caused by modifying all sources at once.

[0059] Output a deviation mapping table and a list of synchronization paths or priorities;

[0060] Preferably, based on the synchronization path, for each data source The timestamp is based on its optimal deviation relative to a high-precision reference and adjacent sources. Make corrections: Make the standard timestamp... Indicates the current source At any moment Based on the initial standard timestamp, a refined model with the following integral correction form is constructed. :

[0061]

[0062] In the formula: Indicates at time Source Measured time deviation function; To modify the kernel function, define a multivariable input and nonlinear output form:

[0063]

[0064] In the formula: This can be viewed as a time-varying weight. This is the overall scaling factor;

[0065] Set a safety boundary for each record. If the correction exceeds this boundary, manual intervention or additional over-adjustment is required. After fine-tuning all data sources, note in the record metadata that the timestamp has been corrected and add the new timestamp. As a final result, an adjustment log will be provided along with the output data;

[0066] Preferably, based on the time alignment results, a fragmentation mapping table is used. Shard the data according to business dimensions or time periods and distribute it to different computing nodes. ;

[0067] Maintain time offset function locally ,in Indicates node number, This indicates the actual processing time; to ensure the time offset function... To remain adaptive to network fluctuations or node clock errors, the following differentiable kernel integral form is adopted:

[0068]

[0069] in: For nodes The scaling factor; Size of the sliding window; Represents a node At any moment Transient clock deviation from a high-precision reference; For kernel functions;

[0070] When any node detects that the load is close to the limit during peak communication periods, it submits a shard migration request to the central scheduler or other nodes, triggering fault-tolerant consensus processing.

[0071] Preferably, when any node In the event of an unrecoverable failure or a high load exceeding the tolerance threshold, communication with the node is possible. The associated shard status record detects an anomaly in the shard status and sends a failure alarm to other available nodes and the scheduling center;

[0072] After receiving the failure alarm, other nodes utilize the broadcast time offset function. The sharding mapping information triggers a fault-tolerant consensus process to determine the optimal sharding takeover scheme.

[0073] If node If it recovers in a short time, its shard status is set to temporarily offline, and other nodes first save a small amount of key incremental data for it.

[0074] If the fault detection threshold is exceeded If the duration is not restored, then the shards will be officially migrated to available nodes. , by node Based on time offset function Time alignment and state continuation of the migrated data with external calibration references;

[0075] After the data is re-entered, differentiated repairs are performed on the small amounts of data lost or missed during the outage.

[0076] (III) Beneficial Effects

[0077] This invention provides a real-time processing method based on multi-source heterogeneous data, which has the following beneficial effects:

[0078] By adaptively fusing multi-source heterogeneous data, streaming buffering and watermarking, missing data imputation and conflict checking, high-precision timestamp refinement, and distributed fault tolerance, the real-time consistency, integrity, and availability of multi-source heterogeneous data in second-level and even millisecond-level environments are significantly improved.

[0079] By data source The local timestamp uses a unified UTC reference and is equipped with an adjustable gain coefficient. Power index By performing time zone difference and precision scaling, clock skew from various sources can be quickly eliminated during initial access, ensuring that each record has a standard time stamp. By leveraging buffer queues and watermark thresholds in streaming processing The strategy maintains the correct event sequence even in scenarios with out-of-order data and frequent delays, and marks data exceeding the latency tolerance as late or discards it, ensuring greater reliability for subsequent calculations. During the imputation and cross-source verification stages, it distinguishes between general and critical missing data, and incorporates a conflict record penalty factor. Appropriately reducing the weight of information that is obviously abnormal or has an unreliable source not only ensures the high integrity of the data at the numerical level, but also lays a clean foundation for the next step of timestamp refinement.

[0080] By introducing external high-precision signals and related deviations, microsecond-level corrections are performed on the timestamps of each source. When the correction exceeds the safety boundary, a flag is displayed, achieving a balance between accurate alignment and robustness. In the distributed fault-tolerant phase, automatic migration and asynchronous takeover of data shards are achieved through node shard status records and fault duration thresholds. This is supplemented by a kernel integral offset function to maintain cross-node clock synchronization, ensuring the continuity and high availability of global alignment results even when nodes are overloaded or fail.

[0081] By unifying time zones and precision from the outset of multi-source access, and mitigating out-of-order latency risks through streaming watermarking and buffering, the inherent consistency of data is strengthened through targeted interpolation and conflict handling. Fine-tuning of timestamps consolidates high-precision alignment, and multi-node fault tolerance and sharding distribution ensure the continuity of data flow in high-concurrency scenarios. This significantly reduces the risk of data loss or precision degradation caused by timing inconsistencies and node failures in high-frequency environments, effectively improving the accuracy and robustness of real-time analysis of multi-source heterogeneous data, and providing a more reliable time-series foundation and distributed parallel guarantee for industry decision-making. Attached Figure Description

[0082] Figure 1 This is a schematic diagram of the real-time processing method of the present invention. Detailed Implementation

[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] Please see Figure 1 This invention provides a real-time processing method based on multi-source heterogeneous data, including: Step 1, after receiving multi-source heterogeneous data, processing the data based on a unique identifier assigned to each data source. The time zone and precision are initially normalized using UTC reference, adjustable gain coefficient, and power exponent, along with local or server timestamp information. The converted time is then identified as a standard timestamp. At the same time, supplement the necessary metadata information;

[0085] Step one includes the following:

[0086] Step 101: Data Source Identification and Metadata Injection

[0087] For each data source Assign a unique identifier ,in Indicates the data source number and the range of values. , Number of data sources; establish metadata structure in the database or cache. Its contents include:

[0088] The geographical location or time zone label of the data source (record if known, mark as unknown if unknown), the time precision description of the data reported by the data source (e.g., seconds, milliseconds, or microseconds), and if the source does not have a precise timestamp locally, the local time should be noted here; the above information should be formatted as follows: The data is stored in an index table in the form of [database name], and all subsequent operations on the same data source can be performed through [database name]. Quickly retrieve the corresponding metadata ;

[0089] By each data source Assign a unique identifier and establishing metadata structure It can accurately distinguish and record key information such as time zone and precision of data in the initial stage; special marking processing such as local time can retain the source of missing or unclear timestamps for subsequent correction, rather than discarding them rashly; the method of pre-injecting metadata can help the system flexibly identify data type and time characteristics in subsequent steps, improving processing efficiency and error location capabilities.

[0090] Step 102: Local Time Resolution and Preliminary Benchmarking

[0091] When receiving new data records, based on the registered... , Information, read data source The accuracy, time zone, or marking status of the data will be considered. If the data is marked as local time, the arrival time of the receiving server will be used instead of the local time.

[0092] For normal reporting of local time Based on the time zone difference or precision prompts, perform a preliminary conversion and map it onto a unified reference axis; for example, first convert the local time... Turn into an intermediate time Based on this, the following formula is used for benchmarking to obtain a preliminary standard timestamp. :

[0093]

[0094] In the formula: A predefined UTC reference for the system;

[0095] For data sources The scaling factor is used to combine the accuracy of the source (seconds, milliseconds, etc.) with time zone deviation. Its possible value range is determined according to business needs, for example... ; The power exponent is adjustable, and its value can be between 1 and 2. In typical scenarios, a value of [value missing] is acceptable. If a non-linear mapping is required for clock offset conditions such as acceleration / deceleration, then it can be set as follows: ; This is a compensation item for server time drift; if there is no such requirement, it can be set to 0.

[0096] Finally obtained And replace the local timestamp or placeholder timestamp in the original record with it to achieve unified benchmarking;

[0097] Compared to the traditional method of adding or subtracting time zones and converting precision, the above formula introduces a power exponent. With compensation amount And set an independent scaling factor for each data source. Therefore, it can flexibly handle multiple precision levels or multiple potential offset modes, improving the adaptability of time alignment. If some data sources require special treatment (e.g., geographic remote monitoring equipment has significant drift), they can be adjusted individually. or This will not interfere with other data sources;

[0098] Step 103: Timing Index and Output Connection

[0099] All data records that have been benchmarked will have updated standard timestamps. ,Will , ) are written sequentially to the index record table, where A reference to a specific data record (including the data content itself and its source information);

[0100] This index record table sorts and stores standard timestamps from different data sources. If several data entries from the same or multiple sources are detected under the same standard timestamp, a batch marker can be added to the index entry.

[0101] Use an indexed record table to , Indexed storage ensures that all data is stored in the same order during queries and subsequent processing. Batch markers identify multiple data entries arriving at the same time, facilitating batch processing and bandwidth optimization. All metadata information and correction results are passed to the next step, saving computational resources by eliminating the need for repeated parsing of the first step's details.

[0102] Step 2: When a standard timestamp is detected After the marked data officially enters the streaming processing engine, a buffer queue is created for it and the tolerance latency range is determined based on the watermark threshold. Within a given window, out-of-order records are reordered and expired, and the completed synchronization batches are marked with batch tags.

[0103] Step two includes the following:

[0104] Step 201: Buffer window initialization and dynamic capacity management

[0105] For each record that has completed the first step of processing Establish a unified streaming access point and temporarily store it in a data structure called a buffer queue. Depending on business needs, this can be either a single queue mode to handle all data sources uniformly, or a multi-queue mode to handle different... Each buffer is established, and at the implementation level, a circular queue or a double-ended queue can be used to improve concurrency efficiency;

[0106] To prevent queue congestion during peak data source periods, a capacity limit needs to be set for the buffer queue. This capacity limit This can be determined based on available memory or estimated traffic, for example, The range of values ​​can be Between records; where: if the peak concurrency of the data source is Initial settings can be set ;

[0107] Introducing an adaptive control function Used to determine the current queue length during operation. The deviation from the expected capacity automatically makes expansion or reduction decisions, which can be defined as follows:

[0108]

[0109] In the formula: Indicates time The number of records in the queue at that time. This represents the capacity that the queue is expected to maintain under ideal load. and These are tuning parameters used to control the rate of expansion (positive values) or contraction (negative values): If the exponent is a power, then... It can impose stronger penalties for exceeding the target capacity, accelerating expansion or limiting capacity;

[0110] Through adaptive control function It can automatically increase buffer resources during peak traffic periods (if hardware conditions permit) and reclaim excess memory when traffic decreases, maintaining good real-time performance and resource utilization.

[0111] By setting the buffer queue and its capacity limit To prevent data loss and ensure real-time performance and integrity in high-concurrency scenarios, unified or distributed buffer queues are used to ensure that data is not lost due to out-of-order delivery or momentary congestion before processing, thus maintaining data integrity. Dynamic capacity management enables the system to automatically adjust the queue size in a flexible resource environment, avoiding frequent overflows or performance waste caused by fixed-size buffers. This is achieved through adaptive control functions. The function adaptively adjusts the queue load, rather than using a simple linear threshold strategy, thus providing more flexible control in high-concurrency or extreme scenarios.

[0112] Step 202: Watermark Generation and Late Arrival Detection

[0113] In real-time stream processing frameworks (such as Apache Flink and Spark Streaming), define a watermark function. This is used to represent an estimate of the time of the maximum event that has been reached, and it is extended to include a non-linear margin. Format:

[0114]

[0115] In the formula: This represents the maximum standard timestamp of all records in the current buffer queue, used to indicate the upper time limit of the latest event. For buffer queue, A custom time delay margin function, whose input is the current time. and allowable delay error To reserve some waiting space for high-latency data, a custom latency margin function is used. The specific form can be a piecewise exponential or polynomial to appropriately relax the watermark trigger threshold when traffic fluctuates extremely, for example:

[0116]

[0117] in, The gain coefficient is adjustable. Indicates the start time or a certain reference time; watermark threshold. It can tolerate latency fluctuations more in the later stages of operation, preventing some data from being prematurely judged as late due to occasional delays;

[0118] When retrieving data records At that time, its standard timestamp is compared in real time. With the current watermark ;

[0119] like If the record is not found in the metadata, it is considered late data, a delay flag is added to it, and the delay flag is added to the metadata. Mark in; if If the arrival time is normal, it will be considered as a normal arrival and will not be treated as late.

[0120] Records marked with a delay flag remain in the buffer queue for a short period, awaiting downstream decisions on whether to remove them or perform special processing. The output includes the updated queue status, including the record's delay flag and watermark value. wait;

[0121] Watermark function Combined with custom latency margin It can dynamically adapt to network congestion or data source latency fluctuations, and will not excessively discard late records. It marks late data with a delay mark so that subsequent steps can determine whether to continue to retain or mark it.

[0122] The watermarking mechanism gives real-time stream processing a certain degree of tolerance for out-of-order and delayed data, preventing overall computational errors or premature output due to latency from a single source. It integrates late arrival detection with an adaptive latency margin function. This combination enables the system to adapt to various network conditions and data source differences, providing ample time for subsequent interpolation and correction; it upgrades the common fixed watermark subtraction method to include logarithmic or polynomial forms. It maintains high efficiency in stable scenarios while providing greater tolerance for high latency anomalies.

[0123] Step 203: Timing Reordering and Computation Triggering

[0124] Each data record already has a normal or late flag, now based on the queue... Information, for all normal records, is formatted using standard timestamps. Sort in ascending order and package into logical batch units for unified output:

[0125]

[0126] After sorting is completed and a certain batch output condition is met (e.g., the number of records reaches a predefined threshold or the watermark trigger point is detected), missing data imputation and consistency correction are performed on the batch.

[0127] For records where the delay flag is true, the following two strategies can be used:

[0128] Delayed output strategy: Store it temporarily in the late arrival area and wait for a certain period of time, then determine whether it will arrive in the next watermark period and reorder it. If it still cannot catch up with the next batch in the delay period, add an expired mark to the metadata.

[0129] Directly mark and discard: If such late data is determined to have exceeded the tolerable time window. If the time is too long, mark it as discarded and stop sending it to subsequent steps to save computing resources, but retain only the most concise information, such as... This is for subsequent log analysis or data tracing.

[0130] After each reordering and batch cell generation is completed, an internal event is triggered to inform subsequent processes that missing data imputation and other operations can be performed; late or discarded data is also recorded in the flow monitoring daily.

[0131] By packaging normal records in an orderly manner, downstream algorithms no longer need to worry about out-of-order issues during processing, thus simplifying implementation and improving the accuracy of missing data identification. Implementing graded processing (delaying or directly discarding) for late data can ensure integrity while avoiding resource waste caused by unlimited waiting. It also enables the system to have controllable adaptive capabilities to extreme network latency, retaining some suspicious data for backtracking, and avoiding resource waste caused by unlimited waiting.

[0132] Two modes, delayed output strategy and direct marking and discarding, are added to allow late data to be treated differently according to the degree of delay, so as to flexibly set the waiting time or tolerance for different business needs.

[0133] Step 3: When null fields or cross-source data conflicts are found in the sorted batch, the missing values ​​are calculated for the missing parts according to the imputation method, and the weight of abnormal records is reduced according to the conflict rules. The complete time series data after consistency correction is output, while the imputation mark is retained for reference when fine-tuning the timestamps later.

[0134] Step three includes the following:

[0135] Step 301: Identification and Classification of Missing Records

[0136] Retrieve each record from the batch data batch unit or the delayed waiting area; for data that has been discarded or expired, retain only the minimum reference information; for data that is output normally or delayed, check each record to see if the required key fields (such as observations, text loads, etc.) exist.

[0137] If a record is found , , If a numeric field is missing or its value is marked as null, it is considered as imputable missing. If a data source has no data arriving at all during a certain period of time, it is also classified as imputable missing.

[0138] Based on data source metadata The system attaches information such as importance, data type, and frequency characteristics to records. Missing information can be categorized as follows: critical missing information: such as core financial indicators or key fields that have a significant impact on subsequent calculations; general missing information: relatively minor fields, such as optional additional attributes.

[0139] Through systematic scanning and classification, it is possible to quickly identify which missing values ​​require priority imputation and which can tolerate null values, improving imputation efficiency and reducing unnecessary computational overhead on secondary fields. Different categories can be used with different strategies in subsequent imputation, and it also facilitates priority management when imputation fails. The missing value categories are linked to the data source metadata. This approach allows for differentiated processing based on data importance or impact in the same missing data scenario, which is far more flexible and precise than simply imputing data whenever it is missing from a uniform perspective.

[0140] Step 302: Multi-strategy interpolation and external feature prediction

[0141] For commonly missing data, low-cost methods are used for rapid data filling, including:

[0142] Previous value extension: directly take the value from the previous non-empty record of the same data source as the current fill value;

[0143] Interval interpolation: If at adjacent times and If all values ​​are valid, then the missing time points will be... The values ​​are obtained by applying a linear ratio:

[0144]

[0145] in for For key missing fields, machine learning or rule-based algorithms can be introduced to predict missing values ​​with high accuracy. The kernel regression imputation formula is as follows:

[0146]

[0147] In the formula: Indicates the time to be interpolated; Indicates data source A set of known sample indices within adjacent time periods;

[0148] This is a kernel function used to measure the weight of historical observations that are closer in distance;

[0149] This is the distance scaling factor, used to control the decay rate of the kernel function;

[0150] If external features (such as market fluctuations, upstream and downstream indicators of the industrial chain) can be obtained, they can be incorporated into the kernel function or combined with a regression model to further improve the prediction accuracy; after the imputation is completed, a filled mark is added to the corresponding record, and the specific imputation method is written into the metadata; if the imputation fails (such as insufficient historical data), an unfilled mark is added and null values ​​are retained in the result set;

[0151] By applying low-cost, simple interpolation and machine learning-driven high-order interpolation to fields of different importance, a kernel function can be introduced to balance computational efficiency and improve the accuracy of key variable interpolation. and scaling factor This approach makes the interpolation process more adaptable and better reflects the temporal patterns and similarity weights compared to traditional average or linear methods.

[0152] The method employs kernel regression in predictive imputation and allows external features to be incorporated into the construction of the kernel function, overcoming the limitations of conventional interpolation or moving average, and is suitable for handling complex data missing scenarios with multiple sources and dimensions.

[0153] If a record has both a delayed flag and a null field, it is considered missing data and can be entered into the imputation algorithm; if the record is missing, no imputation process is triggered. This allows for the creation of a priority table for all flags, indicating their compatibility or exclusion relationships, thus avoiding conflicts.

[0154] Step 303: Cross-source consistency verification and data validity adjustment

[0155] If logical conflicts exist between values ​​from different data sources within the same or adjacent time windows, they should be identified and adjusted. For example, the product of trading volume and trading value should be within a certain fluctuation range; if there is a significant divergence between upstream raw material prices and downstream output, slight inconsistencies should be noted.

[0156] Based on metadata The predefined constraints or association rules are used to verify each record with its imputed or original value. If a conflict is found, it is marked as follows:

[0157] Minor inconsistencies: minor conflicts that can be ignored; Major inconsistencies: serious contradictions that require reduced weighting or special handling in subsequent analysis.

[0158] If a record is marked as severely inconsistent, a penalty coefficient can be applied to its value in subsequent calculations. The adjusted results are formed. :

[0159]

[0160] in, This serves as a safety baseline value (such as a weighted average of nearest neighbor times or a balance value set by the business). The smaller the value, the more serious the conflict and the lower the trust level in the record;

[0161] For minor inconsistencies, monitoring or logging can be performed without immediately lowering the weight. The final validity or correction value of each record will be updated, and conflict markers or adjustment information will be added to the metadata.

[0162] Cross-source consistency checks can quickly identify discrepancies between data from different sources, providing higher data credibility for data analysis and ROI prediction. By assigning penalty coefficients to severely inconsistent records, the impact of these records on subsequent calculations can be reduced to a reasonable range while retaining the original data, without having to completely discard suspicious data. When serious conflicts are found, instead of simply discarding the relevant records, penalty coefficients are used to reduce their impact on subsequent calculations: by comparing with a safety benchmark value... By performing weighted fusion, the most basic temporal continuity is preserved.

[0163] Step 4: After receiving the interpolated data without serious contradictions, use the external high-precision reference signal or the cross-correlation function of the internal trigger sequence, combined with adjustable parameters, to fine-tune the second-level or millisecond-level offset and record the adjustment amount. If the correction range exceeds the safety boundary, it is marked as fine-tuned but suspicious, and the final timing sequence is more refined and can meet the requirements of high-frequency or low-tolerance scenarios.

[0164] Step four includes the following:

[0165] Step 401: Microscopic Precision Difference Identification and External Benchmark Acquisition

[0166] The refinement of timestamps is based solely on cross-source comparisons and external benchmarks. The numerical fields obtained from the third step of interpolation are not modified again. Even if the interpolation result affects the inference of the event triggering order, the previously interpolated values ​​will not be changed in reverse.

[0167] Combine metadata The recorded time precision information is used to detect the existence of a data source. If the actual acquisition accuracy is lower than required, and a data source is marked as having millisecond precision, but requires microsecond-level alignment in actual applications, and the precision does not meet the requirements, then an accuracy alarm or a low-precision marker will be added at this stage.

[0168] If an external signal is available, such as a GPS clock server or a high-precision benchmark from a high-frequency trading platform, then the time of this external signal is recorded as... ; can be combined with one's own time at this time For comparison, if the deviation between the two exceeds the preset deviation threshold... This will trigger a fine-calibration algorithm at the next moment, providing a basis for subsequent fine-tuning; within the preset deviation threshold Within a certain range, no significant calibration is performed to save costs or avoid interference; when external signals are unavailable or unstable, this step is skipped and subsequent corrections are only performed based on internal correlation analysis; low-precision markers and available high-precision external time signals are recorded in the metadata.

[0169] This allows for early identification of data sources with low time accuracy or significant uncertainty, enabling focused attention on these sources during subsequent calibration. If an available external high-precision benchmark exists, subsequent adjustments will be more accurate and authoritative; otherwise, internal cross-correlation or trigger sequence can be used for estimation, enhancing system adaptability. Combining the availability of external high-precision signals with the marked accuracy status allows for pre-processing to differentiate between different sources, providing multiple branch options for subsequent fine-tuning logic, rather than forcing reliance on a single alignment method.

[0170] Step 402: Deviation estimation and synchronization sequence inference

[0171] Cross-evaluation is performed on different data sources within the same time window or the same event trigger, using labeled data (such as imputed key variables and cross-source consistency results) to calculate the correlation. For example, if it is known that multiple sources recorded the same market event within a certain second, their standard timestamps are used to determine the correlation. They should be close to or equal. Below, we construct a cross-correlation function with an integrator kernel. ;

[0172] within a specific time window Internally, regarding the source With source The time difference relationship is integrated, and the similarity is measured using a kernel function, thus balancing globality and differentiability; let... To align the time windows for evaluation (e.g., the joint sample interval of the most recent seconds or minutes), the following can be defined:

[0173]

[0174] In the formula: and The timestamp is not yet refined (or a consistent timestamp that has undergone preliminary interpolation).

[0175] This is the kernel function used to characterize the contribution of timestamp differences to alignment. You can choose a Gaussian kernel, an exponential kernel, or a custom fast decay kernel function, for example:

[0176]

[0177] Here and It can be set according to data characteristics to control the decay rate and the degree of nonlinearity;

[0178] This is an optional weighting function, allowing different importance to be assigned to different moments within a time window. For example, a higher weight can be set during peak trading hours to increase the priority of alignment during this period. If this is not required, it can be left as is. ;

[0179] By integrating over the entire time interval A higher value will accumulate when the difference between the two timestamps is small, thus reflecting a better fit. Maximizing this integral will find the optimal value. Used to estimate the source and The optimal alignment offset is determined globally. If a high-precision reference is available, the current clock is first calibrated based on the external signal, and then the cross-correlation calculation is performed on the local drift between different data sources using the above formula. If the external signal is missing or unstable, the metric function is used directly. To estimate the offset between sources;

[0180] The calculation results are stored in the deviation mapping table, recording each pair of data sources. optimal lag The deviation related to the external reference is denoted as Based on all optimal hysteresis values ​​in the offset mapping table Graph theory or sorting algorithms can be used to find the synchronization order that most sources are most closely aligned globally. For example, a complete graph can be constructed. Its nodes correspond to data sources, and the edge weights can be set to... The relative positions of the sources are determined by using the minimum spanning tree or minimum loop coverage method to establish a synchronization path and then fine-tune the timestamps of each source in sequence to avoid creating new conflicts by modifying all sources at once.

[0181] Output deviation mapping table and synchronization path or priority list; finely evaluate time misalignment between different sources in high-frequency or complex data scenarios. When used with an external high-precision benchmark, it can perform global correction and local alignment simultaneously, using graph algorithms or topological sorting to infer the synchronization order, avoiding the chaos caused by cyclic adjustments or mutual constraints among multiple sources, and improving the reliability of single alignment; for multi-source scenarios, it proposes a comprehensive scheme of cross-correlation function + graph theory synchronization order, getting rid of the limitations of using only a fixed reference source or manually setting priorities, and combining flexibility and global optimization.

[0182] Step 403: Refined timestamp correction and output

[0183] Based on the synchronization path, for each data source The timestamp is based on its optimal deviation relative to a high-precision reference and adjacent sources. Make corrections: Make the standard timestamp... Indicates the current source At any moment Based on the initial standard timestamp, a refined model with the following integral correction form is constructed. :

[0184]

[0185] In the formula: Indicates at time Source The measured time deviation function may originate from the aforementioned cross-source alignment analysis or external reference signals, for example:

[0186]

[0187] It is a set of multiple deviation factors, representing the degree of inconsistency between the global and local levels;

[0188] To modify the kernel function for cumulative correction based on deviations at different times, a kernel similar to the one described above can be selected. The kernel concept used in this process can also be defined in the form of multivariable inputs and nonlinear outputs, such as:

[0189]

[0190] In the formula: It can be viewed as a time-varying weight (similar to) ), or additional consideration of the source At any moment Data integrity, business priority, etc.; The scaling factor determines the extent to which the integral term affects the final timestamp. A larger value indicates a greater willingness to make significant corrections to accumulated historical biases; a smaller value indicates that more of the initial timestamp will be retained. The result;

[0191] To avoid overcorrection during fine-tuning, a safety boundary can be set for each record. If the correction exceeds this boundary, manual intervention or an additional record of overcorrection is required. This can prevent unexpected abnormal data sources from distorting the overall alignment result.

[0192] After completing fine-tuning of all data sources, note in the record metadata that the timestamp has been corrected, and include the new timestamp. As a final result; if any special markers (such as minor inconsistencies) appear in any of the interpolation or consistency checks mentioned above, they can be referenced at this time to make final weight adjustments; an adjustment log will be provided along with the output data;

[0193] When the adjustment of a record exceeds a safety threshold, an over-adjustment event is automatically issued, providing a pause / continue option. Pause: Only review the record individually or retain the original value; Continue: Treat this record as a "potential anomaly" and include it in the final result; Records that have been adjusted but are questionable are alerted downstream users to consider their reliability. Based on actual business needs, it can be decided whether to automatically correct or terminate the global alignment process after manual confirmation. This is achieved by progressively incorporating global deviations. Local deviation This allows for more consistent multi-source timestamps at the millisecond or even microsecond level. After correction, the entire dataset can be used as a high-precision input for direct use in real-time ROI analysis or other time-sensitive applications.

[0194] Instead of simple superposition, external global biases are nonlinearly combined with local biases from adjacent sources, allowing for flexible error allocation strategies tailored to different industries or business scenarios. A safety boundary verification mechanism is introduced to prevent over-adjustment and provide greater safety redundancy.

[0195] It can align data at the second or even millisecond level to a more refined level, maintaining stability and consistency even under multi-source concurrency, network latency, and high-frequency scenarios. It provides solid underlying support for real-time industry ROI analysis or other calculations that require precise timing guarantees. Through multi-stage collaboration, it not only achieves time unification and interpolation correction of multi-source heterogeneous data, but also finds a robust balance between high precision and security.

[0196] Step 5: After completing the fine-grained timestamp alignment, the corrected data is distributed to multiple nodes according to the sharding mapping strategy, and the sharding status is maintained for each node. Once a node failure or abnormal load is detected and exceeds the fault judgment threshold, the fault-tolerant consensus mechanism is triggered to perform shard migration and difference patching, and to continuously maintain the highly available real-time data output after alignment in the multi-node environment.

[0197] Step five includes the following:

[0198] Step 501: Multi-node sharding deployment and clock synchronization broadcasting

[0199] High-precision timestamps With corresponding data stream All datasets have been refined and are ready for global use. Based on the time alignment results, a sharding mapping table can be used. This data is sharded according to business dimensions or time periods and distributed to different computing nodes. ;

[0200] After receiving the allocated data fragments, each node will continuously obtain synchronization calibration information from an external high-precision signal high-precision reference in order to maintain the time offset function locally. ,in Indicates node number, Indicates the actual processing time;

[0201] To ensure the time offset function To remain adaptive to network fluctuations or node clock errors, the following differentiable kernel integral form can be used:

[0202]

[0203] in: For nodes The scaling factor is used to determine the extent to which the calibration corrects the local clock. The size of the sliding window limits the integration interval to the length of the nearest Ω. Represents a node At any moment Transient clock deviation from a high-precision reference; As a kernel function, combined with exponential or logarithmic decay forms, it can smoothly fuse the drift rate of the node in the most recent period, thereby dynamically correcting the node's local clock and maintaining high-precision alignment.

[0204] When any node detects that the load is close to the limit during peak communication periods, it submits a shard migration request to the central scheduler or other nodes, triggering fault-tolerant consensus processing.

[0205] By partitioning and mapping The framework distributes data across multiple nodes, significantly reducing the computational pressure on a single point and maintaining throughput even under massive concurrent data flows. Simultaneously, each node utilizes a time offset function. Adaptively synchronizes with a high-precision reference, effectively absorbing network latency jitter and node clock drift, ensuring that the cross-node time scale remains consistent with the global refining results of the fourth step;

[0206] Using a combination of kernel integration and sliding window The function enables each node's local clock correction to both accumulate historical errors and respond quickly to the current network state. Compared with traditional fixed offset or simple linear compensation, it is more flexible and suitable for fine-grained distributed collaborative deployment in high-frequency scenarios.

[0207] Step 502: Fault Tolerance Consensus and Failover

[0208] When any node In the event of an unrecoverable failure or a high load exceeding the tolerance threshold, communication with the node is possible. The associated shard status record detects an anomaly in the shard status and sends a failure alarm to other available nodes and the scheduling center;

[0209] After receiving the failure alarm, other nodes utilize the broadcast time offset function. The sharding mapping information triggers a fault-tolerant consensus process to determine the optimal sharding takeover scheme.

[0210] If node If it recovers quickly, its shard status is set to temporarily offline, and other nodes first save a small amount of critical incremental data for it; if the fault threshold is exceeded... If the duration is not restored, then the shards will be officially migrated to available nodes. , by node Based on time offset function Time alignment and state continuation of the migrated data with external calibration references;

[0211] After the data is re-entered into the queue (i.e., the node) Restore online or node (Takeover complete) Perform differentiated repairs on the minute amounts of data lost or missed during the failure:

[0212] If the local timestamps of some data records can be found through the logs These data records are then re-incorporated into the shards and merged into the node alignment queue in the form of clock calibration. ;

[0213] The fault-tolerant consensus module prevents global stagnation or data loss due to single-node failure, maintaining continuous updates to the time-series chain even when a node is offline; even if a node rejoins, updates can be made based on the previous time offset function. Information from high-precision benchmarks is rapidly and smoothly integrated, and missing window compensation is performed. A dynamic load switching mechanism allows the system to automatically allocate data shards to idle or low-load nodes, smoothing out peaks and valleys while maintaining overall real-time processing efficiency. In distributed fault-tolerant scenarios, refined timestamps are used to calibrate the time offset function. Deep integration with the sharding takeover process allows the new takeover node to correct the timestamp along with the fault switch, truly ensuring global consistency of multi-node alignment rather than simply performing master-slave replication: by performing differentiated repairs on the failure window, the data integrity after fault tolerance recovery is further improved.

[0214] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0215] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0216] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0217] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0218] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time processing method based on multi-source heterogeneous data, characterized in that: include, After receiving multi-source heterogeneous data, based on the identifier and timestamp information assigned to each data source, the time zone and precision are initially normalized and converted using adjustable gain coefficients and power exponents, and the converted time is identified as a standard timestamp. Once data with standard timestamps is detected and formally processed in streaming mode, a buffer queue is created for it, and the tolerance delay range is determined based on the watermark threshold. Within a given window, out-of-order records are reordered and marked as expired. When null fields or cross-source data conflicts are found, missing value calculations are performed on the missing parts, and the weight of abnormal records is reduced based on conflict rules. The complete time series data after consistency correction is output and the imputation mark is retained. If an external signal is available, first perform an overall clock correction based on the external signal, then estimate the offset between sources based on the cross-correlation function; if the external signal is missing or unstable, directly estimate the offset between sources based on the cross-correlation function; fine-tune the second-level or millisecond-level offset and record the adjustment amount; if the correction magnitude exceeds expectations, mark it as fine-tuned but suspicious, and output the accurate time series; let... To align the evaluation time windows, the cross-correlation function is defined as: ； In the formula: and Data sources The corresponding, yet unrefined, standard timestamp; For a specific time window; For kernel functions; For the weight function, The current time; The corrected data is distributed to multiple nodes according to the sharding mapping strategy, and the sharding state is maintained for each node. Once a node failure or abnormal load is detected and exceeds the fault judgment threshold, the fault-tolerant consensus mechanism is triggered to perform shard migration and difference patching, and output real-time data.

2. The real-time processing method according to claim 1, characterized in that: Assign a unique identifier to each data source and establish metadata in the database or cache, including: the geographical location or time zone label of the data source, the time precision description of the data reported by the data source, and if the data source does not have a precise timestamp on its local side, the local time should be noted here and stored in the index table. For normal local time reporting, a preliminary conversion is performed based on the time zone difference or precision prompt, and it is mapped onto a unified reference axis for benchmarking to obtain a preliminary standard timestamp, which replaces the local timestamp or placeholder timestamp in the original record. Once the data records have been standardized, they will have updated standard timestamps and will be written to the index record table in sequence. If several data records from the same or multiple sources are detected under the same standard timestamp, a batch marker will be added to the index entry.

3. The real-time processing method according to claim 1, characterized in that: A unified streaming access point is established for each record, and it is temporarily stored in a data structure called buffer queue. Depending on the business requirements, it can be divided into single queue mode or multi queue mode. Set a capacity limit for the buffer queue, which is determined based on available memory or estimated traffic. An adaptive control function is introduced to automatically make expansion or reduction decisions based on the deviation between the current queue length and the expected capacity during operation.

4. The real-time processing method according to claim 3, characterized in that: In a real-time stream processing framework, a watermark function is defined to represent an estimate of the maximum event time of the arrived data. This function is then extended to include a non-linear margin, where: ； In the formula: This represents the maximum value of the standard timestamps of all records in the current buffer queue. For buffer queue, A custom time delay margin function, whose input is the current time. and allowable delay error This is to reserve some waiting space for high-latency data; ； in, The gain coefficient is adjustable. Indicates the start time or a reference time; When a data record is retrieved, its standard timestamp is compared with the current watermark in real time. If the standard timestamp is less than the current watermark, the record is considered late data, a delay mark is added to it, and it is marked in the metadata. Otherwise, it is considered to have arrived normally, no late processing is performed, and the output includes the updated queue status.

5. The real-time processing method according to claim 4, characterized in that: Based on the data records in the queue, all normal records are sorted in ascending order by standard timestamp and packaged into logical batch units. After sorting is completed and a certain batch output condition is met, missing data imputation and consistency correction are performed on the corresponding batch. For records that are correctly marked for delay, a delayed output strategy or direct marking for discarding is adopted. After each resort and batch unit generation is completed, an internal event is triggered to inform subsequent steps to perform missing imputation operations. Data in the late area or that is discarded is also recorded in the flow monitoring log.

6. The real-time processing method according to claim 5, characterized in that: Retrieve each record from the batch unit or the delayed waiting area. For data that has been discarded or expired, retain only the minimum reference information. For data that is output normally or delayed, check each record for the existence of the required key fields. If a data record is found to be missing a numeric field or its value is marked as null, it is considered to be imputed missing. If a data source has no data arriving at all within a certain period of time, it is also classified as imputed missing. Based on the data source metadata, the corresponding category information is attached to the record, and the missing information is divided into critical missing information and general missing information.

7. The real-time processing method according to claim 6, characterized in that: For general missing data, previous value extension and interval interpolation are used for rapid filling. For critical missing data, machine learning or rule-based algorithms are introduced to make high-precision predictions of missing values. After interpolation is completed, a filled mark is added to the corresponding record, and the specific interpolation method is written into the metadata; if interpolation fails, an unfilled mark is added and null values ​​are retained in the result set. If a record has both a delayed flag and a null field, it is considered as missing data and the imputation algorithm is initiated; if the record is lost, no imputation process is triggered.

8. The real-time processing method according to claim 7, characterized in that: If there are logical conflicts between values ​​from different data sources within the same or adjacent time windows, they should be identified and adjusted. Based on the constraints or association rules predefined in the metadata, the records of each imputed or original value should be checked. If conflicts are found, they should be marked as minor inconsistencies and major inconsistencies. If a record is marked as seriously inconsistent, a penalty coefficient is applied to its value in subsequent calculations to form an adjusted result; Minor inconsistencies will only be monitored or recorded, and the final validity or corrected value of each record will be updated, with conflict markers or adjustment information added to the metadata.

9. The real-time processing method according to claim 8, characterized in that: By combining the time precision information recorded in the metadata, it is possible to detect whether the actual collection precision of a data source is lower than the requirement. If a data source is marked as having millisecond precision but the precision does not meet the actual application standard, a precision alarm or a low precision mark is added at this stage. If an external signal is available, it is compared with the maximum value of the timestamp at this time. If the deviation between the two exceeds the preset deviation threshold, a fine calibration algorithm will be triggered at the next moment. No significant calibration is performed within the preset deviation threshold. If the external signal is unavailable or unstable, this step is skipped and subsequent corrections are performed only based on the internal correlation analysis.

10. The real-time processing method according to claim 9, characterized in that: Cross-evaluate different data sources within the same time window or the same event trigger, calculate the correlation degree with labeled data, and construct a metric function with an integral kernel; The current clock is calibrated based on the external signal. The cross-correlation function is used to calculate the local drift between different data sources. If the external signal is missing or unstable, the offset between sources is estimated based on the metric function. The calculation results are stored in the deviation mapping table, and the optimal lag of each pair of data sources is recorded. The synchronization order that can make the vast majority of sources most closely aligned globally is found by graph theory or sorting algorithm. A synchronization path is determined by the minimum spanning tree or minimum loop coverage method. The timestamps of each source are finely adjusted in sequence to avoid new conflicts caused by modifying all sources at once. The deviation mapping table and synchronization path are output.

11. The real-time processing method according to claim 10, characterized in that: Based on the synchronization path, the timestamps of each data source are corrected according to their optimal deviation relative to the high-precision reference and adjacent sources: Set a safety boundary for each record. If the correction exceeds the boundary, manual intervention or an over-adjustment record is required. After completing the fine-tuning of all data sources, the timestamps are noted in the record metadata as corrected, and the new timestamps are used as the final result. Adjustment logs are provided along with the output of the result data.

12. The real-time processing method according to claim 11, characterized in that: Based on the time alignment results, the data is sharded according to business dimensions or time periods using a sharding mapping table and allocated to different computing nodes. When any node detects that the load is close to the limit during peak communication periods, it submits a sharding migration request to the central scheduler or other nodes, triggering fault-tolerant consensus processing.

13. The real-time processing method according to claim 12, characterized in that: When any node experiences an unrecoverable failure or a high load exceeding the tolerance threshold, the anomaly is detected through the shard status record associated with the node, and a failure alarm is sent to other available nodes and the scheduling center. After receiving the failure alarm, other nodes use the broadcast time offset function and sharding mapping information to trigger the fault-tolerant consensus process in order to determine the optimal sharding takeover scheme. If a node recovers within a short period of time, its shard status is set to temporarily offline, and other nodes first save a small amount of critical incremental data for it. If the fault is not recovered within the time limit specified by the fault assessment threshold, the shard will be officially migrated to an available node.

Images