A transaction timestamp injection data timing reconstruction method compatible with cloud native architecture

Abstract

The application discloses a transaction timestamp injection data time sequence reconstruction method compatible with a cloud native architecture, a data production end acquires transaction data and uploads the transaction data to a cloud message queue, a data consumption end acquires the transaction data from the cloud message queue and injects transaction time acquired from a transaction house, the cloud acquires the transaction data injected with the transaction time, creates a partition according to the transaction time, and stores new transaction data in a hot data area and old transaction data in a cold data area. The application reconstructs the transaction data time sequence through authoritative time source injection, ensures transaction sequence authenticity, constructs a time sequence reconstruction scheme based on native database functions, designs a lightweight field injection mechanism, avoids invasive modification of cloud service APIs, establishes a lightweight data fault tolerance mechanism, improves data fault tolerance, and finally solves the fundamental contradiction between unordered storage of the cloud native architecture and financial time sequence requirements and the fault tolerance requirement.

Description

Technical Field

[0001] This invention belongs to the field of real-time data processing technology based on cloud platforms, and specifically relates to a method for time-series reconstruction of transaction timestamp-injected data that is compatible with cloud-native architecture. Background Technology

[0002] In the fintech field, stock market data changes dynamically in real time, and traditional batch processing leads to analytical lag. One feasible solution is to achieve real-time data analysis based on the Kafka architecture. However, cloud-native systems based on Kafka, such as Confluent Cloud, have significant shortcomings and defects when used for real-time stock data processing. Kafka itself does not guarantee global order, and message queues like Confluent Cloud, which use distributed storage mechanisms to improve performance, have inherently unordered characteristics that conflict significantly with the strict time order required for stock trading. Traditional solutions attempt to reconstruct the time order through the producer's local timestamp or message offset, but due to the lack of an authoritative time source reference, significant errors occur in cross-timezone and multi-partition scenarios. Furthermore, when network jitter causes message retransmission, duplicate data and out-of-order issues overlap, resulting in severely distorted analysis results. In addition, existing solutions have poor data fault tolerance; when the data consumer fails, data loss or duplicate consumption may occur. The linear accumulation of massive historical transaction data leads to a continuous deterioration in query efficiency, and the full-scale storage model imposes a heavy cost burden. Existing technical solutions focus primarily on hardware expansion, failing to effectively address core issues such as data timeliness and fault tolerance. Therefore, the industry urgently needs a time-series data assurance mechanism compatible with cloud-native architectures to ensure data orderliness while improving fault tolerance. Summary of the Invention

[0003] To address the shortcomings of existing technologies, achieve market data processing compatible with mainstream cloud platforms based on transaction timeline reconstruction, avoid time-series errors caused by disordered storage of cloud message services, and improve the reliability of cloud platforms, this invention adopts the following technical solution:

[0004] A time-series reconstruction method for transaction timestamp injection compatible with cloud-native architecture is proposed. Applied to the data consumer, it retrieves transaction data uploaded by the data producer from a cloud message queue and injects transaction timestamps obtained from the exchange into the transaction data. The injected transaction data is then sent to the cloud, enabling the cloud to create partitions based on transaction times. This allows new transaction data to be stored in the hot data area and older transaction data in the cold data area. Simultaneously, this provides the database with authoritative transaction timestamp references, enhancing the credibility of historical data analysis.

[0005] Furthermore, when the transaction data obtained from the cloud does not include the transaction time, a new transaction time is generated based on the transaction time obtained from the exchange and the network time delay compensation obtained from the exchange's authentication channel, and then injected into the transaction data to ensure that the timestamp has legal authority and lays the foundation for subsequent time sequence reconstruction.

[0006] When the transaction data obtained from the cloud contains transaction time, the difference between the included transaction time and the transaction time obtained from the exchange is compared with a threshold. If it is less than the threshold, the included transaction time is downgraded to the main data as an auxiliary field. If it is greater than or equal to the threshold, the credibility audit of the transaction data is triggered.

[0007] Furthermore, the auxiliary field serves as the original time certificate for the generation of transaction data, thereby establishing a complete data chain. When it is necessary to verify the authenticity of transaction data, auditors can verify it by comparing the auxiliary field with the time obtained from the transaction.

[0008] Based on the exchange's transaction time and the transaction time as an auxiliary field in the digital text, the fault can be quickly located. When there is a discrepancy between the two transaction times, if it is determined that the exchange's transaction time distribution is normal, the cause of the fault can be located based on the transaction time in the auxiliary field. At the same time, the data quality score is determined based on the magnitude of the discrepancy between the two transaction times to determine the repair priority.

[0009] Furthermore, cloud transaction data is acquired and parsed, and key metadata is pre-written to local log files in a sequential append manner. The log files are stored in shards according to the consumer group ID and Kafka partition number.

[0010] Log pre-writing adopts the O_DIRECT mode to bypass the operating system cache, reduce multiple copies of data from user space to kernel space and then to disk and cache latency, thereby improving I / O performance; each shard follows a rolling over strategy, creating a new log file when a single log file exceeds a certain size and deleting the oldest log file to control storage overhead;

[0011] If transaction data parsing fails, the retry engine will be started to perform multiple parsing attempts. If multiple parsing attempts fail, the circuit breaker mechanism will be triggered, marking the Kafka partition corresponding to the transaction data as degraded. The consumer will then suspend consumption of the corresponding Kafka partition and move the corresponding transaction data to the dead-letter queue. This will prevent data loss or duplicate consumption in the event of a consumer failure.

[0012] Furthermore, the retry engine employs adaptive hierarchical retries, with the initial retry interval being a fixed value. The subsequent interval will be multiplied by an exponential backoff algorithm:

[0013]

[0014] in, Indicates the basic interval. This indicates the number of retries already completed. Indicates the maximum interval.

[0015] Furthermore, the dead letter queue is periodically scanned by the daemon process. When a new dead letter message is detected, an asynchronous retry is triggered. If the retry fails, an alarm is triggered to repair the issue.

[0016] Hot-deployed data conversion patches automatically retrieve dead-letter messages via a daemon process. For successfully repaired dead-letter messages, the root cause of the failure is recorded so that repair rules can be automatically applied to similar errors in the future.

[0017] A time-series reconstruction method for transaction timestamp injection compatible with cloud-native architecture is applied to the data production end. It acquires transaction data and uploads it to the cloud message queue, so that the data consumption end can inject transaction data from the transaction timestamp obtained from the exchange. This allows the cloud to create partitions according to the injected transaction timestamps, storing new transaction data in the hot data area and old transaction data in the cold data area. At the same time, it provides the database with authoritative transaction timestamp references, improving the credibility of historical data analysis.

[0018] Furthermore, the data production end obtains the latest transaction data through a third-party interface. After the transaction data is encrypted, it is sent to a topic in the cloud-hosted message queue. The cloud service automatically distributes the messages and stores them in different partitions. The production end maintains the original data collection function and writes market data containing core fields such as stock codes, prices, and trading volumes into the message queue. The production end explicitly relinquishes the control over the timing and fully follows the unordered storage characteristics of the cloud service, providing a compatibility basis for upper-layer innovation.

[0019] A time-series reconstruction method for transaction timestamp injection compatible with cloud-native architecture is proposed. Applied to the cloud, it acquires transaction data uploaded by the data producer and stores it in a cloud message queue. This allows the consumer to access the transaction data in the message queue and inject transaction timestamps obtained from the exchange. The method retrieves the transaction data with injected transaction timestamps and creates partitions based on these timestamps. New transaction data is stored in the hot data area, while older transaction data is stored in the cold data area. This provides the database with authoritative transaction timestamp references, enhancing the reliability of historical data analysis.

[0020] Furthermore, after the cloud-based database obtains transaction data with transaction times, it constructs a B+ tree index with transaction time as the primary key to reconstruct the time sequence of the storage layer. Specifically, based on the original unused dataset and the query start time boundary and query end time boundary, the time sequence is reconstructed through the sorting operator of transaction time, thereby transferring the time sequence control from the infrastructure layer to the application layer, so that the out-of-order stored data can be reorganized according to the real transaction sequence during querying.

[0021] The advantages and beneficial effects of this invention are as follows:

[0022] This invention provides a cloud-native architecture-compatible transaction timestamp-injected data time-series reconstruction method. It aims to resolve the fundamental contradiction between the disordered storage of cloud-native architecture and the requirements of financial time-series processing, as well as the need for fault tolerance. The method establishes a time-series protection layer independent of the infrastructure. Targeting the characteristics of market transaction data, it reconstructs the transaction data time-series through the injection of an authoritative time source (transaction timestamp), ensuring the authenticity of the transaction sequence. It constructs a time-series reconstruction scheme based on native database functions, designs a lightweight field injection mechanism to avoid intrusive modifications to cloud service APIs, and improves data fault tolerance by establishing a lightweight data fault tolerance mechanism. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method in an embodiment of the present invention. Detailed Implementation

[0024] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0025] Taking stock trading as an example, in the industry's common architecture, producers write stock market data to a cloud-hosted message queue (such as Confluent Cloud), and then consumers dump it to a cloud database (such as Aiven PostgreSQL). This framework has an inherent flaw:

[0026] 1. The partitioning mechanism and parallel consumption of message queues lead to disordered data storage;

[0027] 2. In the event of consumer failure or network anomaly, the message offset cannot reconstruct the true transaction sequence;

[0028] 3. The database lacks authoritative time references, resulting in low reliability of historical data analysis.

[0029] 4. In addition, the data has poor fault tolerance. When the data consumer fails, it may result in data loss or duplicate consumption.

[0030] Therefore, this invention proposes a transaction timestamp injection data time-series reconstruction method compatible with cloud-native architecture, used to process real-time market data of stock transactions, focusing on monitoring the abnormal movements of several technology stocks. Deployed in a multi-cloud environment, producers and consumers run on different cloud platforms, with Confluent Cloud serving as the message hub and Aiven PostgreSQL providing data persistence. Figure 1 As shown, the normal processing flow includes the following steps:

[0031] Step 101: Data Acquisition and Transmission; The producer obtains the latest stock trading data package through a third-party API, including stock price, trading volume, and the time stamp generated by the API. The data is encrypted and sent to a specific topic in the cloud-hosted message queue ConfluentCloud. The cloud service automatically distributes the messages across different partitions.

[0032] Transparent transmission at the production layer: Producers retain the original data acquisition function, writing market data containing core fields such as stock codes, prices, and trading volumes into the message queue. The production layer explicitly relinquishes control over timing, fully complying with the unordered storage characteristics of cloud services, providing a compatibility foundation for upper-layer innovation.

[0033] Step 102: Transaction timestamp injection; When the consumer pulls raw data from the message queue, it directly connects to the exchange's official time service via a dedicated line, obtains the precise time point based on the transaction order number, generates an authoritative transaction timestamp (trade_time field), and injects it into the message body. At the same time, it verifies the deviation between the third-party API timestamp and the exchange's time. If the deviation value is within the allowable range, it is marked as trustworthy.

[0034] Consumer layer timestamp injection: After the consumer reads the message but before writing it to the database, the dynamic injection process is carried out by embedding the transaction timestamp to support subsequent time sequence reconstruction. It also takes into account the two cases where the original data contains or does not contain the transaction time.

[0035] (1) For cases where the original data does not contain transaction time information:

[0036] Obtain accurate and authoritative trading times through the exchange's official time-authorization interface to generate... Fields are embedded in data records. The time synchronization process follows financial industry standard protocols (such as NTP-FIX) and satisfies the formula:

[0037]

[0038] in, Indicates the standard timestamp of the exchange from which the injection occurred. This indicates the actual time when the transaction occurred (recorded by the exchange system). This represents the network time delay compensation value (obtained through the exchange's certified channel).

[0039] This mechanism ensures that timestamps have legal authority, laying the foundation for subsequent time-series reconstruction.

[0040] (2) For cases where the original data already contains transaction time information:

[0041] Upgrade the timestamp on the consumer side by downgrading the original timestamp (the transaction time in the original data) to the main data text. Simultaneously, similar to step (1), obtain and inject an authoritative transaction timestamp from the exchange's official website. Based on this, establish a dual timestamp verification system:

[0042]

[0043] in, This indicates the precise time point at which the exchange's official trade matching engine generated the trades. This indicates the timestamp added by the third-party API during data encapsulation. ( If a pre-set threshold is met, then accept. As a supplementary field, when data enters the system from a third-party API, the supplementary field can serve as original time credentials, thus establishing a complete data chain. When verifying the authenticity of a transaction, auditors can compare the time in the supplementary field with the authoritative time from the exchange. This will trigger a data credibility audit.

[0044] Use the updated and upgraded timestamp as Fields are embedded in data records.

[0045] Step 103: Data Persistence; The enhanced message is written to the cloud database Aiven PostgreSQL, which automatically creates daily partitioned tables based on the trade_time field. The latest data is stored in the hot data area in native JSON format, while expired data is moved to compressed cold storage.

[0046] Storage layer timeline reconstruction: Database receiving band After processing the data in the field, a B+ tree index is built using it as the primary key. The time-series reconstruction process can be formally expressed as:

[0047]

[0048] in, Indicates based on sorting operators, Represents the original unordered dataset. Indicates the start time boundary of the query. Indicates the query termination time boundary.

[0049] This step transfers timing control from the infrastructure layer to the application layer, enabling out-of-order stored data to be reassembled according to the actual transaction sequence during querying.

[0050] The troubleshooting process for fault scenarios includes the following steps:

[0051] Step 201: The third-party API that provides stock trading data temporarily adjusted the JSON structure due to a system upgrade, causing some messages to fail to be parsed. The consumer layer retried the engine and failed to parse the messages in n consecutive attempts.

[0052] A lightweight fault-tolerance mechanism based on the exponential backoff algorithm and DLQ queue improves the fault tolerance of stock trading data.

[0053] Persistent write-ahead local logs: After the consumer process pulls messages, it first stores the key metadata of the messages (including message offset, authoritative transaction timestamp, etc.). Data fingerprints, hashes, etc., are written to the write-ahead log on the local SSD in a sequential append manner. Log writing uses the O_DIRECT mode to bypass the operating system cache. Log files are stored in shards based on consumer group ID and Kafka partition number. Each shard follows a rolling overwrite strategy: when a single log file exceeds a certain size (e.g., 128MB), a new file is created, and the oldest file is deleted to control storage overhead.

[0054] Adaptive tiered retries: When message processing fails due to a temporary failure (such as a database connection interruption), the retry engine initiates a tiered recovery process. The initial retry interval is a fixed value. (For example, 100ms), the interval is subsequently increased exponentially using the backoff algorithm:

[0055]

[0056] in, Indicates the basic interval. This indicates the number of retries already completed. Indicates the maximum interval.

[0057] If repeated attempts are made If the process fails again, the circuit breaker mechanism is triggered: the Kafka partition containing the message is marked as degraded, and the consumer suspends consumption of that partition for a fixed period of time. (For example, 30 seconds), while transferring the message to the dead-letter queue (DLQ).

[0058] Step 202: The system automatically transfers the abnormal message to the Dead-Letter Queue (DLQ) to fully preserve the original message and double timestamps. The independent daemon process detects the new dead-letter record and triggers the first asynchronous retry, but it still fails. The operation and maintenance personnel check the error details through the diagnostic panel and find that the field name has changed from close_price to last_price.

[0059] The main functions of dual timestamps include the following: First, dual timestamps can be used for rapid fault location. When there is a discrepancy between the dual timestamps, if the official transaction time distribution is confirmed to be normal, the cause of the fault can be located based on the time of the third-party API. At the same time, the data quality score can be determined based on the level of the discrepancy value, thereby determining the priority of repair.

[0060] Each record in the Dead Letter Queue (DLQ) contains fields such as the original message binary, failure reason, number of retries, and last failure timestamp. A background daemon is deployed in the system to periodically scan the DLQ table. For retries... Second-rate( If the message still fails (based on a fixed empirical value), an alarm will be triggered to notify manual intervention.

[0061] Step 203: Developers create a data transformation patch and hot-deploy it to the consumer cluster. The daemon automatically retrieves dead-letter messages, and the problem is successfully resolved after repair. The system records the root cause of this failure, and the repair rules will be automatically applied to similar errors in the future.

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A transaction timestamp injection data timing reconstruction method compatible with a cloud-native architecture, applied to a data consumer end, characterized in that: Obtain transaction data uploaded by a data production end from a cloud message queue, and inject a transaction time obtained from an exchange into the transaction data; Send the transaction data with the injected transaction time to the cloud to enable the cloud to create partitions according to the transaction time, so that new transaction data is stored in a hot data area and old transaction data is stored in a cold data area; When the transaction data obtained from the cloud does not contain a transaction time, generate a new transaction time based on the transaction time obtained from the exchange and network time service delay compensation obtained from an authentication channel of the exchange, and inject the new transaction time into the transaction data; When the transaction data obtained from the cloud contains a transaction time, compare a difference between the contained transaction time and the transaction time obtained from the exchange with a threshold value, and if the difference is less than the threshold value, downgrade the contained transaction time to a data body as an auxiliary field, and if the difference is greater than or equal to the threshold value, trigger credibility auditing of the transaction data.

2. The transaction timestamp injection data time sequence reconstruction method compatible with the cloud native architecture according to claim 1, characterized in that: The auxiliary field serves as an original time certificate of generation of the transaction data, and when it is necessary to verify authenticity of the transaction data, the authenticity is verified by comparing the auxiliary field with the transaction time obtained from the exchange; Based on the transaction time of the exchange and the transaction time in the digital body as the auxiliary field, fault positioning is performed, when the two transaction times deviate, if it is determined that the transaction time distribution of the exchange is normal, the fault cause is located according to the transaction time of the auxiliary field, and meanwhile, according to the level of the deviation value of the two transaction times, a data quality score is determined to decide a repair priority.

3. The transaction timestamp injection data time sequence reconstruction method compatible with the cloud native architecture according to claim 1, characterized in that: Obtain cloud transaction data and perform parsing, and pre-write key metadata to a local log file in a sequential appending manner, and the log file is stored in shards according to a consumer group ID and a Kafka partition number; If the transaction data fails to be parsed, a retry engine is started to perform multiple times of parsing, if the multiple times of parsing all fail, a fuse mechanism is triggered, a consumer suspends consumption of a corresponding Kafka partition, and corresponding transaction data is transferred to a dead letter queue.

4. The transaction timestamp injection data timing reconstruction method compatible with the cloud-native architecture according to claim 3, characterized in that: The retry engine employs adaptive hierarchical retries, with a fixed value for the first retry interval and subsequent intervals multiplied by an exponential backoff algorithm: ， wherein, denotes a base interval, denotes a number of retries, denotes a maximum interval.

5. The transaction timestamp injection data timing reconstruction method compatible with the cloud-native architecture according to claim 3, characterized in that: A daemon periodically scans the dead letter queue, when a new dead letter message is detected, asynchronous retry is triggered, if the retry fails, an alarm is triggered for repair; A hot deployment data conversion patch is used to automatically recover dead letter messages by a daemon, for dead letter messages that are successfully repaired, fault root causes are recorded to automatically apply a repair rule for future similar errors.

6. A transaction timestamp injection data timing reconstruction method compatible with a cloud-native architecture, applied to a data production end, characterized in that: Obtain transaction data and upload the transaction data to a cloud message queue, so that a data consumer injects a transaction time obtained from an exchange into the transaction data, and then enables the cloud to create partitions according to the injected transaction time, so that new transaction data is stored in a hot data area and old transaction data is stored in a cold data area; When transaction data obtained by the data consumer from the cloud does not contain a transaction time, generate a new transaction time based on the transaction time obtained from the exchange and network time service delay compensation obtained from an authentication channel of the exchange, and inject the new transaction time into the transaction data; When the transaction data obtained by the data consumer from the cloud contains a transaction time, compare a difference between the contained transaction time and the transaction time obtained from the exchange with a threshold value, and if the difference is less than the threshold value, downgrade the contained transaction time to a data body as an auxiliary field, and if the difference is greater than or equal to the threshold value, trigger credibility auditing of the transaction data.

7. The transaction timestamp injection data timing reconstruction method compatible with the cloud-native architecture according to claim 6, characterized in that: The data production end obtains the latest transaction data through a third-party interface, and the transaction data is sent to a certain topic of a cloud hosting message queue after being encrypted, and the cloud service automatically stores the message in different partitions.

8. A transaction timestamp injection data time sequence reconstruction method compatible with a cloud-native architecture, applied to a cloud, characterized in that: The transaction data uploaded by the data production end is obtained and stored in the cloud message queue, so that the consumption end obtains the transaction data in the message queue and injects the transaction time obtained from the exchange; The transaction data with the injected transaction time is obtained, and a partition is created according to the transaction time, the new transaction data is stored in the hot data area, and the old transaction data is stored in the cold data area; When the transaction data given to the data consumption end does not contain the transaction time, a new transaction time is generated based on the transaction time obtained from the exchange and the network time delay compensation obtained from the exchange authentication channel, and the new transaction time is injected into the transaction data; When the transaction data given to the data consumption end contains the transaction time, the difference between the contained transaction time and the transaction time obtained from the exchange is compared with the threshold value, if it is less than the threshold value, the contained transaction time is degraded as the data body, if it is greater than or equal to the threshold value, the credibility of the transaction data is audited.

9. The transaction timestamp injection data timing reconstruction method compatible with the cloud-native architecture according to claim 8, characterized in that: After the database of the cloud obtains the transaction data with the transaction time, the B+ tree index is constructed with the transaction time as the primary key to reconstruct the time sequence of the storage layer, specifically, based on the original data set without data and the query start time boundary and the query end time boundary, the time sequence is reconstructed through the sorting operator of the transaction time, so that the control right of the time sequence is transferred from the infrastructure layer to the application layer, and the out-of-order stored data is reorganized according to the real transaction sequence when queried.

Images