Transaction data processing method

Abstract

The invention discloses a transaction data processing method, relates to the technical field of data processing, and is suitable for transaction data real-time processing and risk control monitoring in a high-concurrency scene. The method comprises the steps that transaction data are collected and preprocessed, and a transaction data structure of a unified format is constructed; performing data fragmentation based on a distributed consistency hash rule; extracting multi-dimensional feature vectors to construct a transaction behavior feature set; calculating an abnormal score of the transaction sample based on the isolated forest model, and judging a transaction abnormal state according to a score result; verifying the multi-fragment data processing consistency through a consistency verification function; a detection result and abnormal feedback are reported through a RESTfulAPI interface; and the number of system fragments is dynamically adjusted according to the real-time throughput, so that load balancing is realized. The method has the advantages of efficient data processing, intelligent anomaly detection, flexible system expansion and the like, and is suitable for various electronic commerce, financial payment and real-time transaction monitoring systems.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method for processing transaction data. Background Technology

[0002] With the rapid development of emerging business models such as e-commerce, internet finance, and mobile payment, user transaction behavior is exhibiting characteristics of high frequency, strong real-time nature, and complexity and diversity. This has led to a continuous increase in the scale of data generated in transaction systems, increasingly complex data structures, and inconsistent data quality. In practical applications, enterprises and platforms typically need to rapidly collect, clean, identify, analyze, and store massive amounts of transaction data to support subsequent business decisions, risk control identification, anomaly detection, and user profiling.

[0003] Traditional transaction data processing methods often rely on static rules or preset processes, which are ill-suited to the dynamic and uncertain nature of transaction data. This is especially true in high-concurrency scenarios such as flash sales, limited-time offers, and holiday promotions, where systems are prone to data lag, anomaly omissions, or processing bottlenecks. Furthermore, in identifying abnormal transactions, existing methods often depend on fixed thresholds or rule-based judgments, making it difficult to identify sophisticated and frequently changing fraudulent transactions, resulting in high false positive and false negative rates.

[0004] Furthermore, current distributed transaction processing systems face challenges in data consistency verification, dynamic resource scheduling, and load balancing. Traditional consistency verification methods may lead to performance bottlenecks in high-concurrency environments, failing to guarantee the consistency and accuracy of processing results. Static sharding strategies often cannot adaptively adjust to business pressure, potentially causing some nodes to overload and affecting the overall system stability.

[0005] Therefore, there is an urgent need for a transaction data processing method with real-time acquisition capabilities, intelligent anomaly detection mechanisms, support for distributed consistency verification, and dynamic resource scheduling capabilities, in order to improve the processing efficiency and accuracy of the transaction system in high-concurrency and highly dynamic environments, and ensure the stability of business operations and the reliability of data processing.

[0006] Therefore, we urgently need to design a transaction data processing method to solve these problems. Summary of the Invention

[0007] The purpose of this invention is to solve the existing technical problems mentioned in the background section and to provide a transaction data processing method.

[0008] The above-mentioned objective of the present invention is achieved as follows: a transaction data processing method, comprising the following steps:

[0009] (a) Real-time acquisition and preprocessing of transaction data, wherein the real-time acquisition is achieved through a high-throughput message queue, which receives raw data streams from the transaction system. These raw data streams include transaction records encapsulated in JSON or XML format. Each transaction record contains transaction time t, transaction amount a, user identifier u, transaction type c, and transaction identifier tid. The message queue employs a partitioning mechanism, with the number of partitions P dynamically adjusted based on the transaction throughput Q. The partitioning formula is:

[0010]

[0011] Where Q is the number of transactions per second, and κ is the maximum processing capacity of a single partition; the preprocessing includes data cleaning and format standardization. Data cleaning removes redundant fields and fills in missing values ​​through a rule engine, and format standardization converts transaction records into a unified structure, which is defined as a quintuple (t, a, u, c, tid).

[0012] (b) A distributed database is used to store transaction data, and a parallel computing framework is used to shard and process the data;

[0013] (c) Detect anomalies in transaction data based on machine learning algorithms and generate anomaly scores;

[0014] (d) Maintain data consistency through a distributed transaction management mechanism;

[0015] (e) Store the processing results and feed them back to the trading system via the application programming interface.

[0016] As a preferred technical solution of the present invention, the data cleaning includes removing redundant fields and filling in missing values, and the format standardization converts the transaction data into a unified structure. The transaction data includes transaction time t, transaction amount a, user identifier u, and transaction type c.

[0017] As a preferred embodiment of the present invention, the distributed database is a columnar storage database, and the parallel computing framework adopts a memory-based stream processing mechanism. Data sharding is performed based on transaction time t or user identifier u, and the sharding formula is as follows:

[0018] S i =hash(t mod N) or S i =hash(u mod N),

[0019] Among them, S i Let represent the i-th shard, N represent the total number of shards, hash is the hash function, t is the transaction time, and u is the user identifier.

[0020] As a preferred technical solution of the present invention, the anomaly detection is based on the isolated forest algorithm, and the feature vector x = [a, f, t] d ,c e ], where a is the transaction amount, f is the transaction frequency, and t is the transaction value. d The transaction time interval c e For transaction type encoding, the anomaly score A(x) is calculated using the following formula:

[0021]

[0022] Where E(h(x)) is the average path length of sample x in the isolated forest, c(n) is the average path length of the binary search tree, n is the number of training samples, and c(n) is defined as:

[0023]

[0024] Where H(k) is the harmonic number,

[0025] As a preferred technical solution of the present invention, the transaction frequency f is calculated by the following formula:

[0026]

[0027] Among them, C u (Δt) represents the number of transactions by user u within the time window Δt, where |Δt| is the length of the time window. The time interval Δt is calculated using the following formula:

[0028] Δt=t i -t i-1 ,

[0029] Among them, t i t represents the current trading time. i -1 represents the previous trading time.

[0030] As a preferred technical solution of the present invention, the distributed transaction management mechanism adopts a two-phase commit protocol, including a preparation phase and a commit phase. In the preparation phase, each shard node verifies data consistency, and the consistency verification formula is:

[0031]

[0032] Where V is the check value, and D j Let M be the number of shard nodes, and H be the data of the j-th shard. global This is the global data hash value.

[0033] As a preferred technical solution of the present invention, if the consistency check fails, a rollback mechanism is triggered, and the rollback operation is logged. The log format includes the transaction identifier tid and the operation time t.o and error code e c .

[0034] As a preferred technical solution of the present invention, the processing result includes transaction status s, anomaly marker e, and processing timestamp t. p The transaction status is reported in JSON format via a RESTful API interface. The transaction status includes three states: success, failure, and pending.

[0035] As a preferred technical solution of the present invention, the isolated forest model is trained through the following optimization objectives:

[0036]

[0037] Among them, y i For sample x i The true anomaly label, λ is the regularization parameter, T is the number of trees, and d k Let A(x) be the depth of the k-th tree. i ) indicates an anomaly score.

[0038] As a preferred technical solution of the present invention, the method further includes adaptively adjusting the number of shards N, and the adjustment formula is:

[0039]

[0040] Where Q is the current transaction throughput, θ is the maximum processing capacity of a single shard, and N... new The throughput Q, after adjustment of the number of fragments, is calculated using the following formula:

[0041]

[0042] Where C(t) is the number of transactions at time t, and T w For the statistical time window, |T w | represents the window length.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This invention proposes a transaction data processing method that enables real-time acquisition, cleaning, unified structure transformation, and dynamic sharding of transaction data in high-concurrency scenarios. Combined with an anomaly detection mechanism based on the isolated forest model, it can intelligently identify potential abnormal transaction behaviors without relying on fixed rules, effectively improving the accuracy and generalization ability of detection. Through distributed consistency verification methods and hash comparison mechanisms, it ensures the global consistency of data processing results in a multi-sharded system, preventing the omission or duplicate identification of abnormal data in the distributed architecture.

[0045] Furthermore, this invention introduces an adaptive sharding adjustment strategy, dynamically estimating and adjusting system resource distribution based on actual transaction throughput. This achieves an optimal balance between resource utilization efficiency and response speed in the transaction data processing system, avoiding resource waste or processing bottlenecks. Through logging and RESTful API feedback mechanisms, the system can achieve full-process tracking and status feedback of anomaly handling results, enhancing system controllability and maintainability, and possessing significant technical promotion value and practical application prospects. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 A flowchart of a transaction data processing method;

[0048] Figure 2 This is a diagram illustrating the components of data cleaning in a transaction data processing method. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments, but this is not intended to limit the scope of protection of the present invention.

[0051] See Figure 1 and Figure 2 This invention provides a transaction data processing method applicable to the real-time collection, cleaning, storage, anomaly detection, and dynamic sharding management of massive transaction data in scenarios such as internet finance, e-commerce platforms, and online payment systems. Its core technical approach is based on high-throughput data stream acquisition, efficient data cleaning and preprocessing, an outlier detection algorithm based on isolated forests, a sharding consistent hash allocation strategy, and a feedback dynamic adjustment mechanism, ensuring the accuracy, real-time performance, and scalability of transaction data processing.

[0052] During system operation, real-time transaction data is first collected through a high-throughput data acquisition module. The collected data includes transaction time t, transaction amount a, user identifier u, transaction category code c, and unique transaction identifier tid. The collected data is then sorted by timestamp t and combined into a unified structured quintuple (t, a, u, c, tid). The amount of data collected per second is denoted as Q, and the number of dynamic partitions P is estimated using the following formula:

[0053]

[0054] Where k represents the data processing capacity of a single partition. The preprocessing process includes steps such as missing value imputation, format normalization, outlier correction, and consistent structure construction.

[0055] Subsequently, the system enters the management module based on distributed data storage, where the standardized data is sharded and distributed. The shard number is calculated using a consistent hash function, and the sharding rule is defined as: S t =hash(tmodN),S u =hash(umodN),

[0056] Where N is the total number of current shards, t is the transaction timestamp, and u is the user identifier. To ensure a balanced distribution of data across time and user dimensions and improve distributed processing efficiency, the system introduces an anomaly detection algorithm based on Isolation Forest to identify abnormal transaction behavior. The training feature vector is set to [a, f, t]. d [,c,e], where a is the transaction amount, f is the user's transaction frequency within the time window Δt, and t d Where is the transaction time interval, 'c' is the transaction category code, and 'e' is the system anomaly feedback value. The formula for calculating transaction frequency is:

[0057] Among them, C u (Δt) represents the time window Δt = t for user u. i -t i-1 The number of transactions within the isolated forest. The average path depth c(n) of each tree in an isolated forest can be expressed as:

[0058] For sample x, its anomaly scoring function is:

[0059]

[0060] Where E(h(x)) is the average path length of the sample in the isolated forest.

[0061] After anomaly detection, the consistency of the results is verified using a consistent hashing mechanism: Among them, D j H represents the data block in the j-th slice. global This represents the global hash check value, used to verify consistency between shards.

[0062] For the detected abnormal transaction x i If its score A(x) i If the threshold is exceeded, an exception handling process is triggered, and the transaction identifier tid and operation time t are recorded. p With system error code e c The processing status (success / failure / pending) is reported via a RESTful API interface in JSON format.

[0063] The anomaly detection model is tuned by optimizing the objective function, which is defined as: Among them, y i For sample x i The true anomaly label, λ is the regularization factor, d k This is a depth metric for tree models, used to balance model complexity and generalization ability.

[0064] To adapt to different business peaks, the system further introduces an adaptive adjustment mechanism, which dynamically adjusts the number of shards N based on the current throughput Q and the maximum processing capacity θ of a single shard. new The formula is adjusted as follows: The throughput Q is calculated as follows:

[0065] Where C(t) is the number of transactions at time point t, and T w For the statistical window interval, |T w | represents the window length. This mechanism can automatically scale the system's processing resources according to fluctuations in business pressure, ensuring the stability and responsiveness of data processing.

[0066] In summary, the transaction data processing method provided by this invention has high throughput processing capability, flexible anomaly identification mechanism, distributed data consistency verification and adaptive resource allocation capability, which significantly improves the accuracy, real-time performance and system elasticity of transaction data processing, and is suitable for promotion and application in various large-scale online trading systems.

[0067] Example 2: Taking the "Daily Flash Sale" business of an e-commerce platform as an example, during peak hours (such as 10:00 to 11:00 daily), the platform generates an average of 3600 transaction data entries per second (Q = 3600). The system uses the transaction data processing method described in this invention to process transaction behavior in real time and identify anomalies, ensuring stable system operation.

[0068] I. Data Acquisition and Preprocessing: The system collects user transaction data in real time through the Kafka streaming data platform. The data structure is as follows:

[0069] (t,a,u,c,tid)

[0070] For example, a transaction record is as follows: Timestamp: t = 10:01:25.312, Amount: a = 239.00; User ID: u = 985721; Transaction Type: c = "Flash Sale"; Transaction ID: tid = SK20250619A001;

[0071] The collected data is cleaned through an ETL process, missing values ​​are filled in and the format is standardized. The timestamp is converted to UNIX format, the monetary unit is standardized to "yuan", and the transaction type is numerically encoded, such as "flash sale" is encoded as 1.

[0072] II. Data Sharding and Hash Allocation: The system sets the maximum processing capacity of each shard to k = 1000, and estimates the number of shards needed based on the transaction rate.

[0073]

[0074] The segment number is generated as follows:

[0075] S t =hash(tmod4),S u =hash(umod4)

[0076] For example, the hash value of user 985721 is:

[0077] S u =hash(985721mod 4)=hash(1)=2

[0078] The transaction was distributed to shard processing node #2.

[0079] III. Application of the Anomaly Detection Model: The system collects the following features to form the training vector:

[0080] Transaction amount a = 239.00 yuan; Current frequency f = 0.25 (the user transacts once within this time window, |Δt| = 4 seconds); Category code c = 1; Anomaly feedback value e = 0; Transaction time interval t d =t i -t i-1 =4s

[0081] The vector is denoted as [239.00, 0.25, 4, 1, 0]; the training model uses an isolated forest of 100 trees for prediction, and the average path length of the samples is:

[0082] H(99)≈ln(99)+γ≈5.18+0.577≈5.76;

[0083]

[0084] Assuming the average path length of the sample is E(h(x)) = 3.6, then the anomaly score is: According to the platform's experience rules, if A(x) > 1.5, it is a suspicious transaction, and this sample is not considered abnormal.

[0085] IV. Consistency Verification and Anomaly Feedback: If a shard detects an anomaly in a transaction record (e.g., A(x)... i (2.3), this record submits an exception feedback via a RESTful API, and the log recording format is:

[0086] {"tid":"SK20250619A077",

[0087] "tp":"10:01:37.108",

[0088] The feedback "ec":"ERR-AFP-003"} is marked as "Pending Exception" in the log system.

[0089] V. Example of dynamic adjustment mechanism: If the platform detects a trading surge within the current one-minute window, the sliding window T... w Internal statistics are as follows: T w =[10:01:00,10:02:00], |T w = 60 seconds

[0090] Calculate the current throughput: If the maximum processing capacity per slice is set to θ = 1200, then adjust the number of slices: The system maintains the existing 4-chip configuration without expansion; if Q increases to more than 4800, the system will automatically expand to 5 chips to ensure stable data access and processing.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing transaction data, characterized in that, Includes the following steps: (a) Real-time acquisition and preprocessing of transaction data, wherein the real-time acquisition is achieved through a high-throughput message queue, which receives raw data streams from the transaction system. These raw data streams include transaction records encapsulated in JSON or XML format. Each transaction record contains transaction time t, transaction amount a, user identifier u, transaction type c, and transaction identifier tid. The message queue employs a partitioning mechanism, with the number of partitions P dynamically adjusted based on the transaction throughput Q. The partitioning formula is: Where Q is the number of transactions per second, and κ is the maximum processing capacity of a single partition; the preprocessing includes data cleaning and format standardization. Data cleaning removes redundant fields and fills in missing values ​​through a rule engine, and format standardization converts transaction records into a unified structure, which is defined as a quintuple (t, a, u, c, tid). (b) A distributed database is used to store transaction data, and a parallel computing framework is used to shard and process the data. Anomaly detection is performed on the transaction data based on machine learning algorithms to generate anomaly scores. (d) Maintain data consistency through a distributed transaction management mechanism; (e) Store the processing results and feed them back to the trading system via the application programming interface.

2. The transaction data processing method according to claim 1, characterized in that, The data cleaning includes removing redundant fields and filling in missing values. The format standardization converts the transaction data into a unified structure. The transaction data includes transaction time t, transaction amount a, user identifier u, and transaction type c.

3. The transaction data processing method according to claim 1, characterized in that, The distributed database is a columnar storage database, and the parallel computing framework adopts a memory-based stream processing mechanism. Data sharding is performed based on transaction time t or user identifier u, and the sharding formula is as follows: S i = hash(t mod N) or S i = hash(u mod N), Among them, S i Let represent the i-th shard, N represent the total number of shards, hash is the hash function, t is the transaction time, and u is the user identifier.

4. The transaction data processing method according to claim 1, characterized in that, The anomaly detection is based on the Isolation Forest algorithm, with feature vector x = [a, f, t]. d ,c e ], where a is the transaction amount, f is the transaction frequency, and t is the transaction value. d The transaction time interval c e For transaction type encoding, the anomaly score A(x) is calculated using the following formula: Where E(h(x)) is the average path length of sample x in the isolated forest, c(n) is the average path length of the binary search tree, n is the number of training samples, and c(n) is defined as: Where H(k) is the harmonic number, 5. A transaction data processing method according to claim 4, characterized in that, The trading frequency f is calculated using the following formula: Among them, C u (Δt) represents the number of transactions by user u within the time window Δt, where |Δt| is the length of the time window. The time interval Δt is calculated using the following formula: Δt=t i -t i-1 , Among them, t i t represents the current trading time. i -1 represents the previous trading time.

6. The transaction data processing method according to claim 1, characterized in that, The distributed transaction management mechanism adopts a two-phase commit protocol, including a preparation phase and a commit phase. In the preparation phase, each shard node verifies data consistency, and the consistency verification formula is: Where V is the check value, and D j Let M be the number of shard nodes, and H be the data of the j-th shard. global This is the global data hash value.

7. A transaction data processing method according to claim 6, characterized in that, If the consistency check fails, a rollback mechanism is triggered. The rollback operation is logged, and the log format includes the transaction identifier tid and the operation time t. o and error code e c .

8. The transaction data processing method according to claim 1, characterized in that, The processing result includes transaction status s, anomaly marker e, and processing timestamp t. p The transaction status is reported in JSON format via a RESTful API interface. The transaction status includes three states: success, failure, and pending.

9. A transaction data processing method according to claim 4, characterized in that, The isolated forest model is trained with the following optimization objectives: Among them, y i For sample x i The true anomaly label, λ is the regularization parameter, T is the number of trees, and d k Let A(x) be the depth of the k-th tree. i ) indicates an anomaly score.

10. A transaction data processing method according to claim 1, characterized in that, The method further includes adaptively adjusting the number of fragments N, with the adjustment formula being: Where Q is the current transaction throughput, θ is the maximum processing capacity of a single shard, and N... new The throughput Q, after adjustment of the number of fragments, is calculated using the following formula: Where C(t) is the number of transactions at time t, and T w For the statistical time window, |T w | represents the window length.

Images