Transaction data processing method and device, storage medium and computer program product

Abstract

The invention discloses a transaction data processing method and device, a storage medium and a computer program product, and relates to the technical field of data processing.The method comprises the steps that after online transaction of a distributed account is completed, real-time transaction data of the online transaction is registered to a dual-cluster message queue through a bypass asynchronous thread, the dual-cluster message queue comprises a main cluster queue and an auxiliary cluster queue; consuming the first transaction data in the main cluster queue based on the aggregation application instance, and storing the first transaction data to a main transaction table of a database; and on the basis of the aggregation application instance, consuming the second transaction data in the secondary cluster queue, determining missing target transaction data in the first transaction data stored in the main transaction table in the second transaction data, and storing the target transaction data to the main transaction table of the database. According to the invention, the accuracy and integrity of the real-time transaction data of the distributed account in the database are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a transaction data processing method, device, storage medium and computer program product. BACKGROUND

[0002] With the advent of the cloud era, transaction account management gradually evolves from the traditional centralized mode to distributed management. This change greatly improves the horizontal expansion capability of transaction business and better meets the growing and diversified needs of transaction business.

[0003] However, distributed management also brings new challenges to operation management. When managing transaction accounts, it is necessary to timely understand the internal operation status in dozens or even hundreds of clusters, including the transaction and balance of accounts. The conventional solution is to call another microservice to register transaction processing in the online transaction mode. However, the online transaction calling another microservice for processing will prolong the calling link and increase the risk of single point failure. Once a microservice fails, it may block the entire transaction processing flow, even cause data loss or inconsistency, and greatly affect the integrity and accuracy of transaction data. SUMMARY

[0004] The main purpose of the present application is to provide a transaction data processing method, device, storage medium and computer program product, aiming to solve the technical problem of low integrity of transaction data.

[0005] To achieve the above purpose, the present application provides a transaction data processing method, which comprises:

[0006] After the completion of the online transaction of the distributed account, the real-time transaction data of the online transaction is registered to a double-cluster message queue through a bypass asynchronous thread, the double-cluster message queue comprises a main cluster queue and a secondary cluster queue, the main cluster queue is used for the persistence of first transaction data, the secondary cluster queue is used for the delay compensation of the first transaction data, and the first transaction data is real-time transaction data in the main cluster queue;

[0007] Based on the aggregation application instance, the first transaction data in the main cluster queue is consumed, and the first transaction data is stored to the main transaction table of the database;

[0008] Based on the aggregation application instance, the second transaction data in the secondary cluster queue is consumed, the target transaction data missing in the first transaction data stored in the main transaction table is determined in the second transaction data, and the target transaction data is stored to the main transaction table of the database, and the second transaction data is real-time transaction data in the secondary cluster queue.

[0009] In an embodiment, the step of determining target transaction data in the first transaction data stored in the main transaction table from the second transaction data comprises:

[0010] querying whether the second transaction data exists in the main transaction table according to a keyword of the second transaction data;

[0011] determining the second transaction data that does not exist in the main transaction table as target transaction data;

[0012] storing the target transaction data into at least two partitions in a transaction sub-table of the database, the transaction sub-table being a partition table;

[0013] the step of storing the target transaction data into the main transaction table of the database comprises:

[0014] after a preset time period, consuming the target transaction data in the transaction sub-table based on an aggregated application instance;

[0015] storing the target transaction data in the transaction sub-table into the main transaction table.

[0016] In an embodiment, the method comprises:

[0017] determining at least two data processing tasks according to a number of partitions corresponding to the target transaction data in the transaction sub-table and a number of aggregated application instances;

[0018] executing at least two data processing tasks;

[0019] wherein the data processing tasks are independent of each other, and the data processing tasks are used to execute the consumption of the target transaction data in the transaction sub-table based on an aggregated application instance after a preset time period, and store the target transaction data in the transaction sub-table into the main transaction table.

[0020] In an embodiment, after the step of executing at least two data processing tasks, the method further comprises:

[0021] querying a task state of the data processing task in a task list according to a business type of the data processing task;

[0022] performing an associated processing operation according to the task state of the data processing task;

[0023] wherein the task list is generated based on a preset rule, and the preset rule comprises an account transaction volume.

[0024] In an embodiment, the step of performing an associated processing operation according to the task state of the data processing task comprises:

[0025] When the task state is pending, the task state is modified to be in execution, and the online processing is converted to asynchronous processing;

[0026] When the task state is in execution, according to the processed time length and the preset timeout time length, it is determined whether the data processing task is timed out, and the task state of the timed-out data processing task is updated to be in execution exception;

[0027] When the task state is execution success, the task state is reset to be in pending state;

[0028] When the task state is in execution exception, a submission retry processing is performed.

[0029] In an embodiment, the step of registering the real-time transaction data of the online transaction to the dual-cluster message queue through the bypass asynchronous thread after the online transaction of the distributed account is completed comprises:

[0030] After the online transaction of the distributed account is completed for a preset stage, the real-time transaction data of the online transaction is sent to a cache instance;

[0031] The real-time transaction data in the cache instance is sent to an asynchronous to-be-sent cache queue;

[0032] Based on a background thread, the real-time transaction data in the to-be-sent cache queue is registered to the dual-cluster message queue respectively.

[0033] In an embodiment, the step of consuming the first transaction data in the master cluster queue based on the aggregation application instance and storing the first transaction data to the main transaction table of the database comprises:

[0034] Based on the aggregation application instance, the first transaction data in the master cluster queue is consumed in real time, and the first transaction data is batch-persisted to the main transaction table of the database.

[0035] In addition, to achieve the above-mentioned purposes, the present application also proposes a transaction data processing device, which comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the transaction data processing method as described above.

[0036] In addition, to achieve the above-mentioned purposes, the present application also proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the transaction data processing method as described above.

[0037] In addition, to achieve the above object, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the transaction data processing method as described above.

[0038] The one or more technical solutions proposed in the present application have at least the following technical effects:

[0039] By storing real-time transaction data in the dual-cluster message queue, even if the primary cluster queue fails, the real-time transaction data in the secondary cluster queue can serve as a backup, reducing the risk of data loss and enhancing the fault tolerance of the system. By comparing the real-time transaction data in the primary cluster queue and the secondary cluster queue, the target transaction data is determined and stored in the same database, thereby ensuring the eventual consistency of the transaction data in the database and avoiding the problem of inconsistent data caused by network failures, system abnormalities, etc., ensuring the accuracy and integrity of the real-time transaction data of the distributed accounts in the database. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0042] Figure 1 Flowchart provided for the transaction data processing method embodiment one of the present application;

[0043] Figure 2 Brief flowchart of transaction data registration provided for the transaction data processing method embodiment one of the present application;

[0044] Figure 3 Flowchart of transaction data persistence in the primary cluster queue provided for the transaction data processing method embodiment one of the present application;

[0045] Figure 4 Flowchart provided for the transaction data processing method embodiment two of the present application;

[0046] Figure 5 Flowchart of transaction data delay compensation in the secondary cluster queue provided for the transaction data processing method embodiment two of the present application;

[0047] Figure 6A flowchart provided for the third embodiment of the transaction data processing method of the present application;

[0048] Figure 7 A flowchart provided for the third embodiment of the transaction data processing method of the present application;

[0049] Figure 8 A flowchart provided for the fourth embodiment of the transaction data processing method of the present application;

[0050] Figure 9 A flowchart provided for the fourth embodiment of the transaction data processing method of the present application;

[0051] Figure 10 A flowchart provided for the fourth embodiment of the transaction data processing method of the present application;

[0052] Figure 11 A device structure diagram of the hardware running environment involved in the transaction data processing method of the embodiments of the present application.

[0053] The object, function features and advantages of the present application will be further explained in combination with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0055] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below in combination with the drawings and the specific embodiments.

[0056] Distributed management also brings new challenges to operation management. When managing transaction accounts, it is necessary to timely understand the internal operating conditions in dozens or hundreds of clusters, including account transactions and balance conditions. The current conventional solution is to call another microservice to register transaction processing in the mode of online transaction. However, the online transaction calling another microservice to process will prolong the calling link and increase the risk of single point failure. Once a microservice fails, it may cause the entire transaction processing flow to be blocked, and even cause data loss or inconsistency problems, which greatly affects the integrity and accuracy of transaction data.

[0057] The main solution of the embodiment of the application is: after the online transaction of the distributed account is completed, the real-time transaction data of the online transaction is registered to a double-cluster message queue through a bypass asynchronous thread, the double-cluster message queue includes a master cluster queue and a slave cluster queue, the master cluster queue is used for persistence of first transaction data, and the slave cluster queue is used for delay compensation of the first transaction data, the first transaction data is real-time transaction data in the master cluster queue; based on an aggregation application instance, the first transaction data in the master cluster queue is consumed, and the first transaction data is stored to a master transaction table of a database; based on the aggregation application instance, second transaction data in the slave cluster queue is consumed, target transaction data missing in the first transaction data stored in the master transaction table is determined in the second transaction data, and the target transaction data is stored to the master transaction table of the database, the second transaction data is real-time transaction data in the slave cluster queue.

[0058] In the embodiment, for convenience of description, the following describes a processing device of transaction data as an execution subject.

[0059] The application provides a solution for storing real-time transaction data through a double-cluster message queue. Even if the master cluster queue fails, the real-time transaction data in the slave cluster queue can be used as a backup, reducing the risk of data loss and enhancing the fault tolerance of the system. By comparing the real-time transaction data in the master cluster queue and the slave cluster queue, target transaction data is determined and stored in the same database, thereby ensuring the eventual consistency of transaction data in the database, avoiding the problem of inconsistent data caused by network failure, system anomaly, etc., and ensuring the accuracy and integrity of real-time transaction data of the distributed account in the database.

[0060] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a processing device of transaction data, etc. The following describes the embodiment and each of the following embodiments by taking the processing device of transaction data as an example.

[0061] Based on this, the embodiment of the application provides a processing method of transaction data, which is described in detail with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the processing method of transaction data of the application is shown in the figure.

[0062] In the embodiment, the processing method of transaction data includes steps S10-S30:

[0063] Step S10, after the online transaction of the distributed account is completed, real-time transaction data of the online transaction is registered to a dual-cluster message queue through a bypass asynchronous thread, the dual-cluster message queue includes a primary cluster queue and a secondary cluster queue, the primary cluster queue is used for persistence of first transaction data, the secondary cluster queue is used for delayed compensation of the first transaction data, and the first transaction data is real-time transaction data in the primary cluster queue.

[0064] In the embodiment, there is no centralized agency to control and manage all account information in the distributed account system, but multiple nodes in the network jointly participate in the management and maintenance of the account. Each account has a corresponding public key and private key, the public key can be public, and the private key is kept by the user himself, which is used for digital signature of the transaction to ensure the security and non-repudiation of the transaction.

[0065] The distributed account refers to that account data is dispersed in multiple independent nodes or clusters in physical storage, and logically unified management is realized through distributed technology, wherein the online transaction time and transaction data of different accounts are not the same. In actual application, banks and other financial institutions can use the distributed account to simplify the traditional payment and settlement process, improve transaction efficiency and security, and reduce operational risk and cost.

[0066] Transaction data is a core information set that records a complete operation behavior in a business system, and its essence is a digital carrier of a business process. Optionally, the transaction data includes an order number, an amount, a channel, a commission, a settlement currency, etc.

[0067] In the embodiment, the online transaction of the distributed account is the main business link, and in addition, the operation management is executed asynchronously through an independent thread as a side business link, i.e., a bypass, so as to avoid blocking the main business link. The bypass is decoupled from the core business logic, realizes physical isolation, and is asynchronously executed in a non-blocking manner through an independent thread resource. The thread is a lightweight concurrent unit, rather than a process or a service, which realizes decoupling of the customer transaction process and the operation management, meets the management requirements, shortens the transaction link, and improves the system stability.

[0068] After the online transaction of the account is completed, real-time transaction data is registered to a message queue through a bypass asynchronous thread, so as to avoid that the online transaction directly waits for the data storage operation to be completed, greatly shorten the response time of the online transaction, and improve the processing efficiency of the online transaction and user experience.

[0069] Optionally, the online transaction is determined to be completed when the online transaction is completed in a TCC (Try-Confirm-Cancel) two-phase. After the online transaction is completed, the transaction data is forwarded to a cluster message queue, such as a double Kafka cluster, through a bypass asynchronous thread, and the double Kafka cluster and double sending mode are used to ensure that the transaction data is not lost and does not affect the transaction process. Kafka uses a publish and subscribe mode, which allows multiple producers to send messages to the message queue and also allows multiple consumers to read messages from the message queue. Kafka classifies messages in the form of topics, and producers publish messages to topics, and consumers subscribe to topics to consume messages.

[0070] Further, the double cluster message queue includes a main cluster A queue and a secondary cluster B queue, and the double cluster and double message queue greatly improve the reliability of transaction data and better cope with occasional network jitter and other short-term burst problems. In this application, the main cluster queue and the secondary cluster queue serve as message middleware, decoupling the direct dependency between online transaction processing and transaction data storage. This allows the various components of the system to be developed, deployed, and expanded relatively independently, improving the flexibility and maintainability of the system. For example, the data storage part can be optimized or upgraded without affecting the online transaction.

[0071] In an optional embodiment, referring to Figure 2 , step S10 includes: after the online transaction of the distributed account completes a preset phase, sending real-time transaction data of the online transaction to a cache instance; sending the real-time transaction data in the cache instance to an asynchronous to-be-sent cache queue; based on a background thread scanning the to-be-sent cache queue, registering the real-time transaction data in the to-be-sent cache queue to the double cluster message queue respectively.

[0072] Optionally, the preset phase can be a TCC two-phase. After the online transaction completes the preset phase, the real-time transaction data is first sent to the cache instance, and at this time the online transaction can continue without waiting for the completion of subsequent operations such as data storage, thereby reducing the waiting time of the online transaction, improving the response speed and processing efficiency of the transaction, and improving the user experience.

[0073] It should be noted that the cache instance can temporarily store real-time transaction data, so that even if there is a short-term system failure or network problem during subsequent message queue sending and the like, the data will not be lost, because the cache instance can serve as an intermediate data buffer, providing a layer of protection for the final storage of data. During a transaction peak, when a large amount of real-time transaction data is generated, the cache instance can first temporarily store these data, and then gradually send the data to the message queue through a background line, thereby smoothing the traffic peak of data processing, avoiding the congestion or performance degradation of the message queue due to the reception of a large amount of data in a short time, and ensuring the stability and orderliness of data processing.

[0074] In step S20, based on the aggregation application instance, the first transaction data in the master cluster queue is consumed, and the first transaction data is stored to the master transaction table of the database.

[0075] The aggregation application instance is a service node or process responsible for receiving, processing and integrating distributed transaction data. The aggregation application instance is a stateless computing layer between the message queue and the database. Multiple instances can be deployed in parallel to consume, and load balancing is achieved through the mechanism of Kafka partition allocation of aggregation application instances.

[0076] Optionally, the aggregation application instance can consume transaction data of a double Kafka cluster. The aggregation application instance subscribes to the message queues of two independent Kafka clusters. The transaction data consumed by the master cluster queue A is directly batch inserted into the database to provide real-time query service. The data consumed by the secondary cluster queue B is temporarily stored in the cache, and is delayed to the database, and is checked for duplication by the primary key to avoid data redundancy.

[0077] In an optional embodiment, referring to Figure 3 , step S20 includes: based on the aggregation application instance, real-time consumption of the first transaction data in the master cluster queue, and batch persistence of the first transaction data to the transaction master table of the database. Based on the aggregation application instance, the transaction data in the master cluster queue is batch stored to the transaction master table of the database, which can reduce the interaction times of the database, reduce the transaction overhead of the database, improve the write performance of the database, and reduce the load pressure of the database, compared with writing the transaction master table of the database one by one.

[0078] Optionally, based on the aggregation application instance, the first transaction data in the master cluster queue is consumed in real time, and when the data amount of the first transaction data is greater than a preset data amount threshold or after a preset time length, the first transaction data is batch persisted to the transaction master table of the database.

[0079] In the embodiment, the first transaction data consumed to the master cluster queue is directly inserted into the transaction master table, and a query service is directly provided. The real-time transaction data is centrally stored in the transaction master table, facilitating unified management, maintenance and backup, reducing data management complexity, and reducing the risk of management omissions or errors caused by data dispersion.

[0080] In step S30, based on the aggregation application instance, the second transaction data in the secondary cluster queue is consumed, target transaction data missing in the first transaction data stored in the master transaction table is determined in the second transaction data, and the target transaction data is stored to the master transaction table of the database, and the second transaction data is real-time transaction data in the secondary cluster queue.

[0081] In the embodiment, the data dispersion problem caused by the distributed system is solved by the way of online to asynchronous and double message queue aggregation processing, and the centralized management demand of business operation is met. By aggregating transaction data through double clusters, the master cluster directly falls into the database to provide services, and the secondary cluster provides compensation processing, reducing the probability of data loss and improving data reliability.

[0082] Optionally, in the second transaction data, the target transaction data missing in the first transaction data stored in the master transaction table can be determined by comparing the hash value of the second transaction data and the hash value of the first transaction data, determining the hash value existing in the second transaction data and not existing in the first transaction data, and taking the second transaction data corresponding to the hash value as the target transaction data.

[0083] Optionally, the aggregation application instance can intelligently deduplicate and delay compensation to ensure data integrity. The data of the secondary cluster queue B is processed with delay, and whether it exists in the master transaction table is checked through keywords. If it exists, it is directly discarded. If it does not exist, the transaction data is hashed and dispersed into multiple partition secondary tables, which maximizes the impact of kafka cluster jitter on the number of missing data, and the missing data is subsequently processed by multiple tasks in parallel. In the embodiment, the secondary cluster queue is used for delay compensation of the first transaction data, and the target transaction data in the master cluster queue is determined to ensure that all real-time transaction data can be finally stored in the database, further improving the reliability and integrity of the real-time transaction data stored in the database.

[0084] Optionally, the aggregation application instance dynamically schedules data processing tasks to realize quasi-real-time aggregation. Dynamic task scheduling and concurrent processing, according to the distribution characteristics of account transaction volume, divide the processing task into multiple independent sub-tasks. Through the online to asynchronous mechanism, the data processing task is submitted, and the horizontal expansion ability is improved. The task performance data such as processing time is monitored, and the task is split or merged in real time to ensure timeliness.

[0085] Optionally, the aggregated application instance is subjected to end-of-day reconciliation, further ensuring business accuracy. At the end of the day, transaction data and account balances are synchronized from each sub-database, the data of the day is reconciled, and possible losses or errors are repaired. Through end-of-day batch task processing, real-time transaction data and account balances of the previous day are batched from each cluster, and after processing, they are directly treated as the balance of the previous day and the transaction of the previous day, compensating for possible errors of the previous day. Through end-of-day batch processing, the transaction data of each sub-database is synchronized again, the transaction and account summary data is compensated, the possible errors of the previous day are compensated, and the accuracy of the business transaction data is further ensured.

[0086] In the technical solution in this embodiment, the real-time transaction data is stored through the dual-cluster message queue. Even if the main cluster queue fails, the real-time transaction data in the secondary cluster queue can serve as a backup, reducing the risk of data loss and enhancing the fault tolerance of the system. By comparing the real-time transaction data in the main cluster queue and the secondary cluster queue, the target transaction data is determined and stored in the same database, thereby ensuring the eventual consistency of the transaction data in the database and avoiding the problem of inconsistent data caused by network failures, system abnormalities, etc., ensuring the accuracy and integrity of the real-time transaction data of the distributed account in the database.

[0087] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiments can be referred to the above introduction, and will not be described in detail. On this basis, please refer to Figure 4 , step S30 comprises:

[0088] Step S31, according to the keyword of the second transaction data, query whether the second transaction data exists in the transaction master table;

[0089] Step S32, determining the second transaction data that does not exist in the transaction master table as the target transaction data;

[0090] Step S33, storing the target transaction data into at least two partitions of the transaction sub-table in the database, the transaction sub-table being a partitioned table;

[0091] Step S34, after a preset time, consuming the target transaction data in the transaction sub-table based on the aggregated application instance;

[0092] Step S35, storing the target transaction data in the transaction sub-table to the transaction master table.

[0093] In this embodiment, the transaction data in the secondary cluster queue is consumed based on the aggregated application instance, and batch processing is performed after a preset time, reducing the frequent write operation to the database, reducing the load of the database, and improving the overall performance and throughput of the system.

[0094] The transaction sub-table adopts a partition table design, which can conveniently increase or decrease the number of partitions according to the growth of data volume and the needs of business development, realize horizontal expansion of the system, and improve the storage capacity and processing performance of the system.

[0095] The preset time length can be flexibly adjusted according to actual business needs and system running conditions to adapt to different transaction volumes and data processing requirements. For example, the preset time length can be appropriately shortened during the transaction peak period to timely find and process missing data, and the preset time length can be appropriately extended during the low peak period to reduce the occupation of system resources.

[0096] Optionally, the second transaction data consumed to the sub-cluster queue is stored in the database after a delay of the preset time length, and it is first checked whether the second transaction data exists in the transaction main table to reduce the amount of transaction data stored in the table. For target transaction data that does not exist, the target transaction data is stored in the multiple partition tables in the transaction sub-table.

[0097] Optionally, before storing the target transaction data in the transaction sub-table into the transaction main table, the target transaction data is queried again to determine whether it exists in the transaction main table. If it does not exist, the target transaction data in the transaction sub-table is stored in the transaction main table. If it exists, the target transaction data is deleted.

[0098] Optionally, after storing the target transaction data in at least two partitions of the transaction sub-table in the database, a background compensation task is processed by multiple tasks according to the partitions, compared with the current database time, and for the storage time exceeding the preset time length, it is checked according to the key whether the transaction main table exists. If it does not exist, the target transaction data is registered in the transaction main table.

[0099] Optionally, according to Figure 5 , the transaction flow of the target transaction data is used as the key, and the target transaction data is queried according to the key to determine whether it exists in the transaction main table. If it does not exist, the transaction flow of the target transaction data is hashed, and the hash value is stored in the partition of the transaction sub-table in the database to prepare for subsequent multi-task concurrent processing. Hashing the target transaction data and storing the hash value in the sub-table can verify the integrity and accuracy of the data by comparing the hash values when the missing target transaction data is stored in the main table, preventing data corruption or tampering during data transmission or storage, and further improving the consistency of the data.

[0100] After determining that the target transaction data does not exist in the transaction main table, the target transaction data is hashed, and the hash value is stored in at least two partitions of the transaction sub-table, which is equivalent to creating multiple copies of the data, increasing the redundancy of data storage, and when a partition fails or data is lost, the data can be recovered and compensated through other partitions, improving the fault tolerance and fault tolerance of the system.

[0101] In this embodiment, by querying the transaction master table based on keywords in real-time transaction data after a preset period of time, it is possible to accurately determine which real-time transaction data was missed during processing in the main cluster queue, effectively avoiding the risk of data loss and ensuring data integrity. Ultimately, the target transaction data is stored in the transaction master table, ensuring that the data in the transaction master table reflects all actual transactions, including those delayed or omitted for various reasons, thereby ensuring the accuracy and reliability of the real-time transaction data in the database.

[0102] Based on any of the above embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction and will not be described in detail later. Figure 6 , after step S33, further comprising:

[0103] Step S36, determining at least two data processing tasks according to the number of partitions corresponding to the target transaction data in the transaction sub-table and the number of the aggregate application instances;

[0104] Step S37: execute at least two of the data processing tasks.

[0105] In this embodiment, the data processing tasks are independent of each other. The data processing tasks are used to execute the target transaction data in the transaction sub-table based on the aggregated application instance consumption transaction after a preset time; and store the target transaction data in the transaction sub-table in the transaction main table. Figure 7 The process of storing the target transaction data in the transaction secondary table into the transaction primary table in the aggregation application instance.

[0106] By rationally planning the number of partitions and aggregate application instances, the number of data processing tasks can be adjusted to achieve efficient processing. Dividing the target transaction data in the transaction sub-table partitions into multiple data processing tasks and executing these data processing tasks in parallel based on aggregate application instances can fully utilize system resources, significantly shorten processing time, and improve overall efficiency.

[0107] When the amount of transaction data is large, if single-threaded or sequential processing is used, bottlenecks are likely to form, leading to processing delays. Multi-task parallel processing can effectively cope with the growth of data volume, ensure timely completion of data processing, and enhance the stability and reliability of the system under high load.

[0108] When business needs change or the amount of transaction data increases, the task division strategy and processing logic can be flexibly adjusted, such as increasing the number of data processing tasks, optimizing the partitioning scheme, etc., making the system more adaptable to business development and improving scalability.

[0109] In the technical solution of the embodiment, according to the number of partitions of the target transaction data in the transaction sub-table and the number of aggregation application instances, at least two independent data processing tasks are determined, and the tasks are executed in parallel, so that system resources can be fully utilized, the data processing speed is significantly improved, and the time for migrating data from the transaction sub-table to the transaction main table is shortened. The data processing tasks are independent of each other, can avoid mutual interference, ensure that each task can efficiently utilize system resources, improve the overall resource utilization rate, and enhance the stability and reliability of the system under high load.

[0110] Based on any of the above embodiments of the present application, in the fourth embodiment of the present application, the same or similar contents as the above embodiments can be referred to the above introduction, and will not be repeated hereinafter. On this basis, please refer to Figure 8 , after step S37, further comprising:

[0111] Step S38, according to the business type of the data processing task, querying the task state of the data processing task in the task list;

[0112] Step S39, according to the task state of the data processing task, executing the associated processing operation.

[0113] In the embodiment, the task list is generated based on a preset rule, and the preset rule includes account transaction volume. Based on the preset rule, the task list is generated, and the task priority and processing method can be flexibly adjusted according to the business demand and transaction volume change. For example, during the business peak period, the business type task with large transaction volume can be preferentially processed, and during the low peak period, all kinds of tasks are processed evenly, so that the system can better adapt to the business fluctuation.

[0114] The data processing tasks are independently processed and complementarily interfered. Through the transaction volume level performance data of the task processing, the number of tasks can be adjusted in real time, for example, new tasks are added or the data processing tasks are combined, so that the data processing tasks can be completed in a reasonable time, and more real-time and reliable summary data can be provided for the business.

[0115] Different business types may have different processing requirements and processes, and corresponding processing operations can be performed according to the business type to meet diversified needs, so that the system is more flexible and can quickly respond to business changes.

[0116] By querying the task state in the task list, abnormal or inconsistent conditions that may occur during task execution, such as task repetition, omission, etc., can be found in time, and corresponding processing can be performed, so that the task state is consistent with the actual business, and the data accuracy is improved.

[0117] In an optional embodiment, refer to Figure 9When the task state is pending, the task state is modified to be in execution, and the online processing is converted to asynchronous processing; when the task state is in execution, according to the processed time length and the preset timeout time length, it is determined whether the data processing task is timed out, and the task state of the timed-out data processing task is updated to be in execution exception; when the task state is execution success, the task state is reset to be pending; when the task state is execution exception, the submission retry processing is performed.

[0118] The task processing is dynamically monitored, the task can be split or merged according to the business scenario, and the flexibility is high. By analyzing the task state and the processing operation, the performance and efficiency of the system can be evaluated, and the basis for optimizing the system is provided. If it is found that some business type tasks often appear delay or failure, targeted optimization and improvement can be performed.

[0119] Optionally, referring to Figure 10 , the target transaction data is processed according to the time slice, if the processing is successful, the task state is updated to be execution success, and if the processing fails, the task state is updated to be execution failure.

[0120] In the technical scheme of the embodiment, the task state is queried according to the business type of the data processing task, the current progress of each task can be accurately understood, for example, for the business type task with large transaction volume, more resources can be allocated for processing in time to ensure that important tasks are completed quickly and the overall efficiency is improved.

[0121] It should be noted that the above examples are only used for understanding the present application and do not constitute a limitation on the processing method of the transaction data of the present application, and more forms of simple transformation based on the technical concept are within the protection scope of the present application.

[0122] The present application provides a transaction data processing device, which comprises at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the transaction data processing method in the above-mentioned embodiment one.

[0123] Reference will now be made to Figure 11 , which shows a structural schematic diagram of a transaction data processing device suitable for implementing the embodiments of the present application. The transaction data processing device in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDA, Personal Digital Assistant), tablet computers (PAD, Portable Application Description), and the like, and fixed terminals such as computers and the like. Figure 11The processing device of transaction data shown is merely an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0124] As shown in Figure 11 The processing device of transaction data can include a processing apparatus 1001 (e.g. a central processor, a graphic processor, etc.) which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the processing device of transaction data are also stored. The processing apparatus 1001, the ROM 1002 and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the processing device of transaction data to communicate with other devices wirelessly or by wire to exchange data. Although the processing device of transaction data with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or less systems can be alternatively implemented or provided.

[0125] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0126] The transaction data processing device provided by the present application adopts the transaction data processing method in the above embodiments, and can solve the technical problem of low completeness of transaction data. Compared with the prior art, the transaction data processing device provided by the present application has the same beneficial effects as the transaction data processing method provided by the above embodiments, and other technical features in the transaction data processing device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0127] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0128] The above is merely specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0129] The present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the transaction data processing method in the above embodiments.

[0130] The computer readable storage medium provided by the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to: electrical wires, optical cables, radio frequency (RF), etc., or any suitable combination of the above.

[0131] The computer readable storage medium can be included in the transaction data processing device, or can exist separately and not be assembled into the transaction data processing device.

[0132] The computer readable storage medium carries one or more programs, and when the one or more programs are executed by the transaction data processing device, the transaction data processing device is caused to: store real-time transaction data through a dual-cluster message queue, so that even if the primary cluster queue fails, the real-time transaction data in the secondary cluster queue can be backed up, reducing the risk of data loss and enhancing the fault tolerance of the system. By comparing the real-time transaction data in the primary cluster queue and the secondary cluster queue, the target transaction data is determined and stored in the same database, thereby ensuring the eventual consistency of the transaction data in the database, avoiding the problem of inconsistent data caused by network failure, system anomaly, etc., and ensuring the accuracy and integrity of the real-time transaction data of the distributed account in the database.

[0133] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages such as C or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0134] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0135] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the names of the modules do not limit the modules themselves.

[0136] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the above-mentioned transaction data processing method, and can solve the technical problem of low completeness of transaction data. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the transaction data processing method provided by the above-mentioned embodiments, which will not be repeated here.

[0137] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the transaction data processing method as described above.

[0138] The computer program product provided by the present application can solve the technical problem of low completeness of transaction data. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the transaction data processing method provided by the above-mentioned embodiments, which will not be repeated here.

[0139] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for processing transaction data, characterized in that: The transaction data processing method includes: After the online transaction of the distributed account is completed, the real-time transaction data of the online transaction is registered to the dual-cluster message queue through a bypass asynchronous thread. The dual-cluster message queue includes a primary cluster queue and a secondary cluster queue. The primary cluster queue is used for persistence of first transaction data, and the secondary cluster queue is used for delay compensation of the first transaction data. The first transaction data is the real-time transaction data in the primary cluster queue. Based on the aggregate application instance, consume the first transaction data in the main cluster queue and store the first transaction data in a main transaction table of a database; Based on the aggregate application instance, second transaction data in the secondary cluster queue is consumed, target transaction data that is missing from the first transaction data stored in the primary transaction table is determined in the second transaction data, and the target transaction data is stored in the primary transaction table of the database, where the second transaction data is real-time transaction data in the secondary cluster queue.

2. The method for processing transaction data according to claim 1, wherein: The step of determining, in the second transaction data, the target transaction data in the first transaction data stored in the main transaction table comprises: querying, based on a keyword of the second transaction data, whether the second transaction data exists in the transaction master table; determining that the second transaction data not existing in the transaction master table is target transaction data; Storing the target transaction data in at least two partitions of a transaction sub-table of the database, where the transaction sub-table is a partitioned table; The step of storing the target transaction data in the main transaction table of the database includes: After a preset time period, based on the aggregated application instance, the target transaction data in the transaction sub-table is consumed; The target transaction data in the transaction sub-table is stored in the transaction main table.

3. The method for processing transaction data according to claim 2, wherein: The method comprises: Determine at least two data processing tasks according to the number of partitions corresponding to the target transaction data in the transaction sub-table and the number of the aggregation application instances; executing at least two of the data processing tasks; Among them, the data processing tasks are independent of each other, and the data processing tasks are used to consume the target transaction data in the transaction sub-table based on the aggregated application instance after a preset period of time; and store the target transaction data in the transaction sub-table into the transaction main table.

4. The method for processing transaction data according to claim 3, wherein: After the step of executing at least two of the data processing tasks, the method further includes: According to the business type of the data processing task, query the task status of the data processing task in the task list; executing an associated processing operation according to the task status of the data processing task; The task list is generated based on preset rules, and the preset rules include account transaction volume.

5. The method for processing transaction data according to claim 4, wherein: The step of executing the associated processing operation according to the task status of the data processing task includes: When the task status is pending, the task status is changed to being executed, and the online processing is converted to asynchronous processing; When the task status is in execution, determining whether the data processing task has timed out based on the processing duration and the preset timeout duration, and updating the task status of the timed-out data processing task to execution exception; When the task status is successfully executed, reset the task status to a pending status; When the task status is execution exception, a submission retry process is performed.

6. The method for processing transaction data according to claim 1, wherein: After the online transaction of the distributed account is completed, the step of registering the real-time transaction data of the online transaction to the dual-cluster message queue through the bypass asynchronous thread includes: After the online transaction of the distributed account completes a preset stage, sending the real-time transaction data of the online transaction to the cache instance; Sending the real-time transaction data in the cache instance to an asynchronous cache queue to be sent; The to-be-sent buffer queue is scanned based on a background thread, and the real-time transaction data in the to-be-sent buffer queue is registered in the dual-cluster message queues respectively.

7. The method for processing transaction data according to any one of claims 1 to 6, characterized in that: The step of consuming the first transaction data in the main cluster queue based on the aggregated application instance and storing the first transaction data in the main transaction table of the database includes: Based on the aggregated application instance, the first transaction data in the main cluster queue is consumed in real time, and the first transaction data is persisted in batches to the transaction master table of the database.

8. A transaction data processing device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the transaction data processing method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the transaction data processing method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the transaction data processing method according to any one of claims 1 to 7 are implemented.

Images