Method and apparatus for processing batch transaction data

Abstract

The disclosure provides a batch transaction data processing method, which can be applied to the technical field of cloud computing. The method is applied to a distributed system and includes: in response to a user's batch transaction remittance request, obtaining batch transaction data; performing first sharding processing on the batch transaction data according to characteristic identification information of the batch transaction data, to determine first sharding data and a target database server corresponding to the first sharding data; saving the first sharding data to the target database server corresponding to the first sharding data; performing second sharding processing on the first sharding data according to a preset splitting rule, to determine second sharding data and a target partition in a message middleware corresponding to the second sharding data; and sending the second sharding data to the target partition. The disclosure also provides a batch transaction data processing device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the field of cloud computing technology, specifically to the field of distributed database technology, and more specifically to a method, apparatus, device, storage medium, and program product for processing batch transaction data. Background Technology

[0002] Most existing large-scale financial industry systems are centralized accounting systems, with transaction processing typically using IBM mainframes. All stages of transaction processing are performed within the same mainframe. However, the cost of purchasing, maintaining, and upgrading mainframe hardware is high, and the efficiency of processing large volumes of accounting data on mainframes is heavily dependent on the performance of the IBM mainframe, which is generally achieved through centralized, single-threaded processing.

[0003] In existing technologies, transaction processing, which is originally handled within a single large computer system, is broken down into multiple relatively independent processes performed by multiple microcomputers—a process known as distributed ledger processing. However, when processing large volumes of remittance data, there are often related transactions. This means that after one transaction is completed, account information, such as balance sheets, needs to be processed before proceeding to the next transaction to prevent accounting errors. However, in distributed ledger processing, transactions for the same account may be processed by multiple computers. To prevent duplicate processing of accounting data, it is usually necessary to wait for the account transaction data processing results before proceeding to the next transaction. This introduces processing delays, affecting the efficiency of accounting data processing and consequently impacting user experience.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a method, apparatus, equipment, medium and program product for processing batch transaction data to improve the efficiency of batch transaction data processing.

[0006] According to a first aspect of this disclosure, a method for processing batch transaction data is provided, applied to a distributed system, the distributed system including multiple database servers, each of which deploys multiple container nodes, the processing method comprising:

[0007] In response to a user's bulk transaction remittance request, obtain bulk transaction data;

[0008] The batch transaction data is first sharded based on the feature identification information of the batch transaction data to determine the first shard data and the target database server corresponding to the first shard data, wherein the feature identification information includes account identification information and / or transaction region identification information;

[0009] Save the first shard data to the target database server corresponding to the first shard data;

[0010] The first shard data is further sharded according to a preset splitting rule to determine the second shard data and the target partition in the corresponding message middleware; and

[0011] The second shard data is sent to the target partition.

[0012] According to embodiments of this disclosure, the step of performing a first sharding process on the batch transaction data based on the feature identification information of the batch transaction data to determine the first shard data and the target database server corresponding to the first shard data includes:

[0013] Obtain the account identifier information and transaction region identifier information of the batch transaction data;

[0014] The batch transaction data is routed and sharded according to the account identification information and / or the transaction region identification information to obtain multiple first shard data and multiple target database servers corresponding to the multiple first shard data.

[0015] According to embodiments of this disclosure, the second fragmentation process of the first fragmented data according to a preset fragmentation rule includes:

[0016] The first shard data is then processed into a second shard based on the account identification information of the first shard data.

[0017] According to embodiments of this disclosure, the second sharding process of the first shard data based on the account identification information of the first shard data includes:

[0018] Using the same account identifier information of the first shard data as the target partition key value, a routing algorithm is used to perform consistent hash shard calculation to determine the target partition corresponding to the second shard data.

[0019] According to embodiments of this disclosure, before performing the first sharding process on the batch transaction data, the method further includes:

[0020] The batch transaction data is then validated.

[0021] According to embodiments of this disclosure, messages from the target partition are consumed by at least one container node of the target database server.

[0022] A second aspect of this disclosure provides a batch transaction data processing apparatus for a distributed system, the distributed system including multiple database servers, each of which deploys multiple container nodes, the apparatus comprising:

[0023] The acquisition module is used to retrieve batch transaction data in response to users' batch transaction remittance requests;

[0024] The first data sharding module is used to perform a first sharding process on the batch transaction data according to the feature identification information of the batch transaction data, so as to determine the first shard data and the target database server corresponding to the first shard data, wherein the feature identification information includes account identification information and / or transaction region identification information;

[0025] The first sending module is used to save the first fragment data to the target database server corresponding to the first fragment data;

[0026] The second data sharding module is used to perform a second sharding process on the first sharded data according to a preset splitting rule, so as to determine the second sharded data and the target partition in the message middleware corresponding to the second sharded data; and

[0027] The second sending module is used to send the second fragment data to the target partition.

[0028] According to embodiments of this disclosure, the first data sharding module includes:

[0029] The acquisition submodule is used to acquire the account identification information and transaction region identification information of the batch transaction data.

[0030] The first sharding submodule is used to route and shard the batch transaction data according to the account identification information and / or the transaction region identification information, so as to obtain multiple first shard data and multiple target database servers corresponding to the multiple first shard data.

[0031] According to embodiments of this disclosure, the second data sharding module includes:

[0032] The second sharding submodule is used to perform a second sharding process on the first sharding data based on the account identification information of the first sharding data.

[0033] According to embodiments of this disclosure, the second segmented submodule includes:

[0034] The sharding unit is used to use the same account identification information of the first shard data as the target partition key value, and to perform consistent hash sharding calculation using a routing algorithm to determine the target partition corresponding to the second shard data.

[0035] According to embodiments of this disclosure, it further includes:

[0036] The data verification module is used to verify the batch transaction data.

[0037] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the aforementioned batch transaction data processing method.

[0038] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the aforementioned method for processing bulk transaction data.

[0039] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for processing bulk transaction data.

[0040] The batch transaction data processing method provided by the embodiments of this disclosure involves first sharding the batch transaction data based on its feature identification information to obtain a first shard, determining the distributed database of the first shard, and then performing a second sharding process on the first shard according to a preset splitting rule. Finally, the batch transaction data of the same account is sent to the target partition of the message middleware. Compared to related technologies, the batch transaction data processing method provided by the embodiments of this disclosure uses a dual sharding system to shard batch data of the same account to the same partition, ensuring that the data in each partition does not affect each other while processing accounts with high concurrency. Through multi-threaded account processing, processing efficiency is improved and the time for large-scale batch processing is shortened. Attached Figure Description

[0041] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0042] Figure 1a This diagram illustrates a schematic representation of a distributed accounting system architecture in the prior art.

[0043] Figure 1b A schematic diagram illustrating a distributed accounting system architecture provided according to embodiments of the present disclosure is shown.

[0044] Figure 2 The illustration schematically depicts an application scenario of a method, apparatus, device, medium, and program product for processing bulk transaction data according to embodiments of the present disclosure.

[0045] Figure 3 A flowchart illustrating a method for processing batch transaction data according to an embodiment of the present disclosure is shown schematically.

[0046] Figure 4 A schematic diagram of a dual-slicing system provided according to an embodiment of the present disclosure is shown.

[0047] Figure 5a A flowchart illustrating a first fragmentation processing method provided according to an embodiment of the present disclosure is shown schematically.

[0048] Figure 5b The illustration shows a schematic diagram of the first fragmentation process in a dual fragmentation system provided according to an embodiment of the present disclosure;

[0049] Figure 6a A flowchart illustrating a second fragmentation processing method provided according to an embodiment of the present disclosure is shown schematically.

[0050] Figure 6b The diagram illustrates the second fragmentation process of a dual fragmentation system provided according to an embodiment of the present disclosure;

[0051] Figure 7 This schematically illustrates a structural block diagram of a batch transaction data processing apparatus according to an embodiment of the present disclosure; and

[0052] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a batch transaction data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0053] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0054] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0055] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0056] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0057] First, the terminology appearing in the embodiments of this disclosure will be explained:

[0058] Distributed accounting: Transactions that were originally processed on a single large computer system are broken down into multiple relatively independent processes, which are then completed by multiple microcomputers.

[0059] Kafka: A distributed streaming platform, similar to a message queue, acting as a channel between message producers and message consumers; essentially, it is a queue.

[0060] Container: In this article, Docker containers refer to instances that run on an image. In addition to the file resources contained in the image, they also contain running application processes.

[0061] To address the aforementioned technical issues, embodiments of this disclosure provide a bandwidth control method applied to a distributed system. The distributed system includes multiple database servers, each deploying multiple container nodes. The processing method includes: responding to a user's batch transaction remittance request by acquiring batch transaction data; performing a first sharding process on the batch transaction data based on feature identification information to determine a first shard and its corresponding target database server, wherein the feature identification information includes account identification information and / or transaction region identification information; saving the first shard to the target database server corresponding to the first shard; performing a second sharding process on the first shard according to a preset splitting rule to determine a second shard and its corresponding target partition in a message middleware; and sending the second shard to the target partition.

[0062] First, combine Figure 1a and Figure 1bThis disclosure introduces the distributed accounting system architecture provided in the embodiments. Figure 1a This diagram illustrates a schematic representation of a distributed accounting system architecture in the prior art. Figure 1b A schematic diagram of a distributed accounting system architecture provided according to an embodiment of the present disclosure is shown.

[0063] like Figure 1a As shown, in a distributed framework, multiple small computers can be deployed according to different application business needs. The containers (Pods) deployed and running on each small computer using Docker+Kubernetes technology can also be adjusted according to specific business requirements. Figure 1a Taking a multi-container processing mechanism with two Pods deployed on each small computer as an example, when a transaction comes in, both containers start normally. The transaction is load balanced to one container, while the other container remains idle. In the existing technology, the utilization rate of container resources is low, the efficiency of batch data processing is low, and there is a problem that batch data of the same account is processed by multiple containers at the same time. In order to prevent the same transaction from being processed multiple times, a waiting mechanism must be set up to wait for the processing result information before the next transaction can be processed, which may even cause the transaction timeout problem, thus resulting in low efficiency of batch data processing.

[0064] In the embodiments disclosed herein, such as Figure 1b As shown, the distributed accounting system framework employs a dual-sharding system. Upon acquiring all batches of remittance data, the system, following the distributed framework, performs routing and sharding based on customer-specific information such as customer or region, completing the first data sharding of the "dual-sharding system." A second sharding is then performed on each corresponding minicomputer. Subsequent container pods then process the corresponding partitioned data. Under normal conditions, maximum scheduling is achieved to ensure simultaneous processing on each pod node. An online small-batch Kafka intermediate message consumption job is deployed on each container node, enabling each container to have an independent process that consumes message queue pushes from Kafka in near real-time. Simultaneously, batch remittance data is processed in real-time, with one transaction completed before the next, achieving uninterrupted high-speed accounting processing. Each pod processes accounting data in an independent thread, without interfering with each other.

[0065] Figure 2 The illustration schematically depicts an application scenario of a method, apparatus, device, medium, and program product for processing bulk transaction data according to embodiments of the present disclosure.

[0066] like Figure 2As shown, application scenario 100 according to this embodiment may include a batch transaction data processing scenario. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0067] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0068] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0069] Server 105 can be a batch transaction data processing server. In response to batch transaction data processing requests initiated by users through the application server, such as transaction requests issued by users using terminal devices 101, 102, and 103, the batch transaction data processing server performs two data fragments on the received batch transaction data according to the batch transaction data processing method provided in this embodiment of the disclosure, and then sends the batch transaction data to the corresponding partition of the message middleware, which is then consumed by the container application node to process the corresponding accounting data.

[0070] It should be noted that the batch transaction data processing method provided in this embodiment can generally be executed by server 105. Correspondingly, the batch transaction data processing apparatus provided in this embodiment can generally be located in server 105. The batch transaction data processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the batch transaction data processing apparatus provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0071] It should be understood that Figure 2 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0072] It should be noted that the batch transaction data processing method and apparatus determined in the embodiments of this disclosure can be used in the field of cloud computing technology, the field of financial technology, and any field other than the financial field. The application field of the batch transaction data processing method and apparatus determined in the embodiments of this disclosure is not limited.

[0073] The following will be based on Figure 2 The described scene, through Figures 3 to 6b The method for processing batch transaction data according to embodiments of this disclosure will be described in detail.

[0074] Figure 3 A flowchart illustrating a method for processing batch transaction data according to an embodiment of this disclosure is shown schematically. Figure 4 A schematic diagram of a dual-slicing system provided according to an embodiment of the present disclosure is shown.

[0075] like Figure 3 As shown, the batch transaction data processing method of this embodiment includes operations S210 to S250, which can be executed by a server or other computing device. This batch transaction data processing method of this embodiment is applied to a distributed system. The distributed system includes multiple database servers, and each database server deploys multiple container nodes.

[0076] In operation S210, in response to the user's batch transaction remittance request, batch transaction data is obtained.

[0077] In one example, the batch remittance transaction in this embodiment of the disclosure involves a user submitting a batch of remittances from an online banking or mobile transaction interface, simultaneously uploading the details to the bank's system. Ultimately, both the batch remittance batches and transaction details are submitted to the underlying accounting processing center, where the application completes all batch remittance transactions. The application in the accounting processing center needs to assist customers in completing batch remittances as efficiently as possible. The batch transaction data processing method in this embodiment of the disclosure is actually executed by the server of the accounting processing center. Upon receiving a user's batch remittance request, the server retrieves the batch transaction data submitted by the user from the front end.

[0078] The system can be expanded to perform verification operations on the batch transaction data. Before proceeding to the next processing step, the batch transaction data needs to be verified. Batch transaction data includes batch data for batch remittances and various batch remittance transaction details, including transaction information such as amount, account name, and card number. Through pre-verification of online banking batch remittances, such as total amount verification, deduction account checks, and risk control checks, some batches that do not conform to the batch remittance criteria are directly rejected. All data that passes the pre-verification is then entered into the first data sharding of the "dual sharding system."

[0079] In operation S220, the batch transaction data is first sharded according to the feature identification information of the batch transaction data to determine the first shard data and the target database server corresponding to the first shard data.

[0080] In operation S230, the first shard data is saved to the target database server corresponding to the first shard data.

[0081] According to embodiments of this disclosure, the feature identification information includes account identification information and / or transaction region identification information.

[0082] In one example, the verified batch transaction data is first sharded based on the feature identification information of the batch transaction data. The batch transaction data is routed and sharded according to the dimension of account or transaction region to generate multiple sets of first shard data. These multiple sets of first shard data are then saved to the corresponding database, thus determining the minicomputer (target database server) corresponding to the first shard data. The routing algorithm in this embodiment can be any mature algorithm in the related technology, which will not be described in detail here.

[0083] In operation S240, the first shard data is split a second time according to the preset splitting rules to determine the second shard data and the target partition in the message middleware corresponding to the second shard data.

[0084] In operation S250, the second fragment data is sent to the target partition.

[0085] According to embodiments of this disclosure, messages from the target partition are consumed by at least one container node of the target database server.

[0086] In one example, such as Figure 4 As shown, after the first data sharding process, to prevent the use of out-of-order, machine-gun-like invocation methods in related technologies, which could lead to multiple executions of account transactions, a second sharding process is required according to a preset splitting rule. This generates multiple sets of second-shard data and determines the target partition corresponding to each set of second-shard data. This target partition is a partition of the message middleware, such as multiple target partitions under a Topic in Kafka. The second-shard data is then sent to the corresponding target partition. Messages in the target partition are consumed by container nodes deployed on the target database server. In this embodiment, the target database server is deployed and running two container nodes based on Docker+Kubernetes technology. The number of container nodes can be determined based on the specific business processing requirements, or multiple container nodes can be deployed; the number of container nodes is not specifically limited. The specific process of operation S240 can be found in [link to documentation]. Figure 6a The operation S241 shown will not be described again here.

[0087] The batch transaction data processing method provided by the embodiments of this disclosure involves first sharding the batch transaction data based on its feature identification information to obtain a first shard, determining the distributed database for the first shard, and then performing a second sharding process on the first shard according to a preset splitting rule. The batch transaction data for the same account is then sent to the target partition of the message middleware. This dual sharding system shards the batch data of the same account to the same partition, ensuring that the data in each partition does not interfere with each other while processing accounts with high concurrency. Through multi-threaded account processing, processing efficiency is improved and the time for large-scale batch processing is shortened.

[0088] Figure 5a A flowchart illustrating a first fragmentation processing method provided according to an embodiment of the present disclosure is shown schematically. Figure 5b This diagram schematically illustrates the first fragmentation process in a dual fragmentation system provided according to an embodiment of the present disclosure. For example... Figure 5a As shown, operation S220 includes operation S221 and operation S222.

[0089] In operation S221, the account identification information and transaction region identification information of the batch transaction data are obtained.

[0090] In operation S222, the batch transaction data is routed and sharded according to the account identification information and / or the transaction region identification information to obtain multiple first shard data and multiple target database servers corresponding to the multiple first shard data.

[0091] In one example, such as Figure 5b As shown, all data that passes the pre-check enters the first sharding of the "dual sharding system". Specifically, the characteristic identification information of the batch transaction data is first obtained, including account identification information and transaction region identification information, such as account number, user number, region code and other identification information. The data is then sharded and split into corresponding minicomputers 1, 2, ... n by account dimension or transaction region dimension, so that batch data of the same region or batch data of the same account can be split into the same target database server. It can be extended, and the batch transaction data can also be divided into multiple first shards based on both account and transaction region dimensions, and multiple first shards are saved to the corresponding target database server.

[0092] Figure 6a A flowchart illustrating a second fragmentation processing method provided according to an embodiment of the present disclosure is shown schematically. Figure 6b This diagram schematically illustrates the second fragmentation process of a dual fragmentation system provided according to an embodiment of the present disclosure. For example... Figure 6aAs shown, operation S240 includes operation S241.

[0093] In operation S241, the first fragment data is processed into a second fragment based on the account identification information of the first fragment data.

[0094] According to an embodiment of this disclosure, the same account identification information of the first shard data is used as the target partition key value, and a routing algorithm is used to perform consistent hash shard calculation to determine the target partition corresponding to the second shard data.

[0095] In one example, the first shard of data on each small computer is further sharded using MurmurHash routing. The first shard is then sharded a second time according to a preset splitting rule in a "dual sharding system." Specifically, the preset splitting rule in this embodiment could be based on the deduction account as the partitioning principle, splitting batch transaction data for the same deduction account into the same partition. Using the deduction account as the partitioning principle is because, generally for batch remittances, one debit corresponds to multiple credits. If a randomized, machine-gun-like call method is used, it can easily lead to a debit account transaction simultaneously deducting money from multiple credit accounts. Furthermore, each remittance transaction itself needs to wait for the completion of the previous debit transaction before proceeding to the next. If the waiting time during the transaction is too long, it will cause the transaction to time out and fail. Bank accounts require the account balance to be processed after each transaction is completed before continuing with the next transaction to prevent accounting errors. This partitioning model allows remittance data for the same deduction account in a transaction to be arranged in a thread-ordered queue within the same partition. The data is then integrated and sent to a Kafka intermediate message queue for consumption by container nodes.

[0096] like Figure 6bAs shown, using the same deduction account as the Kafka partition key, consistent hashing is performed using the murmur2 algorithm for sharding calculation. The result is then modulo-rounded by the number of partitions under the topic to select the corresponding partition. Data is pushed to intermediate messages in Kafka format, placed in partition-1 (partition 1) and partition-2 (partition 2) of the Kafka intermediate messages. This achieves the goal of placing the same deduction account under the same partition, avoiding transaction timeouts caused by multiple threads initiating deductions for the same account simultaneously. Meanwhile, the container Pods deployed on various small computers are continuously running background online batch processing to consume Kafka messages. Each Kafka message consumed by the online batch processing is recorded in the batch remittance transaction details table, while the remittance transaction logic is processed, and the remittance transaction results are recorded in the batch remittance transaction details table. The batch remittance transaction details table can be visualized and displayed to the client. The number of processing threads is determined by the number of container Pod nodes. In this embodiment, two container Pods are deployed under a small computer. The corresponding threads for processing the small computer are Pod1 and Pod2. The processing of Pod1 and Pod2 under the small computer 1 is independent of each other, ensuring concurrency without being affected by the accounting process.

[0097] This distributed, large-scale accounting processing method enables multi-threaded processing, ensuring that transactions for the same account do not interfere with each other. This achieves both high processing efficiency and guarantees correct accounting, while also preventing interference between nodes. This method is particularly suitable for multi-computer operation within a distributed framework.

[0098] Based on the above-described method for processing batch transaction data, this disclosure also provides a device for processing batch transaction data. The following will be combined with... Figure 7 The device is described in detail.

[0099] Figure 7 The diagram illustrates a structural block diagram of a batch transaction data processing apparatus according to an embodiment of the present disclosure.

[0100] like Figure 7 As shown, the batch transaction data processing device 800 of this embodiment includes an acquisition module 810, a first data sharding module 820, a first sending module 830, a second data sharding module 840, and a second sending module 850.

[0101] The acquisition module 810 is used to acquire batch transaction data in response to a user's batch transaction remittance request. In one embodiment, the acquisition module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0102] The first data sharding module 820 is used to perform a first sharding process on the batch transaction data according to the feature identification information of the batch transaction data, so as to determine the first shard data and the target database server corresponding to the first shard data, wherein the feature identification information includes account identification information and / or transaction region identification information. In one embodiment, the first data sharding module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0103] The first sending module 830 is used to save the first fragment data to the target database server corresponding to the first fragment data. In one embodiment, the first sending module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0104] The second data sharding module 840 is used to perform a second sharding process on the first sharded data according to a preset splitting rule, so as to determine the second sharded data and the target partition in the message middleware corresponding to the second sharded data. In one embodiment, the second data sharding module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0105] The second sending module 850 is used to send the second fragment data to the target partition. In one embodiment, the second sending module 850 can be used to perform the operation S250 described above, which will not be repeated here.

[0106] According to an embodiment of this disclosure, the first data sharding module 820 includes: an acquisition submodule and a first sharding submodule.

[0107] The acquisition submodule is used to acquire the account identifier information and transaction region identifier information of the batch transaction data. In one embodiment, the acquisition submodule can be used to perform the operation S221 described above, which will not be repeated here.

[0108] The first sharding submodule is used to route and shard the batch transaction data according to the account identification information and / or the transaction region identification information, so as to obtain multiple first shard data and multiple target database servers corresponding to the multiple first shard data. In one embodiment, the first sharding submodule can be used to perform the operation S222 described above, which will not be repeated here.

[0109] According to embodiments of this disclosure, the second data sharding module includes: a second sharding submodule.

[0110] The second sharding submodule is used to perform a second sharding process on the first sharded data based on the account identification information of the first sharded data. In one embodiment, the second sharding submodule can be used to perform the operation S241 described above, which will not be repeated here.

[0111] According to embodiments of this disclosure, the second sharding submodule includes sharding units.

[0112] The sharding unit is used to perform consistent hash sharding calculation using the same account identification information of the first shard data as the target partition key value, and employs a routing algorithm to determine the target partition corresponding to the second shard data. In one embodiment, the sharding unit can be used to perform the operation S241 described above, which will not be repeated here.

[0113] According to embodiments of this disclosure, it further includes: a data verification module.

[0114] The data verification module is used to verify the batch transaction data. In one embodiment, the data verification module can be used to perform the operation S210 described above, which will not be repeated here.

[0115] According to embodiments of this disclosure, any plurality of modules among the acquisition module 810, the first data fragmentation module 820, the first transmission module 830, the second data fragmentation module 840, and the second transmission module 850 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 810, the first data fragmentation module 820, the first transmission module 830, the second data fragmentation module 840, and the second transmission module 850 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the acquisition module 810, the first data fragmentation module 820, the first transmission module 830, the second data fragmentation module 840, and the second transmission module 850 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0116] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a batch transaction data processing method according to an embodiment of the present disclosure.

[0117] like Figure 8As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0118] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0119] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0120] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the batch transaction data processing method according to the embodiments of this disclosure.

[0121] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0122] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the batch transaction data processing method provided in the embodiments of this disclosure.

[0123] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0124] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0125] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0126] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0129] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for processing batch transaction data, applied to a distributed system, the distributed system comprising multiple database servers, each of which deploys multiple container nodes, characterized in that, The processing method includes: In response to a user's bulk transaction remittance request, obtain bulk transaction data; The batch transaction data is first sharded based on the feature identification information of the batch transaction data to determine the first shard data and the target database server corresponding to the first shard data, wherein the feature identification information includes account identification information and / or transaction region identification information; Save the first shard data to the target database server corresponding to the first shard data; The first shard data is further sharded according to a preset splitting rule to determine the second shard data and the target partition in the corresponding message middleware; and Send the second fragment of data to the target partition; The step of performing a second sharding process on the first sharded data according to a preset sharding rule includes: The first shard data is then sharded a second time based on the account identification information of the first shard data. The second sharding process of the first shard data based on the account identifier information of the first shard data includes: Using the same account identifier information of the first shard data as the target partition key value, a routing algorithm is used to perform consistent hash shard calculation to determine the target partition corresponding to the second shard data.

2. The processing method according to claim 1, characterized in that, The step of performing a first sharding process on the batch transaction data based on the feature identification information of the batch transaction data, to determine the first shard data and the target database server corresponding to the first shard data, includes: Obtain the account identifier information and transaction region identifier information of the batch transaction data; The batch transaction data is routed and sharded according to the account identification information and / or the transaction region identification information to obtain multiple first shard data and multiple target database servers corresponding to the multiple first shard data.

3. The processing method according to claim 1, characterized in that, Before performing the first sharding process on the batch transaction data, the following is also included: The batch transaction data is then validated.

4. The processing method according to any one of claims 1 to 3, characterized in that, The messages for the target partition are consumed by at least one container node of the target database server.

5. A batch transaction data processing apparatus, applied to a distributed system, the distributed system comprising multiple database servers, each of which deploys multiple container nodes, characterized in that, The device includes: The acquisition module is used to retrieve batch transaction data in response to users' batch transaction remittance requests; The first data sharding module is used to perform a first sharding process on the batch transaction data according to the feature identification information of the batch transaction data, so as to determine the first shard data and the target database server corresponding to the first shard data, wherein the feature identification information includes account identification information and / or transaction region identification information; The first sending module is used to save the first fragment data to the target database server corresponding to the first fragment data; The second data sharding module is used to perform a second sharding process on the first sharded data according to a preset splitting rule, so as to determine the second sharded data and the target partition in the message middleware corresponding to the second sharded data; and The second sending module is used to send the second fragment data to the target partition; The second data sharding module is further configured to perform a second sharding process on the first sharding data based on the account identification information of the first sharding data; using the same account identification information of the first sharding data as the target partition key value, and employing a routing algorithm to perform consistent hash sharding calculation, so as to determine the target partition corresponding to the second sharding data and the second sharding data.

6. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the processing method according to any one of claims 1 to 4.

8. A computer program product comprising a computer program that, when executed by a processor, implements the processing method according to any one of claims 1 to 4.

Images