Transaction processing method and device, electronic equipment and storage medium

Abstract

The invention provides a transaction processing method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining original liquidation flow data; determining a plurality of subtasks based on the original liquidation pipeline data; determining the number of target nodes based on the total data amount corresponding to the plurality of sub-tasks, the obtained available resource data of all sub-nodes on the current day and the maximum and minimum value of the total processing time on the current day; distributing the sub-tasks to the corresponding sub-nodes based on the number of the target nodes and the total data amount corresponding to the plurality of sub-tasks, so that each sub-node processes the received sub-tasks respectively; and obtaining the processing result sent by each sub-node, and summarizing the processing results so as to improve the processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a transaction processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the centralized processing scenario of large-scale financial transaction stream data, the processing flow of the existing financial transaction stream batch processing system usually directly performs filtering and aggregation operations on large amounts of data based on a row-oriented relational database. Since the row-oriented database needs to scan the entire row data even if only part of the fields is involved in each query, the I / O overhead will significantly increase when facing processing tasks mainly using part of the fields for aggregation calculation, which is not conducive to processing tasks with billions of data, and is prone to problems such as long execution time and system lag. Therefore, such systems face performance bottlenecks when processing data, and the processing efficiency is difficult to guarantee. SUMMARY

[0003] Therefore, the embodiments of the present application provide a transaction processing method and device, electronic equipment and storage medium to solve the problem of difficult to guarantee the processing efficiency in the prior art.

[0004] To achieve the above object, the embodiments of the present application provide the following technical solutions:

[0005] The first aspect shows a transaction processing method, which comprises:

[0006] obtaining original clearing stream data;

[0007] determining a plurality of sub-tasks based on the original clearing stream data;

[0008] determining the number of target nodes based on the total amount of data corresponding to the plurality of sub-tasks, the available resource data of all sub-nodes obtained on the same day, and the maximum and minimum values of the total processing time on the same day;

[0009] allocating the sub-tasks to the corresponding sub-nodes based on the number of target nodes and the total amount of data corresponding to the plurality of sub-tasks, so that each sub-node processes the sub-tasks received by itself;

[0010] obtaining the processing results sent by each sub-node and aggregating them.

[0011] Optionally, determining a plurality of sub-tasks based on the original clearing stream data comprises:

[0012] performing data aggregation processing on one or more columns of data of each data table in the original clearing stream data to obtain new data;

[0013] combining the new data and the original clearing stream data to obtain target clearing stream data;

[0014] splitting the target clearing flow data according to a business dimension to obtain a plurality of split data;

[0015] taking each of the split data as a subtask to obtain a plurality of subtasks.

[0016] Optionally, the target node quantity is determined based on a total amount of data corresponding to the plurality of subtasks, available resource data of all sub-nodes obtained on the day, and a maximum total processing time on the day, and the target node quantity is determined based on the total amount of data corresponding to the plurality of subtasks, the first value, and a preset maximum node quantity.

[0017] estimating the total amount of data corresponding to the plurality of subtasks to determine a predicted processing speed;

[0018] obtaining available resource data of all sub-nodes on the day, and searching for an adjustment coefficient corresponding to the available resource data;

[0019] determining a first value based on the predicted processing speed, the adjustment coefficient, and the maximum total processing time on the day;

[0020] determining the target node quantity based on the total amount of data corresponding to the plurality of subtasks, the first value, and a preset maximum node quantity.

[0021] Optionally, the target node quantity is determined based on the total amount of data corresponding to the plurality of subtasks, the first value, and a preset maximum node quantity, and the target node quantity is determined based on a ratio of the total amount of data corresponding to the plurality of subtasks to the first value.

[0022] determining a required node quantity on the day based on a ratio of the total amount of data corresponding to the plurality of subtasks to the first value;

[0023] comparing the required node quantity on the day and the preset maximum node quantity;

[0024] if the required node quantity on the day is less than the preset maximum node quantity, taking the required node quantity on the day as the target node quantity;

[0025] if the preset maximum node quantity is less than the required node quantity on the day, taking the preset maximum node quantity as the target node quantity.

[0026] Optionally, the subtasks are allocated to corresponding sub-nodes based on the target node quantity and the total amount of data corresponding to the plurality of subtasks, and the subtasks are allocated to corresponding sub-nodes based on the target node quantity and the total amount of data corresponding to the plurality of subtasks.

[0027] dividing the sub-nodes according to the target node quantity to determine a first sub-node and a second sub-node, and the number of the first sub-nodes is a plurality;

[0028] obtaining a task load quantity of the first sub-nodes by rounding up a ratio of the total amount of data corresponding to the plurality of subtasks to the target node quantity;

[0029] determining a task load quantity of the second sub-node based on the number of the first sub-nodes, the task load quantity of the first sub-node, and the total data amount corresponding to the plurality of sub-tasks;

[0030] allocating the sub-tasks to each first sub-node in turn according to the task load quantity of the first sub-node, and allocating the sub-tasks to the second sub-node according to the task load quantity of the second sub-node.

[0031] Optionally, the method further comprises:

[0032] calculating a product of the predicted processing speed, the adjustment coefficient and the preset value to obtain a second value;

[0033] determining a target value based on the total data amount corresponding to the plurality of sub-tasks, the second value and a required node number of the day, wherein the required node number of the day is a preset value when the first time of calculation, the required node number of the day is an adjusted value when the nth time of calculation, n is a positive integer greater than or equal to 2;

[0034] if it is determined that the target value is greater than the maximum value of the total processing time of the day, adjusting the required node number of the day until it is determined that the target value is less than or equal to the maximum value of the total processing time of the day.

[0035] A second aspect shows a transaction processing device, the device comprising:

[0036] an acquisition unit configured to acquire original clearing stream data;

[0037] a preprocessing unit configured to determine a plurality of sub-tasks based on the original clearing stream data;

[0038] a determination unit configured to determine a target node quantity based on a total data amount corresponding to the plurality of sub-tasks, acquired available resource data of all sub-nodes of the day, and a maximum value of total processing time of the day;

[0039] a processing task scheduling unit configured to allocate the sub-tasks to corresponding sub-nodes based on the target node quantity and the total data amount corresponding to the plurality of sub-tasks, so that each sub-node processes the sub-tasks received by itself respectively;

[0040] a summary processing unit configured to acquire processing results sent by each sub-node and summarize the processing results.

[0041] Optionally, the preprocessing unit is specifically configured to:

[0042] perform data aggregation processing on one or more columns of data of each data table in the original clearing stream data to obtain new data;

[0043] Combine the new data and the original clearing flow data set to obtain target clearing flow data;

[0044] Split the target clearing flow data according to a business dimension to obtain multiple split data;

[0045] Each split data is taken as a subtask to obtain multiple subtasks.

[0046] The third aspect shows an electronic device, which includes a processor and a memory, the memory is used to store data generated program code and data, and the processor is used to call the program instructions in the memory to execute the transaction processing method as any one of the first aspects of the embodiments of the present application.

[0047] The fourth aspect shows a storage medium, which includes a program, wherein when the program runs, the device where the storage medium is located is controlled to execute the transaction processing method as any one of the first aspects of the embodiments of the present application.

[0048] Based on the above-mentioned transaction processing method, device, electronic device and storage medium provided by the embodiments of the present application, the method includes: obtaining original clearing flow data; determining multiple subtasks based on the original clearing flow data; determining the target node quantity based on the total data amount corresponding to the multiple subtasks, the available resource data of all subnodes obtained on the day, and the maximum value of the total processing time on the day; distributing the subtasks to the corresponding subnodes based on the target node quantity and the total data amount corresponding to the multiple subtasks, so that each subnode processes the subtasks received by itself respectively; obtaining the processing results sent by each subnode and performing aggregation. In the embodiments of the present application, the original clearing flow data is preprocessed to determine the number and total data amount of the subtasks to be processed on the day; then, reasonable task division is made based on the total data amount corresponding to the multiple subtasks, the available resource data of all subnodes obtained on the day, and the maximum value of the total processing time on the day, the number of nodes processed daily and the task load of each node are dynamically adjusted to realize the load balancing among the nodes; further, the processing results of the subtasks processed by each subnode are aggregated; thereby the processing efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings.

[0050] Figure 1A flowchart of a transaction processing method according to an embodiment of the present application is shown in FIG. 1.

[0051] Figure 2 A flowchart of a transaction processing method according to an embodiment of the present application is shown in FIG. 1.

[0052] Figure 3 A flowchart of a transaction processing method according to an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0054] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in other than the order illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a list of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0055] It should be noted that the description involving "first", "second" and the like in the present application is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of a person skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection required by the present application.

[0056] In this application, the terms "comprising", "containing" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0057] Referring to Figure 1 A flowchart of a transaction processing method is shown for an embodiment of the application, the method comprising:

[0058] Step S101: Obtain original clearing stream data;

[0059] In the process of implementing step S101, the original clearing stream data issued by the clearing institution is received through file transmission, interface calling, etc.

[0060] It should be noted that multiple types of data transmission methods (such as file transmission, interface calling, etc.) are supported.

[0061] Optionally, it further comprises:

[0062] The original clearing stream data is subjected to data compliance verification to filter data in the original clearing stream data that does not comply with the compliance verification.

[0063] It should be noted that the data compliance verification is set according to the characteristics of the received data. For example, the characters of field A must be symbolic;

[0064] Step S102: Determine a plurality of sub-tasks based on the original clearing stream data;

[0065] It should be noted that in the process of implementing step S102, the following steps are included.

[0066] Step S11: Perform data aggregation processing on one or more columns of data of each data table in the original clearing stream data to obtain new data;

[0067] In the process of implementing step S11, when processing the original clearing stream data according to the preset business rules, specifically, querying and summarizing one or more columns of data in the data table, at this time, aggregation functions including but not limited to sum SUM, average AVG, count COUNT, maximum MAX and minimum MIN are used to do aggregation operation on single column data, and grouping query GROUP BY is also used to group and aggregate multi-column data grouping fields to obtain new data.

[0068] It should be noted that the common processing scenarios include important financial business data such as transaction amount, transaction number, card issuer fee, receiving party fee, installment fee and points, and the calculation needs to be combined with preset business rules such as accounting date, card type, province and city, network point and transaction type.

[0069] Optionally, the above operation needs to be processed by line-by-line scanning if using a general database or processing program, which greatly limits the processing efficiency. The columnar database used has optimized columnar scanning and parallel aggregation capabilities, solving this efficiency bottleneck and reducing the calculation pressure and time consumption of subsequent processing, that is, the number of columns of each data table in the original clearing flow data is aggregated by using the columnar database.

[0070] In the present application, the original data is preprocessed by introducing a columnar database, and is divided and aggregated according to the business dimensions (province and city, network point, transaction type), which can reduce the calculation burden of subsequent processing nodes, improve the overall processing performance, and optimize the I / O resource consumption.

[0071] Step S12: combining the new data and the original clearing flow data to obtain target clearing flow data;

[0072] Step S13: splitting the target clearing flow data according to the business dimensions to obtain a plurality of split data;

[0073] It should be noted that the business dimensions are also set according to multiple experiments in advance, such as transaction type, network point belonging to region and card type.

[0074] Step S14: taking each split data as a subtask to obtain a plurality of subtasks.

[0075] In the process of implementing step S14, each split data can be taken as a subtask, or a plurality of split data can be taken as a subtask according to other rules.

[0076] It should be noted that the task load of each subtask is the same.

[0077] Step S103: determining the target node quantity based on the total amount of data corresponding to the plurality of subtasks, the available resource data of all subnodes obtained on the day, and the total processing time value on the day;

[0078] It should be noted that in the process of implementing step S103, it includes:

[0079] Step S21: estimating the total amount of data corresponding to the plurality of subtasks to determine the predicted processing speed;

[0080] It should be noted that the estimation model is obtained by training according to historical data amounts corresponding to a plurality of sub-tasks per day and corresponding prediction processing speeds.

[0081] In the process of implementing step S21, the obtained estimation model is called to estimate the input data amounts corresponding to the plurality of sub-tasks , and the processing capacity of a single node, i.e., the prediction processing speed , is estimated.

[0082] It should be noted that the unit of the prediction processing speed is pieces per second.

[0083] It should be noted that the data amount corresponding to the plurality of sub-tasks refers to the total amount of data to be processed after pre-processing on the day .

[0084] Step S22: Obtain available resource data of all sub-nodes on the day, and find an adjustment coefficient corresponding to the available resource data .

[0085] In the process of implementing step S22, the available resource data of all sub-nodes on the day is detected, and a corresponding relationship between the preset adjustment coefficient and the available resource data is found to determine the adjustment coefficient corresponding to the available resource data .

[0086] Step S23: Determine a first value based on the prediction processing speed, the adjustment coefficient, and the maximum value of the total processing time on the day

[0087] In the process of implementing step S22, the product of the prediction processing speed , the adjustment coefficient , the maximum value of the total processing time on the day , and a preset value is calculated to obtain the first value S.

[0088] It should be noted that the maximum value of the total processing time on the day is set in advance according to actual conditions.

[0089] The preset value is set in advance and can be set to 60 in general.

[0090] Step S24: Determine a target number of nodes based on the data amounts corresponding to the plurality of sub-tasks, the first value, and a preset maximum number of nodes.

[0091] It should be noted that in the process of implementing step S24, the following steps are included.

[0092] Step S31: determining the required node number of the day based on the ratio of the total data amount corresponding to the plurality of sub-tasks to the first value.

[0093] In the process of implementing step S31, the total data amount corresponding to the plurality of sub-tasks is rounded up to the ratio of the first value S, to obtain the required node number of the day , as shown in formula (1).

[0094] Formula (1):

[0095]

[0096] Optionally, it further comprises:

[0097] calculating the predicted processing speed , the adjustment coefficient and the product of the preset value to obtain the second value W.

[0098] Then, the target value is determined based on the total data amount corresponding to the plurality of sub-tasks, the second value and the required node number of the day.

[0099] Specifically, the product of the second value W and the required node number of the day is calculated to obtain the first product; then, the total data amount corresponding to the plurality of sub-tasks is rounded up to the ratio of the first product to obtain the target value w.

[0100] It should be noted that if the required node number of the day is calculated for the first time, it is a preset value, and if the required node number of the day is calculated for the nth time, the required node number of the day is an adjusted value, i.e., it can be an adjusted required node number of the day , and n is a positive integer greater than or equal to 2.

[0101] Optionally, the adjusted required node number of the day may be determined based on the product of the preset value and the preset weight *n.

[0102] The preset value is set in advance, or it can be set according to historical data; the preset weight is also set in advance according to multiple experiments or experience.

[0103] Then, it is judged whether the target value w is less than or equal to the maximum value of the total processing time of the day .

[0104] If it is less than or equal to, it is determined that the required node number of the day does not need to be adjusted, and step S32 is executed.

[0105] If greater, adjust the required number of nodes for the day , until the target value w is determined to be less than or equal to the maximum value of the total processing time for the day , as shown in equation (2).

[0106] Equation (2):

[0107]

[0108] Step S32: Compare the required number of nodes for the day with the preset maximum number of nodes, if the required number of nodes for the day is less than the preset maximum number of nodes, then execute step S33, if the preset maximum number of nodes is less than the required number of nodes for the day, then execute step S34.

[0109] Step S33: Take the required number of nodes for the day as the target number of nodes to be determined.

[0110] Step S34: Take the preset maximum number of nodes as the target number of nodes to be determined.

[0111] In the process of implementing steps S32 to S34, the required number of nodes for the day and the preset maximum number of nodes are substituted into equation (3) to determine the minimum data, if the required number of nodes for the day is less than the preset maximum number of nodes, then take the required number of nodes for the day as the target number of nodes to be determined, if the preset maximum number of nodes is less than the required number of nodes for the day, then take the preset maximum number of nodes as the target number of nodes to be determined .

[0112] Equation (3):

[0113]

[0114] The present application dynamically generates task fragments according to the amount of data for the day and the current available computing resources, can automatically adapt and balance the task load of each node, and realizes the maximization of computing resource utilization.

[0115] Step S104: Based on the target number of nodes and the total amount of data corresponding to the plurality of sub-tasks, the sub-tasks are allocated to the corresponding sub-nodes, so that each sub-node processes the sub-tasks received by itself respectively;

[0116] It should be noted that in the process of implementing step S104, the following steps are included.

[0117] Step S41: dividing the target node quantity into a first sub-node quantity and a second sub-node quantity according to the target node quantity dividing the sub-nodes to determine a first sub-node and a second sub-node, the first sub-node quantity being a plurality;

[0118] In the process of implementing step S41, since the total data quantity cannot be divided by the node quantity, the target node quantity is divided into a first sub-node and a second sub-node;

[0119] wherein the first sub-node quantity is a plurality, and the second sub-node quantity is one.

[0120] Step S42: determining a task load quantity of the first sub-node based on the total data quantity corresponding to the plurality of sub-tasks and the target node quantity and rounding up to obtain the task load quantity of the first sub-node;

[0121] In the process of implementing step S42, the total data quantity corresponding to the plurality of sub-tasks and the target node quantity are substituted into formula (4) to determine the task load quantity of the first sub-node .

[0122] Formula (4):

[0123]

[0124] Step S43: determining a task load quantity of the second sub-node based on the first sub-node quantity, the task load quantity of the first sub-node, and the total data quantity corresponding to the plurality of sub-tasks.

[0125] In the process of implementing step S43, the product of the first sub-node quantity and the task load quantity of the first sub-node is calculated to obtain the total task quantity allocated to all first sub-nodes; then, the difference between the total data quantity corresponding to the plurality of sub-tasks and the total task quantity allocated to all first sub-nodes is calculated to obtain the task load quantity of the second sub-node.

[0126] Step S44: sequentially allocating the sub-tasks to each first sub-node according to the task load quantity of the first sub-node, and allocating the sub-tasks to the second sub-node according to the task load quantity of the second sub-node.

[0127] In the process of implementing step S44, the same number of sub-tasks as the task load quantity of the first sub-node is allocated to each first sub-node; at the same time, the same number of sub-tasks as the task load quantity of the second sub-node is allocated to the second sub-node.

[0128] In the process of implementing step S104, after step S104 is executed, i.e., after the subtasks are issued to each subnode, each subnode (the first subnode and the second subnode) starts to execute the corresponding business logic according to the allocated task.

[0129] It should be noted that each subnode is an independent running unit and can process all task fragments in parallel, execute specific business logic (such as field parsing, account reconciliation, summary statistics, etc.), and the running of each subnode has idempotency, avoiding abnormal results caused by repeated execution of tasks.

[0130] The present application shows a distributed subnode parallel processing architecture, each subnode independently completes the fragment task, and the overall system has horizontal expansion capability and can stably process daily data of hundreds of millions.

[0131] Step S105: obtaining the processing result sent by each subnode and collecting the processing result.

[0132] In the process of implementing step S105, the execution of each subnode is queried, and the processing result obtained after the subnode completes the processing task, i.e., the intermediate result, is collected and summarized.

[0133] It should be noted that when summarizing the results, different summary granularities can be used for hierarchical summarization to reduce processing pressure during summarization, and finally summarized into a complete processing result. The summary operation process is as follows:

[0134] Obtaining the most fine-grained fragment set after each first subnode and second subnode completes processing, i.e., the processing result ;

[0135] Among them, each fragment does local summary within the fragment, uses an aggregation key with granularity k (such as subnode number and network point number), and obtains a summary result ;

[0136] The local summary result is further summarized, different fragments can be divided into different set groups according to certain rules, and the local summary results of the same set group are summarized within the set to obtain the result, as shown in formula (5).

[0137]

[0138] To reduce processing pressure, the processing result of the set group can still be divided into different granularities to form a multi-level hierarchical upward summary, showing a tree-shaped summary structure. Finally, the results of the set group are summarized into the final result, and is the final summary data.

[0139] Optionally, it also includes: collecting the summary data Write into the target storage database for subsequent review, display and archiving.

[0140] Optionally, the process of implementing steps S101 to S105 can also be implemented by Figure 2 It is described.

[0141] The present application adopts a multi-level structured design. The entire transaction processing flow is decoupled according to functions and processing stages and divided into multiple levels, so that each level can be independently extended, optimized and deployed, with stronger module flexibility, maintainability and evolvability. The multi-level involvement does not affect the normal processing of other layers that are already executing when an exception occurs in a certain layer, improving system fault tolerance and efficiency of post-fault recovery.

[0142] In the embodiment of the present application, the original clearing flow data is preprocessed to determine the number of subtasks to be processed and the total amount of data on the day; then, based on the total amount of data corresponding to the multiple subtasks, the available resource data of all subnodes obtained on the day and the maximum value of the total processing time on the day, reasonable task division is made, the number of nodes processed daily and the task load of each node are dynamically adjusted to realize load balancing between nodes; then the processing results of the subtasks processed by each subnode are aggregated level by level using different aggregation granularity, the processing pressure during aggregation is reduced, and finally a complete processing result is aggregated, and the integrity of the final data output is guaranteed. Thus the processing efficiency is improved.

[0143] Based on the transaction processing method shown in the above embodiment of the present application, correspondingly, the present application also shows a transaction processing device, as shown in Figure 3 The device comprises:

[0144] The acquisition unit 301 is configured to acquire original clearing flow data.

[0145] The preprocessing unit 302 is configured to determine multiple subtasks based on the original clearing flow data.

[0146] The determination unit 303 is configured to determine the target node quantity based on the total amount of data corresponding to the multiple subtasks, the available resource data of all subnodes obtained on the day and the maximum value of the total processing time on the day.

[0147] The processing task scheduling unit 304 is configured to allocate the subtasks to the corresponding subnodes based on the target node quantity and the total amount of data corresponding to the multiple subtasks, so that each subnode processes the subtasks received by itself.

[0148] The aggregation processing unit 305 is configured to acquire the processing results sent by each subnode and aggregate them.

[0149] The specific principles and execution processes of each unit in the transaction processing device disclosed in the embodiments of the present application are the same as the corresponding contents in the transaction processing method provided in the embodiments of the present application, and can be referred to the corresponding parts in the transaction processing method disclosed in the embodiments of the present application, which will not be described here.

[0150] In the embodiments of the present application, the original clearing stream data is preprocessed to determine the number of subtasks and the total amount of data to be processed on the day; then, based on the total amount of data corresponding to the plurality of subtasks, the available resource data of all subnodes obtained on the day and the maximum value of the total processing time on the day, reasonable task division is made, the number of nodes processed daily and the task load of each node are dynamically adjusted to realize load balancing among the nodes; then the processing results of the subtasks processed by each subnode are aggregated in different aggregation granularities level by level to reduce the processing pressure during aggregation, and finally aggregated into a complete processing result, and the integrity of the final data output is ensured. Thus the processing efficiency is improved.

[0151] Optionally, based on the transaction processing device shown in the embodiments of the present application, the preprocessing unit 302 is specifically configured to:

[0152] perform data aggregation processing on one or more columns of data of each data table in the original clearing stream data to obtain new data;

[0153] combine the new data and the original clearing stream data to obtain target clearing stream data;

[0154] split the target clearing stream data according to the business dimension to obtain a plurality of split data;

[0155] take each split data as a subtask to obtain a plurality of subtasks.

[0156] Optionally, based on the transaction processing device shown in the embodiments of the present application, the determining unit 303 is specifically configured to:

[0157] estimate the total amount of data corresponding to the plurality of subtasks to determine a predicted processing speed;

[0158] obtain available resource data of all subnodes on the day, and find an adjustment coefficient corresponding to the available resource data;

[0159] determine a first value based on the predicted processing speed, the adjustment coefficient and the maximum value of the total processing time on the day;

[0160] determine a target number of nodes based on the total amount of data corresponding to the plurality of subtasks, the first value and a preset maximum number of nodes.

[0161] The target node quantity is determined based on the total data quantity corresponding to the plurality of sub-tasks, the first numerical value, and a preset maximum node quantity, including:

[0162] The required node quantity of the day is determined based on a ratio of the total data quantity corresponding to the plurality of sub-tasks to the first numerical value;

[0163] The required node quantity of the day and the preset maximum node quantity are compared in size;

[0164] If the required node quantity of the day is less than the preset maximum node quantity, the required node quantity of the day is taken as the determined target node quantity;

[0165] If the preset maximum node quantity is less than the required node quantity of the day, the preset maximum node quantity is taken as the determined target node quantity.

[0166] Optionally, based on the transaction processing device shown in the above embodiment of the present application, the task scheduling unit 304 is specifically used for:

[0167] The sub-nodes are divided according to the target node quantity, to obtain first sub-nodes and second sub-nodes, and the number of the first sub-nodes is a plurality;

[0168] The task load quantity of the first sub-nodes is obtained by taking an upper limit of a ratio of the total data quantity corresponding to the plurality of sub-tasks to the target node quantity;

[0169] The task load quantity of the second sub-nodes is determined based on the number of the first sub-nodes, the task load quantity of the first sub-nodes, and the total data quantity corresponding to the plurality of sub-tasks;

[0170] The sub-tasks are sequentially assigned to each first sub-node according to the task load quantity of the first sub-nodes, and the sub-tasks are assigned to the second sub-nodes according to the task load quantity of the second sub-nodes.

[0171] Optionally, based on the transaction processing device shown in the above embodiment of the present application, the determination unit 303 is further used for:

[0172] The second numerical value is obtained by calculating a product of the predicted processing speed, the adjustment coefficient, and the preset numerical value;

[0173] The target numerical value is determined based on the total data quantity corresponding to the plurality of sub-tasks, the second numerical value, and the required node quantity of the day, wherein the required node quantity of the day is a preset value when the first time is calculated, the required node quantity of the day is an adjusted value when the n-th time is calculated, and n is a positive integer greater than or equal to 2;

[0174] If it is determined that the target value is greater than the daily total processing time extreme value, the required node number of the day is adjusted until it is determined that the target value is less than or equal to the daily total processing time extreme value.

[0175] The embodiment of the application provides an electronic device, the electronic device includes a processor and a memory, the memory is used for storing transaction processing program code and data, and the processor is used for calling program instructions in the memory to execute steps shown in the transaction processing method in the above embodiment.

[0176] The embodiment of the application provides a storage medium, namely a computer readable storage medium, and the storage medium includes the electronic device provided by the embodiment of the application and is used for executing the transaction processing method disclosed by the embodiment of the application.

[0177] Each of the embodiments in the specification is described in a progressive manner, and the same and similar parts of each embodiment can be referred to each other, and each embodiment mainly describes differences from other embodiments. Especially, for the system or the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment. The system and the system embodiment described above are only illustrative, wherein the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0178] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized by electronic hardware, computer software or combination of the two. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized by hardware or software depends on the specific application and design constraints of the technical scheme. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0179] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A transaction processing method, characterized in that, The method includes: Obtain the original settlement transaction data; Multiple sub-tasks are determined based on the original clearing flow data; The number of target nodes is determined based on the total amount of data corresponding to the multiple sub-tasks, the available resource data of all sub-nodes obtained on the same day, and the maximum and minimum total processing time on the same day. The subtasks are assigned to the corresponding sub-nodes based on the number of target nodes and the total amount of data corresponding to the multiple subtasks, so that each sub-node can process the subtasks it receives. Obtain the processing results sent by each of the child nodes and summarize them.

2. The method according to claim 1, characterized in that, Based on the original clearing log data, several sub-tasks are determined, including: Data aggregation is performed on one or more columns of data from each data table in the original settlement flow data to obtain new data; The new data and the original clearing flow data are combined to obtain the target clearing flow data; The target clearing flow data is split according to business dimensions to obtain multiple split data; Each of the aforementioned split data points is treated as a subtask, resulting in multiple subtasks.

3. The method according to claim 1, characterized in that, The target number of nodes is determined based on the total amount of data corresponding to the multiple sub-tasks, the available resource data of all child nodes obtained for the day, and the maximum / minimum total processing time for the day, including: Estimate the total amount of data corresponding to the multiple sub-tasks to determine the prediction processing speed; Obtain the available resource data of all child nodes for the day, and find the adjustment coefficient corresponding to the available resource data; The first value is determined based on the predicted processing speed, the adjustment coefficient, and the maximum / minimum total processing time for the day; The target number of nodes is determined based on the total amount of data corresponding to the multiple sub-tasks, the first value, and the preset maximum number of nodes.

4. The method according to claim 3, characterized in that, The target number of nodes is determined based on the total amount of data corresponding to the multiple subtasks, the first value, and the preset maximum number of nodes, including: The number of nodes required for the day is determined based on the ratio of the total amount of data corresponding to the multiple sub-tasks to the first value; Compare the number of nodes required for the day with the preset maximum number of nodes; If the number of nodes required on that day is less than the preset maximum number of nodes, the number of nodes required on that day will be used as the target number of nodes. If the preset maximum number of nodes is less than the number of nodes required for the day, the preset maximum number of nodes will be used as the target number of nodes.

5. The method according to claim 1, characterized in that, Based on the number of target nodes and the total amount of data corresponding to the multiple sub-tasks, the sub-tasks are assigned corresponding sub-nodes, including: The child nodes are divided according to the number of target nodes to determine the first child node and the second child node, wherein the number of the first child node is multiple; The task load of the first sub-node is obtained by rounding up the ratio of the total amount of data corresponding to the multiple sub-tasks to the number of target nodes. The number of task loads allocated to the second child node is determined based on the number of the first child node, the task load of the first child node, and the total amount of data corresponding to the multiple sub-tasks. The subtasks are sequentially assigned to each first child node according to the number of tasks in the first child node, and the subtasks are assigned to the second child node according to the number of tasks in the second child node.

6. The method according to claim 1, characterized in that, Also includes: The second value is obtained by multiplying the predicted processing speed, the adjustment coefficient, and the preset value. The target value is determined based on the total amount of data corresponding to the multiple sub-tasks, the second value, and the number of nodes required on the day. The number of nodes required on the day is a preset value for the first calculation and an adjusted value for the nth calculation, where n is a positive integer greater than or equal to 2. If the target value is determined to be greater than the maximum total processing time of the day, the number of nodes required for the day is adjusted until the target value is determined to be less than or equal to the maximum total processing time of the day.

7. A transaction processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire raw settlement transaction data; The preprocessing unit is used to determine multiple sub-tasks based on the original clearing flow data; The determining unit is used to determine the number of target nodes based on the total amount of data corresponding to the multiple sub-tasks, the available resource data of all sub-nodes obtained on the same day, and the maximum and minimum total processing time on the same day. A task scheduling unit is used to allocate the subtasks to the corresponding sub-nodes based on the number of target nodes and the total amount of data corresponding to the multiple subtasks, so that each sub-node can process the subtasks it receives. The aggregation processing unit is used to obtain the processing results sent by each of the child nodes and aggregate them.

8. The apparatus according to claim 7, characterized in that, The preprocessing unit is specifically used for: Data aggregation is performed on one or more columns of data from each data table in the original settlement flow data to obtain new data; The new data and the original clearing flow data are combined to obtain the target clearing flow data; The target clearing flow data is split according to business dimensions to obtain multiple split data; Each of the aforementioned split data points is treated as a subtask, resulting in multiple subtasks.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store program code and data for data generation, and the processor being used to call program instructions in the memory to execute the transaction processing method as described in any one of claims 1-6.

10. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the transaction processing method as described in any one of claims 1-6.

Images