Business data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a business data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining task data corresponding to a to-be-processed service, wherein the to-be-processed service indicates a service of which the response duration is greater than a response duration threshold; fragmenting the task data to obtain target fragmented data, and sending the target fragmented data to a plurality of slave nodes; and receiving processing results returned by the plurality of slave nodes, wherein the processing results are obtained by processing the received target fragment data by the plurality of slave nodes. By adopting the method, the task data processing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a business data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the popularization of the Internet and the increase in the number of users, the timeliness requirements for business distribution are becoming increasingly higher.

[0003] Currently, a better approach is to use Redis to cache business information and release the business data to be released through the thread pool.

[0004] However, business distribution involves processing a series of complex business logic. If business tasks are submitted quickly and threads process complex business tasks slowly, there will be a risk of business task accumulation, thus affecting the efficiency of business distribution. Summary of the invention

[0005] Based on this, it is necessary to provide a business data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a service data processing method, which is applied in a master node, and the method includes:

[0007] Acquire task data corresponding to the pending business, where the pending business indicates a business whose response time is greater than a response time threshold;

[0008] Slice the task data to obtain target slice data, and send the target slice data to multiple slave nodes;

[0009] Receive processing results returned by the multiple slave nodes, where the processing results are obtained by the multiple slave nodes processing the received target shard data.

[0010] In one embodiment, the slave node includes a first slave node; and slicing the task data to obtain target slicing data includes:

[0011] Slicing the task data to obtain initial slicing data, and sending the initial slicing data to a plurality of first slave nodes; the plurality of first slave nodes are used to perform a first logic verification on the initial slicing data;

[0012] Receive the verified initial shard data sent by multiple first slave nodes, and shard the verified initial shard data to obtain target shard data.

[0013] In one embodiment, sending the initial shard data to the plurality of first slave nodes comprises:

[0014] The initial shard data is sent to the first message queue; each of the first slave nodes is used to obtain the initial shard data from the first message queue, perform a first logic verification on the initial shard data, and send the verified initial shard data to the second message queue;

[0015] The receiving the verified initial shard data sent by the plurality of first slave nodes comprises:

[0016] The verified initial shard data returned by each of the first slave nodes is obtained from the second message queue.

[0017] In one embodiment, the slave node includes a second slave node; and sending the target shard data to the plurality of slave nodes includes:

[0018] Send the target shard data to the third message queue; each of the second slave nodes is used to obtain the target shard data from the third message queue, perform distribution processing according to the target shard data, and send the processing result to the fourth message queue;

[0019] The receiving the processing results returned by the plurality of slave nodes comprises:

[0020] The processing result returned by each of the second slave nodes is obtained from the fourth message queue.

[0021] In a second aspect, the present application provides a service data processing method, which is applied to a slave node, and the method includes:

[0022] For each of the slave nodes, target shard data is obtained; the target shard data is determined based on the master node dividing the task data corresponding to the pending business; the pending business indicates a business whose response time is greater than a response time threshold;

[0023] Perform distribution processing according to the target shard data to obtain a processing result;

[0024] The processing result is returned to the master node.

[0025] In one embodiment, the slave node includes a first slave node; and obtaining target shard data includes:

[0026] Acquire initial sharding data through each of the first slave nodes, where the initial sharding data is based on the data sent by the master node;

[0027] A first logical verification is performed on the initial shard data, and the verified initial shard data is returned to the master node; the master node is used to receive the verified initial shard data, and shard the verified initial shard data to obtain the target shard data.

[0028] In one embodiment, the slave node includes a second slave node; and the distributing process is performed according to the target shard data to obtain a processing result, including:

[0029] Performing a second logic verification on the target shard data through each of the second slave nodes;

[0030] If the second logic verification passes, extracting the data records to be distributed from the target shard data;

[0031] Obtaining target data to be distributed according to the distribution rules and the data record to be distributed through the target script;

[0032] The target data to be distributed is distributed and processed through the target script to obtain the processing result of each of the second slave nodes.

[0033] In a third aspect, the present application further provides a service data processing device, applied to a master node, the device comprising:

[0034] A first acquisition module is used to acquire task data corresponding to a pending business, where the pending business indicates a business whose response time is greater than a response time threshold;

[0035] A slicing module, used for slicing the task data to obtain target slicing data, and sending the target slicing data to multiple slave nodes;

[0036] The receiving module is used to receive the processing results returned by the multiple slave nodes, where the processing results are obtained by the multiple slave nodes processing the received target shard data.

[0037] In a fourth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0038] In a fifth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0039] The above-mentioned business data processing method, device, computer equipment, computer-readable storage medium and computer program product obtain the task data corresponding to the business to be processed through the master node, and the business to be processed indicates the business whose response time is greater than the response time threshold, that is, the business that takes a long time; the task data corresponding to the business that takes a long time is sliced ​​to obtain the target slice data, and the target slice data is sent to multiple slave nodes; and the processing results returned by the multiple slave nodes are received. The master node slices the task data corresponding to the business that takes a long time, distributes it to multiple slave nodes, and allows multiple slave nodes to execute and process the task data concurrently, thereby improving the efficiency of task data processing.

[0040] For each slave node, the target shard data is obtained; the target shard data is determined based on the master node dividing the task data corresponding to the pending business; the pending business indicates the business whose response time is greater than the response time threshold; multiple slave nodes distribute and process according to the target shard data to obtain the processing result; and return the processing result to the master node. Multiple slave nodes receive the target shard data distributed by the master node, and distribute and process the target shard data through multiple slave nodes, so that multiple slave nodes distribute and process the task data concurrently, thereby improving the efficiency of task data processing and execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 An application environment diagram of a business data processing method in an embodiment;

[0043] Figure 2 A flowchart of a method for processing business data in one embodiment;

[0044] Figure 3 A schematic diagram of a process of slicing task data to obtain target slicing data in one embodiment;

[0045] Figure 4 A schematic diagram of distributing task data to slave nodes in one embodiment;

[0046] Figure 5 A schematic diagram of distributing initial shard data to slave nodes in one embodiment;

[0047] Figure 6A schematic diagram of a process of distributing initial shard data to a first slave node and receiving data returned by the first slave node in an embodiment;

[0048] Figure 7 A schematic diagram of a process of data interaction between a master node and a slave node through a message middleware in one embodiment;

[0049] Figure 8 A schematic diagram of a process of distributing target shard data to a second slave node and receiving data returned by the second slave node in an embodiment;

[0050] Fig. 9 is a flowchart of a business data processing method in another embodiment;

[0051] Fig.10 A schematic diagram of a process for obtaining target shard data in one embodiment;

[0052] Fig.11 A schematic diagram of a process of distributing and processing target shard data in one embodiment;

[0053] Fig.12 A schematic diagram of a flow chart of distributing and processing target shard data according to another embodiment;

[0054] Fig.13 A schematic diagram of business data processing in another embodiment;

[0055] Fig.14 is a structural block diagram of a business data processing device in an embodiment;

[0056] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The business data processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the master node 102 communicates with the slave node 104 through the network. The data storage system can store the data that the master node 102 and the slave node 104 need to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The master node 102 obtains the task data corresponding to the pending business, and the pending business indicates the business whose response time is greater than the response time threshold; the task data is sliced ​​to obtain the target slice data, and the target slice data is sent to multiple slave nodes; the processing results returned by the multiple slave nodes are received, and the processing results are obtained by processing the received target slice data by multiple slave nodes. Among them, the master node 102 can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers. The slave node can be a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server providing cloud computing services.

[0059] In an exemplary embodiment, Figure 2 As shown, a business data processing method is provided, which is applied to Figure 1 Taking the master node server in as an example, the method includes the following steps S202 to S206. Among them:

[0060] Step S202: Obtain task data corresponding to the business to be processed.

[0061] The pending service indicates a service whose response time is greater than a response time threshold.

[0062] Optionally, before the master node obtains the task data corresponding to the pending business, it includes: calling a third party to perform a response test on the initial pending business to obtain the response time of the initial pending business; obtaining the initial pending business whose response time is greater than the response time threshold as the pending business. That is, before distributing and processing the business, the business logic of the complex pending business that takes a long time is separated from the business logic of the initial pending business, and the complex business is screened out, that is, the pending business. The third party can be a production monitoring service or a test environment stress testing service.

[0063] Optionally, the master node marks the task data corresponding to the pending business in the database with a flag, and obtains the marked data, that is, the task data corresponding to the pending business, from the database. Alternatively, the master node queries the task data corresponding to the pending business from the database through Spring Batch's JdbcCursorItemReader based on the pending business.

[0064] Optionally, the pending business can be a coupon distribution business or a points distribution business. Taking the coupon distribution as an example, the initial coupon issuance business is processed in advance before the formal coupon issuance. Through third-party testing, the time-consuming complex business processing logic is separated from the initial coupon issuance business logic, and the complex business, that is, the pending business, is screened. The task data corresponding to the pending business is marked in the database, and the marked task data in the database is obtained, that is, the task data corresponding to the pending business is obtained.

[0065] Step S204, sharding the task data to obtain target sharded data, and sending the target sharded data to multiple slave nodes.

[0066] Optionally, the master node uses Redis to remotely shard the task data according to certain business rules, obtains the target shard data, and then distributes the split target shard tasks to multiple slave nodes. Each slave node calls the downstream service to distribute and process the target shard tasks.

[0067] Optionally, continuing to use the coupon issuance business as an example, the master node uses Redis to implement Spring Batch remote sharding, and assigns the target sharding tasks obtained by splitting to multiple batch service slave nodes. Each slave node calls the coupon issuance service for distribution and processing.

[0068] Step S206, receiving processing results returned by multiple slave nodes, where the processing results are obtained by multiple slave nodes processing the received target shard data.

[0069] Optionally, the master node receives the processing results returned by multiple slave nodes, and the pending business processing is completed. By responding to the query instruction, the master node queries the processing result, and if it fails, it queries the reason for the failure, adjusts the distribution rules, and redistributes the failed target shard data again.

[0070] In the above-mentioned business data processing method, the task data corresponding to the business to be processed is obtained through the master node, and the business to be processed indicates the business whose response time is greater than the response time threshold, that is, the business that takes a long time; the task data corresponding to the business that takes a long time is sliced ​​to obtain the target slice data, and the target slice data is sent to multiple slave nodes; and the processing results returned by the multiple slave nodes are received. The master node slices the task data corresponding to the business that takes a long time, distributes it to multiple slave nodes, and allows multiple slave nodes to execute and process the task data concurrently, thereby improving the efficiency of task data processing.

[0071] Single point failure risk: In a single-node environment, if the server fails, all coupon issuance tasks in the thread pool will be affected.

[0072] Processing task data through multiple slave nodes can effectively avoid the limitation of single-node thread pool server resources, such as CPU, memory, and I / O, which rely on a single server. When the server resources reach a bottleneck, the performance of the thread pool will also be limited. Multiple slave nodes reduce the dependence on the only server resources. When the amount of tasks increases, it is necessary to deal with it by optimizing the thread pool configuration or increasing the number of threads, but the resources of a single server are limited, and the effect is not good after reaching the bottleneck. The resources of multiple slave nodes are far greater than those of a single server, thus avoiding the problem of low task processing efficiency after resources reach the bottleneck. And through multiple slave nodes, even if a slave node fails, it will not affect business processing.

[0073] In an exemplary embodiment, Figure 3 As shown, the slave node includes a first slave node; slicing the task data to obtain target slicing data includes steps S302 to S304. Among them:

[0074] Step S302, sharding the task data to obtain initial sharding data, and sending the initial sharding data to a plurality of first slave nodes; the plurality of first slave nodes are used to perform a first logic verification on the initial sharding data.

[0075] Among them, the first logic verification is the business logic verification before distribution processing, such as the blacklist and whitelist verification of the distribution object, etc.

[0076] Alternatively, if Figure 4 As shown in the figure, the master node uses Redis to remotely shard the task data according to certain business rules to obtain the initial shard data. The split initial shard data is then distributed to multiple slave nodes. Each slave node calls the downstream service to perform the first logical verification on the initial shard data. After verification, the mark status in the database is updated to mark that the logical verification has passed, for example, flag=1.

[0077] Step S304, receiving the verified initial shard data sent by the plurality of first slave nodes, and sharding the verified initial shard data to obtain target shard data.

[0078] Optionally, the master node receives the verified initial shard data sent by the plurality of first slave nodes, and remotely shards the verified initial shard data again through Redis to obtain the target shard data. Figure 5 As shown, the master node can also distribute the target shard data to the slave nodes, and each slave node calls a third-party service to distribute and process the tasks according to the target shard data.

[0079] In this embodiment, by performing business verification processing on the initial shard data in advance before executing the distribution business, task congestion caused by a too long distribution process is avoided, and the peak-shifting processing mechanism improves the task processing efficiency.

[0080] In an exemplary embodiment, the initial shard data is distributed to the first slave node, and the data returned by the first slave node is received, such as Figure 6 As shown, the initial shard data is sent to multiple first slave nodes, including step S602, sending the initial shard data to the first message queue; each first slave node is used to obtain the initial shard data from the first message queue, perform a first logical verification on the initial shard data, and send the verified initial shard data to the second message queue.

[0081] Optionally, the master node and the slave node transmit request messages and response messages through the message middleware Redis. Figure 7 As shown. The master node sends the initial shard data obtained after the first sharding to the first message queue, such as the request queue, through the master node request channel. Each first slave node is used to obtain the initial shard data from the first message queue (request queue), perform a first logical verification on the initial shard data, and send the verified initial shard data to the second message queue, such as Figure 7 The response queue in .

[0082] Receiving verified initial shard data sent by multiple first slave nodes includes: step S604, obtaining the verified initial shard data returned by each first slave node from the second message queue.

[0083] Optionally, the master node obtains the verified initial shard data returned by each first slave node from the second message queue (response queue).

[0084] Optionally, the Master partitions the steps in the batch task that require remote partitioning, and sends the partitioned multiple task messages to the RequestQueue of the Redis message middleware through the MasterRequestChannel, and listens to the ReplystQueue; multiple Slave nodes compete with each other to obtain messages from the RequestQueue, and start executing tasks after receiving the messages; after the Slave completes the task, it sends the task completion information to the ReplystQueue of the Redis message middleware; the Master node receives the task completion information from each Slave node, puts the information into the aggregator message integrator, counts whether all partition tasks have been completed, and finally determines whether the step has been successfully completed.

[0085] Optionally, taking the coupon issuance business as an example, the master node Master splits the coupon issuance data into several sets of data to be processed according to certain business rules, that is, the initial shard data, and then puts the initial shard data into the message middleware Redis. The slave node Slave listens to the message, reads the initial shard data in the request queue, and processes the initial shard data, such as the blacklist verification of the distribution object, and returns the verification processing results to the master node Master through the response queue. This part of the batch processing logic is configured with a batch processing job task separately so that it can be run in advance before the coupon is issued.

[0086] In this embodiment, by decomposing complex processing services into multiple steps, a first logical verification is performed on the initial shard data in advance before distributing the services, and the risk of task accumulation is reduced through the message queue, thereby improving efficiency.

[0087] In an exemplary embodiment, the target shard data is distributed to the second slave node, and the data returned by the second slave node is received, such as Figure 8 As shown, the slave node includes a second slave node; the target shard data is sent to multiple slave nodes, including step S802, sending the target shard data to the third message queue; each second slave node is used to obtain the target shard data from the third message queue, distribute and process according to the target shard data, and send the processing results to the fourth message queue.

[0088] Alternatively, if Figure 7 As shown. The master node sends the target shard data obtained after the second sharding to the third message queue, such as the request queue, through the master node request channel. Each second slave node is used to obtain the target shard data from the third message queue (request queue), distribute and process the target shard data, and send the processing results to the fourth message queue, such as Figure 7 The response queue in .

[0089] Receiving the processing results returned by the multiple slave nodes includes step S804 of obtaining the processing results returned by each second slave node from the fourth message queue.

[0090] Optionally, the master node obtains the distribution processing result for the target shard data returned by each second slave node from the fourth message queue (response queue).

[0091] In this embodiment, by sending the target shard data to multiple slave nodes, the multiple slave nodes are allowed to concurrently execute the distribution task according to the target shard data, and after the distribution task is completed, the processing result is returned to the master node through the queue. The distribution service can be executed concurrently, which improves efficiency.

[0092] In an exemplary embodiment, Fig. 9As shown, the business data processing method includes steps S902 to S906. Among them:

[0093] Step SS902, for each slave node, obtain the target shard data; the target shard data is determined based on the master node dividing the task data corresponding to the pending business; the pending business indicates a business whose response time is greater than the response time threshold.

[0094] Among them, the target shard data is determined based on the master node dividing the task data corresponding to the pending business; the pending business indicates a business whose response time is greater than the response time threshold.

[0095] Optionally, for each slave node, the slave node obtains the target shard data through the message middleware Redis.

[0096] Step S904, performing distribution processing according to the target shard data to obtain a processing result.

[0097] Optionally, multiple slave nodes call downstream services according to the target shard data and perform distribution processing concurrently to obtain processing results corresponding to each slave node.

[0098] Step S906, returning the processing result to the master node.

[0099] Optionally, the slave node returns the processing results of each downstream service to the master node. The master node receives the information of each slave node completing the task, puts the information into the aggregator message integrator, counts whether all partition tasks have been completed, and finally determines whether the step is successfully completed. By responding to the query instruction, the master node queries the processing result. If it fails, it queries the reason for the failure, adjusts the distribution rules, and redistributes the failed target shard data.

[0100] Optionally, the coupon issuance task is used as an example for explanation. Each slave node obtains the target shard data, that is, the data corresponding to the coupon issuance business, through the message middleware Redis. Multiple slave nodes call the downstream coupon issuance service according to the target shard data and perform coupon distribution processing concurrently to obtain the processing results corresponding to each slave node. The slave node returns the processing results of each downstream service to the master node. The master node receives the information that each slave node has completed the task, puts the information into the aggregator message integrator, counts whether all partition tasks have been completed, and finally determines whether the step has been successfully completed. By responding to the query instruction, the master node queries the distribution processing results. If the coupon distribution fails, the reason for the failure is queried, and the failed target shard data is redistributed and processed after adjusting the coupon distribution rules.

[0101] In this embodiment, for each slave node, by obtaining the target shard data; the target shard data is determined based on the master node dividing the task data corresponding to the pending business; the pending business indicates the business whose response time is greater than the response time threshold; multiple slave nodes distribute and process according to the target shard data to obtain the processing result; and return the processing result to the master node. Multiple slave nodes receive the target shard data distributed by the master node, and distribute and process the target shard data through multiple slave nodes, so that multiple slave nodes distribute and process the task data concurrently, thereby improving the efficiency of task data processing and execution.

[0102] In an exemplary embodiment, Fig.10 As shown, the slave node includes a first slave node; obtaining target shard data includes steps S1002 to S1004. Among them:

[0103] Step S1002, obtaining initial sharding data through each first slave node, where the initial sharding data is sent based on the master node.

[0104] The tasks to be processed are divided into two subtasks: verification task and distribution task. The initial shard data is the data corresponding to the verification task.

[0105] Optionally, each first slave node receives the initial shard data sent by the master node through the Redis message middleware. The Redis message middleware includes a request queue and a response queue, such as Figure 7 As shown. The master node sends the initial shard data obtained after the first sharding to the first message queue, such as the request queue, through the master node request channel. Each first slave node obtains the initial shard data from the first message queue (request queue).

[0106] Step S1004, perform a first logic verification on the initial shard data, and return the verified initial shard data to the master node; the master node is used to receive the verified initial shard data, and shard the verified initial shard data to obtain the target shard data.

[0107] Optionally, each first slave node calls the downstream service to perform the first logic verification on the initial shard data, and sends the verified initial shard data to the second message queue, such as the response queue. The master node obtains the verified initial shard data returned by each first slave node from the second message queue (response queue), puts it into the aggregator message integrator, counts whether all partition tasks have been completed, and finally determines whether the step is successfully completed. After that, the master node shards the verified initial shard data to obtain the target shard data, that is, the data corresponding to the distribution task.

[0108] In this embodiment, by decomposing complex processing services into multiple steps, before distributing the services, multiple slave nodes pre-concurrently perform a first logical verification on the initial shard data, and through the message queue, the risk of task accumulation is reduced through decoupling, thereby improving efficiency.

[0109] In an exemplary embodiment, Fig.11 As shown, the slave node includes the second slave node; distribution processing is performed according to the target shard data to obtain a processing result, including steps S1102 to S1108. Among them:

[0110] Step S1102, performing a second logical verification on the target shard data through each second slave node.

[0111] Optionally, before the slave node performs the second logical verification on the target shard data, the master node sends the target shard data obtained after the second sharding to a third message queue, such as a request queue, through the master node request channel. Each second slave node is used to obtain the target shard data from the third message queue (request queue) and obtain the redis distributed lock at the same time to avoid repeated acquisition of the target shard data under high concurrency of multiple slave nodes, such as Fig.12 shown.

[0112] Step S1104: if the second logic verification passes, extract the data records to be distributed from the target shard data.

[0113] Optionally, after successfully acquiring the distributed lock, each second slave node performs a second logic verification on the target shard data, such as a distribution deadline, a distribution upper limit, etc. If the second logic verification passes, the data record to be distributed is extracted from the target shard data.

[0114] Step S1106, obtaining the target data to be distributed through the target script according to the distribution rules and the data record to be distributed.

[0115] The target data to be distributed refers to the data distributed to each user, such as the coupons of user A or the points of user B.

[0116] Optionally, the target data to be distributed is obtained by the target script according to the pre-established distribution rules and the data records to be distributed. The distribution rules are configured on the distribution template, such as the distribution type, distribution period, distribution upper limit, etc. The target data to be distributed for each user is obtained by deducting the distribution template inventory, calculating the distribution quantity, etc. through the target script.

[0117] Alternatively, if Fig.12As shown, the target data to be distributed is calculated by the target script according to the pre-established distribution rules and the data records to be distributed, and the target script deducts the target data to be distributed from the corresponding activities. If the target data to be distributed involves multiple activities, the target data to be distributed is deducted from multiple activities together. For example, the target data to be distributed is 100, corresponding to 3 activities. If there are 20 remaining in activity 1, 30 remaining in activity 2, and 200 remaining in activity 3, then the deductions are made in the order of the activities, 20 of activity 1, 30 of activity 2, and 50 of activity 3.

[0118] Optionally, take the distribution of coupons as an example for explanation. The rules for distributing coupons are configured on the coupon template, such as coupon type, daily / weekly / monthly issuance limit, etc. After the business verification is passed, the coupon template inventory is deducted and the number and amount of coupons issued are calculated through the Lua script. Since the Lua script syntax is concise, supports object-oriented programming, has low memory usage, fast execution speed, and is easy to embed, the consistency of the calculation of each numerical value of the coupon template is guaranteed. Coupons may involve one or more subsidies and budgets. The budget amount is configured on the coupon activity. After the coupon template deducts the inventory, the Lua script is used to cyclically deduct the activity budget and calculate the used and remaining budgets. If the coupon needs a budget of 100 yuan, and involves multiple activities A, B, and C with a remaining amount of 20 yuan, 30 yuan, and 200 yuan respectively, a cyclic deduction will be performed. After deducting the budget amounts of activities A and B, 50 yuan will be deducted from C; if the budget of activity C is 40 yuan, the budget is not enough to issue a coupon of 100 yuan, and the deduction of activity C fails. The Lua script rolls back the budgets of activities A and B that have been successfully deducted, and rolls back the inventory of the coupon template.

[0119] Step S1108, distributing and processing the target data to be distributed through the target script to obtain the processing result of each second slave node.

[0120] Optionally, the second slave node calls the target script in the downstream service to distribute and deduct the target data to be distributed according to the distribution object corresponding to the target data to be distributed. After the deduction is successful and the target data to be distributed is successfully distributed, the user distribution details are updated and the redis distributed lock is released to indicate that the distribution task is completed. If the second verification fails, the target script calculation fails, or the target script deduction fails, the redis lock will be released in time to avoid the lock being invalidly occupied, thereby obtaining the distribution processing result of each second slave node.

[0121] Optionally, each second slave node sends the distribution processing result to a fourth message queue, such as a response queue. The master node receives the distribution results of each slave node from the fourth message queue, and puts the information into the aggregator message integrator, counts whether all partition tasks have been completed, and finally determines whether the step is successfully completed.

[0122] In this embodiment, the target shard data is distributed and processed concurrently by the second slave node to improve the distribution efficiency. Due to the limited computing power of Redis: Redis is usually used as a cache and a simple database, and is not originally designed to perform complex operations. Therefore, when the business logic is complex and requires complex operations, Redis alone cannot meet business needs. Therefore, since the target script is executed as a whole and will not be interrupted by other commands, the target script is used to calculate and distribute the relevant parameters of the target data to be distributed, which ensures the execution efficiency while also ensuring the consistency of the calculation.

[0123] In an exemplary embodiment, the master node uses Redis to remotely shard the task data according to certain business rules to obtain the initial shard data. The master node and the slave node transmit request messages and response messages through the message middleware Redis. The master node sends the initial shard data obtained after the first sharding to the first message queue, such as the request queue, through the master node request channel. Each first slave node obtains the initial shard data from the first message queue (request queue) and performs a first logic verification on the initial shard data, such as the distribution user blacklist and whitelist verification, etc. The user blacklist and whitelist are time-consuming and are not necessarily executed in the distribution task. Therefore, the first business logic verification is performed before the formal issuance of the coupon. After verification, the mark status in the database is updated to mark that the logic verification has passed, for example, flag=1. The initial shard data after verification is sent to the second message queue, such as the response queue. The Master master node receives the information that each Slave node has completed the task, puts the information into the aggregator message integrator, counts whether all partition tasks have been completed, and finally determines whether the step is successfully completed. This part of the batch processing logic is configured with a separate batch processing job task so that it can be run in advance before issuing coupons.

[0124] The master node performs remote sharding again on the verified initial sharding data through Redis to obtain the target sharding data. The target sharding data obtained after the second sharding is sent to the third message queue, such as the request queue, through the master node request channel. Each second slave node is used to obtain the target sharding data from the third message queue (request queue) and obtain the redis distributed lock at the same time to avoid repeated acquisition of the target sharding data under high concurrency of multiple slave nodes. Each second slave node performs a second logic verification on the target sharding data, such as the distribution period, the distribution upper limit, etc. If the second logic verification passes, the data record to be distributed is extracted from the target sharding data. The target data to be distributed is obtained through the target script according to the pre-established distribution rules and the data records to be distributed. The distribution rules are configured on the distribution template, such as the distribution type, the distribution period, the distribution upper limit, etc. The target data to be distributed for each user is obtained by deducting the distribution template inventory and calculating the distribution quantity through the target script. The second slave node calls the target script in the downstream service to distribute and deduct the target data to be distributed according to the distribution object corresponding to the target data to be distributed. After the deduction is successful and the target data to be distributed is successfully distributed, the user distribution details are updated and the redis distributed lock is released to indicate that the distribution task is completed. If the second verification fails or the target script calculation fails or the target script deduction fails, the redis lock will be released in time to avoid the lock being invalidly occupied, thereby obtaining the distribution processing result of each second slave node. Each second slave node also sends the distribution processing result to the fourth message queue, such as the response queue. The master node receives the distribution results of each slave node from the fourth message queue, and puts the information into the aggregator message integrator, counts whether all partition tasks have been completed, and finally determines whether the step is successfully completed. By responding to the query instruction, the master node queries the distribution processing result. If the coupon distribution fails, the failure reason is queried, and the coupon distribution rules are adjusted to redistribute the failed target shard data.

[0125] In this embodiment, the complex business process to be processed is decomposed into multiple steps and disassembled into multiple microservices for batch processing calls, verification and distribution, which avoids task congestion caused by the long business process to be processed, and the peak-shifting processing mechanism improves the execution efficiency of the business to be processed. Redis remote sharding multi-node parallel execution distribution processing is used; distributed service resources are fully utilized to improve the timeliness of Spring Batch processing large-scale data tasks, thereby improving the efficiency of task distribution. Calculation is performed through the target script, which solves the problem of limited Redis computing power, and the target script is executed as a whole (including both the calculation process and the distribution process), ensuring the atomicity of the execution operation and will not be interrupted by other commands, thereby ensuring the integrity of the operation and the consistency of the data.

[0126] In an exemplary embodiment, taking the distribution of coupons as an example, the initial coupon issuance business is processed in advance before the formal issuance of coupons to identify the business processes that are slow or complex in the coupon issuance business; the master node calls a third party to perform a response test on the initial pending business to obtain the response time of the initial pending business; the initial pending business with a response time greater than the response time threshold is obtained as the pending business. The master node marks the task data corresponding to the pending business in the database with a flag. Based on the pending business, the master node queries the task data corresponding to the pending business from the database through Spring Batch's JdbcCursorItemReader.

[0127] like Fig.13 As shown in the figure, the master node uses Redis to remotely shard the task data according to certain business rules to obtain the initial shard data. The master node and the slave node transmit request messages and response messages through the message middleware Redis. The master node sends the initial shard data obtained after the first sharding to the first message queue, such as the request queue, through the master node request channel. Each first slave node obtains the initial shard data from the first message queue (request queue) and performs the first logic verification on the initial shard data, such as the distribution user blacklist and whitelist verification, etc. The user blacklist and whitelist are time-consuming and are not necessarily executed in the distribution task. Therefore, the first business logic verification is performed before the formal issuance of coupons. The master node obtains the verified initial shard data returned by each first slave node from the second message queue (response queue). The Master master node receives the information that each Slave node has completed the task, puts the information into the aggregator message integrator, counts whether all partition tasks have been completed, and finally determines whether the step has been successfully completed. This part of the batch processing logic is configured with a batch processing job task separately so that it can be run in advance before issuing coupons.

[0128] The master node performs remote sharding again on the verified initial sharding data through Redis to obtain the target sharding data. The target sharding data obtained after the second sharding is sent to the third message queue, such as the request queue, through the master node request channel. Each second slave node is used to obtain the target sharding data from the third message queue (request queue) and obtain the redis distributed lock at the same time to avoid repeated acquisition of the target sharding data under high concurrency of multiple slave nodes. Each second slave node performs a second logic verification on the target sharding data, such as the distribution period, the distribution upper limit, etc. If the second logic verification passes, the data record to be distributed is extracted from the target sharding data. The Lua script is used to deduct the inventory of the coupon template and calculate the number and amount of coupons issued. Because the Lua script has concise syntax, supports object-oriented programming, has low memory usage, fast execution speed, and is easy to embed, the consistency of the calculation of each numerical value of the coupon template is guaranteed. The coupon may involve one or more subsidies and budgets. The budget amount is configured on the coupon activity. After the coupon template deducts the inventory, the Lua script is used to cyclically deduct the activity budget and calculate the used and remaining budgets. If the budget for the coupon is 100 yuan, and there are multiple activities A, B, and C with a remaining balance of 20 yuan, 30 yuan, and 200 yuan respectively, a circular deduction will be performed. After deducting the budget amounts of activities A and B, 50 yuan will be deducted from C; if the budget of activity C is 40 yuan, the budget is not enough to issue a coupon of 100 yuan, and the deduction of activity C fails. The Lua script rolls back the budgets of activities A and B that have been successfully deducted, and rolls back the inventory of the coupon template. The second slave node calls the target script in the downstream service to distribute and deduct the target data to be distributed according to the distribution object corresponding to the target data to be distributed. After the deduction is successful and the target data to be distributed is successfully distributed, the user distribution details are updated and the redis distributed lock is released to indicate that the distribution task is completed. If the second verification fails, the target script calculation fails, or the target script deduction fails, the redis lock will be released in time to prevent the lock from being invalidly occupied, thereby obtaining the distribution processing result of each second slave node. Each second slave node sends the distribution processing result to the fourth message queue, such as the response queue. The master node receives the distribution results of each slave node from the fourth message queue, and puts the information into the aggregator message integrator to count whether all partition tasks have been completed, and finally determines whether the step is successfully completed. By responding to the query instruction, the master node queries the distribution processing result. If the coupon distribution fails, the failure reason is queried, and the coupon distribution rules are adjusted to redistribute the failed target shard data.

[0129] Each second slave node sends the distribution processing result to the fourth message queue, such as the response queue. The master node receives the distribution results of each slave node from the fourth message queue, and puts the information into the aggregator message integrator, counts whether all partition tasks have been completed, and finally determines whether the step is successfully completed.

[0130] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0131] Based on the same inventive concept, the embodiment of the present application also provides a business data processing device for implementing the business data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more business data processing device embodiments provided below can refer to the limitations on the business data processing method above, and will not be repeated here.

[0132] In an exemplary embodiment, Fig.14 As shown, a service data processing device is provided, which is applied in a master node, including: a first acquisition module 1401, a slicing module 1402 and a receiving module 1403, wherein:

[0133] The first acquisition module 1401 is used to acquire task data corresponding to the pending business, where the pending business indicates a business whose response time is greater than a response time threshold.

[0134] The slicing module 1402 is used to slice the task data to obtain target slicing data, and send the target slicing data to multiple slave nodes.

[0135] The receiving module 1403 is used to receive processing results returned by multiple slave nodes, where the processing results are obtained by processing the received target shard data by the multiple slave nodes.

[0136] In an exemplary embodiment, the slave node includes a first slave node; the sharding module 1402 includes:

[0137] The first sharding unit is used to shard the task data to obtain initial sharding data, and send the initial sharding data to multiple first slave nodes; the multiple first slave nodes are used to perform a first logic verification on the initial sharding data.

[0138] The second sharding unit is used to receive the verified initial sharding data sent by the multiple first slave nodes, and to slice the verified initial sharding data to obtain target sharding data.

[0139] In an exemplary embodiment, the first sharding unit is also used to send the initial sharding data to the first message queue; each first slave node is used to obtain the initial sharding data from the first message queue, perform a first logical verification on the initial sharding data, and send the verified initial sharding data to the second message queue.

[0140] The second sharding unit is further used to obtain the verified initial sharding data returned by each first slave node from the second message queue.

[0141] In an exemplary embodiment, the slave node includes a second slave node; the sharding module 1402 is also used to send the target sharding data to the third message queue; each second slave node is used to obtain the target sharding data from the third message queue, perform distribution processing according to the target sharding data, and send the processing results to the fourth message queue.

[0142] The receiving module 1403 is further configured to obtain the processing result returned by each second slave node from the fourth message queue.

[0143] In an exemplary embodiment, a service data processing device is provided, which is applied in a slave node and includes:

[0144] The second acquisition module is used to obtain target shard data for each slave node; the target shard data is determined based on the master node dividing the task data corresponding to the pending business; the pending business indicates a business whose response time is greater than a response time threshold.

[0145] The processing module is used to distribute and process the target shard data to obtain the processing result.

[0146] The return module is used to return the processing results to the master node.

[0147] In an exemplary embodiment, the slave node includes a first slave node; and a second acquisition module includes:

[0148] The first acquisition unit is used to acquire initial shard data through each first slave node, where the initial shard data is sent based on the master node.

[0149] The first verification unit is used to perform a first logical verification on the initial shard data and return the verified initial shard data to the master node; the master node is used to receive the verified initial shard data and shard the verified initial shard data to obtain target shard data.

[0150] In an exemplary embodiment, the slave node includes a second slave node; a processing module, including:

[0151] A second verification unit, configured to perform a second logic verification on the target shard data through each second slave node;

[0152] The extraction unit is used to extract the data records to be distributed from the target shard data when the second logic verification passes.

[0153] The data determination unit is used to obtain target data to be distributed according to the distribution rules and the data records to be distributed through the target script.

[0154] The distribution processing unit is used to distribute and process the target data to be distributed through a target script to obtain a processing result of each second slave node.

[0155] Each module in the above-mentioned business data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0156] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.15 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store business task data to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a business data processing method is implemented.

[0157] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0158] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0159] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0160] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0161] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0162] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0163] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A business data processing method, characterized in that: Applied in the master node, the method includes: Acquire task data corresponding to the pending business, where the pending business indicates a business whose response time is greater than a response time threshold; Slice the task data to obtain target slice data, and send the target slice data to multiple slave nodes; Receive processing results returned by the multiple slave nodes, where the processing results are obtained by the multiple slave nodes processing the received target shard data.

2. The method according to claim 1, characterized in that: The slave node includes a first slave node; and slicing the task data to obtain target slicing data includes: Slicing the task data to obtain initial slicing data, and sending the initial slicing data to a plurality of first slave nodes; the plurality of first slave nodes are used to perform a first logic verification on the initial slicing data; Receive the verified initial shard data sent by multiple first slave nodes, and shard the verified initial shard data to obtain target shard data.

3. The method according to claim 2, characterized in that The sending the initial shard data to the plurality of first slave nodes comprises: The initial shard data is sent to the first message queue; each of the first slave nodes is used to obtain the initial shard data from the first message queue, perform a first logic verification on the initial shard data, and send the verified initial shard data to the second message queue; The receiving the verified initial shard data sent by the plurality of first slave nodes comprises: The verified initial shard data returned by each of the first slave nodes is obtained from the second message queue.

4. The method according to any one of claims 1 to 3, characterized in that The slave node includes a second slave node; sending the target shard data to multiple slave nodes includes: Send the target shard data to the third message queue; each of the second slave nodes is used to obtain the target shard data from the third message queue, perform distribution processing according to the target shard data, and send the processing result to the fourth message queue; The receiving the processing results returned by the plurality of slave nodes comprises: The processing result returned by each of the second slave nodes is obtained from the fourth message queue.

5. A business data processing method, characterized in that: Applied in a slave node, the method comprises: For each of the slave nodes, target shard data is obtained; the target shard data is determined based on the master node dividing the task data corresponding to the pending business; the pending business indicates a business whose response time is greater than a response time threshold; Perform distribution processing according to the target shard data to obtain a processing result; The processing result is returned to the master node.

6. The method according to claim 5, characterized in that The slave node includes a first slave node; and obtaining target shard data includes: Acquire initial sharding data through each of the first slave nodes, where the initial sharding data is based on the data sent by the master node; A first logical verification is performed on the initial shard data, and the verified initial shard data is returned to the master node; the master node is used to receive the verified initial shard data, and shard the verified initial shard data to obtain the target shard data.

7. The method according to claim 5, characterized in that The slave node includes a second slave node; the distribution processing is performed according to the target shard data to obtain a processing result, including: Performing a second logic verification on the target shard data through each of the second slave nodes; If the second logic verification passes, extracting the data records to be distributed from the target shard data; Obtaining target data to be distributed according to the distribution rules and the data record to be distributed through the target script; The target data to be distributed is distributed and processed through the target script to obtain the processing result of each of the second slave nodes.

8. A business data processing device, characterized in that: Applied to a master node, the device comprises: A first acquisition module is used to acquire task data corresponding to a pending business, where the pending business indicates a business whose response time is greater than a response time threshold; A slicing module, used for slicing the task data to obtain target slicing data, and sending the target slicing data to multiple slave nodes; The receiving module is used to receive the processing results returned by the multiple slave nodes, where the processing results are obtained by the multiple slave nodes processing the received target shard data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images