A data sharding and parallel data processing method and device for reinsurance business

Abstract

The present invention discloses a data slicing and parallel data processing method and device for reinsurance business, the method comprising: receiving reinsurance business data, slicing the reinsurance business data and storing it in a slicing interface table of a database; while successfully storing the reinsurance business data in the slicing interface table, pushing the reinsurance business data to a matching message queue; monitoring the matching message queue, matching the reinsurance business data in the matching message queue with the business contract, and storing the matching result in the slicing interface table according to the organization code included in the corresponding reinsurance business data; extracting the reinsurance business data from the slicing interface table in parallel according to the organization code as data to be processed in batches, storing the data to be processed in batches in a batch message queue; monitoring the batch message queue, batch processing the data to be processed in the batch message queue, and obtaining the processing result. The present invention solves the problem of how to process massive business data and improve the timeliness of system processing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data slicing and parallel data processing method and device for reinsurance business. Background Art

[0002] The insurance system of some companies is a national centralized system. The number of insurance policies and claims received every day is in the millions, totaling in the tens of millions. The amount of data is huge. The data received on the same day needs to be processed on the same day such as reinsurance or cession, proportional reinsurance, and apportionment, and the account data must be summarized and sent to the financial system for accounting. Therefore, the timeliness and accuracy of data processing are required to be high, and it is difficult for the system to ensure that data processing is completed on time every day. Summary of the invention

[0003] The present invention provides a data slicing and parallel data processing method and device for reinsurance business, which solves the problem of how to process massive business data and improve the timeliness of system processing.

[0004] A data sharding and parallel data processing method for reinsurance business, comprising:

[0005] Receiving reinsurance business data, slicing the reinsurance business data and storing the slicing interface table of the database;

[0006] When the reinsurance business data is successfully stored in the shard interface table, the reinsurance business data is pushed to a matching message queue;

[0007] Monitoring the matching message queue, matching the reinsurance business data in the matching message queue with the business contract, and storing the matching result in the slicing interface table according to the institution code included in the corresponding reinsurance business data;

[0008] Extract the reinsurance business data from the slice interface table in parallel according to the organization code as data to be processed in batches, and store the data to be processed in batch message queues;

[0009] The batch message queue is monitored, and batch processing is performed on the data to be batch processed in the batch message queue to obtain a processing result.

[0010] In an embodiment of the present invention, the sharding of the reinsurance business data specifically includes: performing a first sharding process according to a first attribute of the reinsurance business data to obtain a first layer of sharded data corresponding to each first attribute; determining the data volume of each first layer of sharded data, and judging whether the data volume of each first layer of sharded data exceeds a preset threshold; if so, performing a second sharding process on the first layer of sharded data whose data volume exceeds the preset threshold according to its second attribute.

[0011] In one embodiment of the present invention, the reinsurance business data is extracted in parallel from the shard interface table according to the organization code as data to be batch processed, and the data to be batch processed is stored in a batch message queue, which specifically includes: configuring the correspondence between each batch node and the shard interface table and the execution parameters of each batch node in the database according to the organization code; when the batch program is started, the batch node starts a thread pool according to the execution parameters; each thread in the thread pool extracts data from the shard interface table according to the execution parameters and stores it in the batch message queue.

[0012] In one embodiment of the present invention, the execution parameters include the number of master shard interface tables in the shard interface table, the time interval for extracting data from the database, and the number of data items extracted from the shard interface table each time.

[0013] In one embodiment of the present invention, the shard interface table includes a primary shard interface table and a backup shard interface table; the method also includes: determining whether the weight strategy is enabled, and if so, only obtaining data from the primary shard interface table with a high weight configuration; if not, polling the primary shard interface table and the backup shard interface table to obtain data.

[0014] In one embodiment of the present invention, the method further includes: when determining that the weight strategy is enabled, determining whether the amount of data obtained from the primary shard interface table is less than a preset threshold, and if so, polling the primary shard interface table and the backup shard interface table to obtain data.

[0015] In an embodiment of the present invention, the slicing the reinsurance business data and storing them in a slicing interface table of a database specifically includes: slicing the reinsurance business data to obtain multi-layer slicing data, and acquiring an organization code corresponding to each piece of slicing data; generating a slicing interface table with the organization code in a database according to the organization code; matching the organization code included in the reinsurance business data with the organization code in the slicing interface table; and storing the reinsurance business data in a slicing interface table having the same organization code as the reinsurance business data.

[0016] A data sharding and parallel data processing device for reinsurance business, comprising:

[0017] The interface data entry module is used to receive reinsurance business data, and store the reinsurance business data in a sharding interface table of a database after sharding; and when the reinsurance business data is successfully stored in the sharding interface table, the reinsurance business data is pushed to a matching message queue;

[0018] A contract matching module, used for monitoring the matching message queue, matching the reinsurance business data in the matching message queue with the business contract, and storing the matching result in the slicing interface table according to the institution code included in the corresponding reinsurance business data;

[0019] A batch scheduling module is used to extract the reinsurance business data from the slice interface table in parallel according to the institution code as data to be processed in batches, and store the data to be processed in batches in a batch message queue;

[0020] The batch processing module is used to monitor the batch processing message queue, batch process the data to be batch processed in the batch processing message queue, and obtain the processing result.

[0021] A data sharding and parallel data processing device for reinsurance business, comprising:

[0022] at least one processor; and,

[0023] A memory connected to the at least one processor via a bus; wherein,

[0024] The memory stores instructions executable by the at least one processor, and the instructions are executed to implement:

[0025] Receiving reinsurance business data, slicing the reinsurance business data and storing the slicing interface table of the database;

[0026] When the reinsurance business data is successfully stored in the shard interface table, the reinsurance business data is pushed to a matching message queue;

[0027] Monitoring the matching message queue, matching the reinsurance business data in the matching message queue with the business contract, and storing the matching result in the slicing interface table according to the institution code included in the corresponding reinsurance business data;

[0028] Extract the reinsurance business data from the slice interface table in parallel according to the organization code as data to be processed in batches, and store the data to be processed in batch message queues;

[0029] The batch message queue is monitored, and batch processing is performed on the data to be batch processed in the batch message queue to obtain a processing result.

[0030] A non-volatile storage medium stores computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the following steps:

[0031] Receiving reinsurance business data, slicing the reinsurance business data and storing the slicing interface table of the database;

[0032] When the reinsurance business data is successfully stored in the shard interface table, the reinsurance business data is pushed to a matching message queue;

[0033] Monitoring the matching message queue, matching the reinsurance business data in the matching message queue with the business contract, and storing the matching result in the slicing interface table according to the institution code included in the corresponding reinsurance business data;

[0034] Extract the reinsurance business data from the slice interface table in parallel according to the organization code as data to be processed in batches, and store the data to be processed in batch message queues;

[0035] The batch message queue is monitored, and batch processing is performed on the data to be batch processed in the batch message queue to obtain a processing result.

[0036] The present invention provides a data sharding and parallel data processing method and device for reinsurance business, which at least includes the following beneficial effects: by using a sharding strategy, massive data can be processed efficiently, ensuring the timeliness and accuracy of data processing; by adding a weighting strategy, the throughput of each batch processing machine is balanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0038] Figure 1 A schematic diagram of the steps of a data sharding and parallel data processing method for reinsurance business provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a data sharding and parallel data processing flow for reinsurance business provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a data sharding and parallel data processing device for reinsurance business provided by an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of a data sharding and parallel data processing device for reinsurance business provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be described clearly and completely in conjunction with the specific embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in the present invention should be the usual meanings understood by people with ordinary skills in the technical field to which the present invention belongs. The words "one", "a", "a", "the" and the like involved in the present invention do not indicate a quantitative limitation and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in the present invention are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in the present invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0044] The present invention proposes a data slicing and parallel data processing method and device for reinsurance business, which will be described in detail below.

[0045] Figure 1 A schematic diagram of a method for data sharding and parallel data processing for reinsurance business provided by an embodiment of the present invention may include the following steps:

[0046] S110: receiving reinsurance business data, slicing the reinsurance business data and storing them in the slicing interface table of the database; while successfully storing the reinsurance business data in the slicing interface table, pushing the reinsurance business data to the matching message queue.

[0047] Specifically, Figure 2 As shown in the figure, the reinsurance business data comes from the data of the external underwriting, claim settlement system and other business systems, and the types include data of policies / batch / settled / pending, etc. The interface data cluster and the business system transmit data through the Http servlet communication method. After receiving the reinsurance business data from the business system, the interface data cluster sends it to the database DB, and stores it in the database after slicing the reinsurance business data. At the same time as the storage is successful, the data is pushed to the matching message queue.

[0048] In one embodiment of the present invention, the reinsurance business data is subjected to sharding processing, specifically including: performing a first sharding processing according to a first attribute of the reinsurance business data to obtain a first layer of sharding data corresponding to each first attribute; determining the data volume of each first layer of sharding data, and judging whether the data volume of each first layer of sharding data exceeds a preset threshold; if so, performing a second sharding processing on the first layer of sharding data whose data volume exceeds the preset threshold according to its second attribute.

[0049] Specifically, before being stored in the sharding interface table in the database, the reinsurance business data needs to be sharded according to the sharding strategy to improve data processing efficiency. For example, according to the source attributes (province, city) of the reinsurance business data, the reinsurance business data is sharded for the first time according to the provincial institutions. Each copy after the first sharding corresponds to an institution code and a corresponding batch running node. For provinces with a large amount of data, the second sharding is performed according to the municipal institutions to prepare for subsequent parallel data processing.

[0050] For example, the sharding strategy is shown in Table 1:

[0051]

[0052]

[0053]

[0054] Table 1

[0055] For example, the sharding strategy can be to evenly shard the reinsurance business data. After the first sharding, each sharded data has a corresponding institution code and a corresponding batch running node, and can continue to be evenly sharded for the second time.

[0056] In one embodiment of the present invention, the reinsurance business data is sliced ​​and then stored in a slice interface table of a database, which specifically includes: slicing the reinsurance business data to obtain multi-layer slice data, and obtaining an organization code corresponding to each slice data; generating a slice interface table with the organization code in the database according to the organization code; matching the organization code included in the reinsurance business data with the organization code in the slice interface table; and storing the reinsurance business data in the slice interface table having the same organization code as the reinsurance business data.

[0057] Specifically, before storing the reinsurance business data in the database, it is necessary to open a shard interface table in the database. After the reinsurance business data is sharded, the organization code corresponding to each shard data is obtained, and a shard interface table with corresponding space is opened in the database according to the organization code. After matching the organization code of each piece of shard data with the organization code of the shard interface table, the shards are stored in the corresponding shard interface table.

[0058] S120: Monitor the matching message queue, match the reinsurance business data in the matching message queue with the business contract, and store the matching result into the sharding interface table according to the institution code included in the corresponding reinsurance business data.

[0059] Specifically, the contract matching cluster monitors the matching message queue, obtains data from it and consumes it, matches the reinsurance business data with the contract or rules of the business, and stores the matching results in the corresponding sharding interface table according to the sharded institution code.

[0060] S130: Extract reinsurance business data from the shard interface table in parallel according to the organization code as data to be processed in batches, and store the data to be processed in the batch message queue.

[0061] In one embodiment of the present invention, reinsurance business data is extracted in parallel from the shard interface table according to the organization code as data to be processed in batches, and the data to be processed in batches is stored in a batch message queue, specifically including: configuring the correspondence between each batch node and the shard interface table and the execution parameters of each batch node in the database according to the organization code; when the batch program is started, the batch node starts a thread pool according to the execution parameters; each thread in the thread pool extracts data from the shard interface table according to the execution parameters and stores it in the batch message queue.

[0062] In one embodiment of the present invention, the execution parameters include the number of master shard interface tables in the shard interface table, the time interval for extracting data from the database, and the number of data items extracted from the shard interface table each time.

[0063] In one embodiment of the present invention, the shard interface table includes a primary shard interface table and a backup shard interface table; determine whether the weight strategy is enabled, if so, only retrieve data from the primary shard interface table with a high weight configuration; if not, poll the primary shard interface table and the backup shard interface table to retrieve data.

[0064] In one embodiment of the present invention, when determining that the weight strategy is enabled, it is determined whether the amount of data obtained from the primary shard interface table is less than a preset threshold. If so, the primary shard interface table and the backup shard interface table are polled to obtain data.

[0065] Specifically, the correspondence between each batch running node and the shard interface table is configured in the database. For example, under any sharding strategy, batch running node 1 corresponds to six shards: p_3_1, p_3_2, p_3_3, p_3_4, p_3_5, and p_3_6 (where p_3_6 is a backup shard interface table and the rest are primary shard interface tables). In each batch running node, a timed thread pool will be started when the batch running program is started. The maximum number of threads in the thread pool is determined according to the number of primary shard interface tables in the shard interface table corresponding to the batch running node. For example, if the shard interface table includes 3 primary shard interface tables, the number of threads started is 3; if the shard interface table includes 5 primary shard interface tables and 1 backup shard interface table, the number of threads started is 5. Taking the above configuration as an example, the batch running node will generate 5 threads (t1~t5) to process the data of p_3_1~p_3_5 shards respectively. Each thread extracts X=1000 data from the database shard interface table at each interval N=1 second and puts them into the batch running message queue. The parameters M, N, and X can be configured according to actual needs. The default values ​​are M=5, N=1, and X=1000.

[0066] The weight policy in the batch running node is turned off by default. Take thread t1 as an example. When it is turned off, it will poll p_3_1 and p_3_6. If the weight configuration is turned on, the polling of p_3_1 and p_3_6 will be enabled only when the amount of data X obtained by p_3_1 for 60 consecutive times is less than 10. Otherwise, only p_3_1 will be accessed to balance the throughput of each batch processing machine.

[0067] S140: monitor the batch message queue, batch process the data to be batch processed in the batch message queue, and obtain the processing result.

[0068] Specifically, the batch running cluster monitors the batch running message queue, extracts the data in the batch running message queue for batch processing, and obtains the reinsurance result.

[0069] The above is a data sharding and parallel data processing method for reinsurance business provided by an embodiment of the present invention. Based on the same inventive idea, an embodiment of the present invention also provides a corresponding data sharding and parallel data processing device for reinsurance business, such as Figure 3 shown.

[0070] The interface data input module 301 is used to receive the reinsurance business data, and store the reinsurance business data into the shard interface table of the database after sharding; when the reinsurance business data is successfully stored in the shard interface table, the reinsurance business data is pushed to the matching message queue; the contract matching module 302 is used to monitor the matching message queue, match the reinsurance business data in the matching message queue with the business contract, and store the matching result in the shard interface table according to the organization code included in the corresponding reinsurance business data; the batch scheduling module 303 is used to extract the reinsurance business data from the shard interface table in parallel according to the organization code as the data to be batch processed, and store the data to be batch processed in the batch message queue; the batch module 304 is used to monitor the batch message queue, batch process the data to be batch processed in the batch message queue, and obtain the processing result.

[0071] The embodiment of the present invention also provides a corresponding data slicing and parallel data processing device for reinsurance business, such as Figure 4 As shown, including:

[0072] At least one processor 410, a communication interface 420, a memory 430, and a communication bus 440; wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440; the processor 410 may call the logic instructions stored in the memory 430, so that at least one processor 410 executes:

[0073] Receive the reinsurance business data, store the reinsurance business data in the sharding interface table of the database after sharding; push the reinsurance business data to the matching message queue while successfully storing the reinsurance business data in the sharding interface table; monitor the matching message queue, match the reinsurance business data in the matching message queue with the business contract, and store the matching result in the sharding interface table according to the institution code included in the corresponding reinsurance business data; extract the reinsurance business data from the sharding interface table in parallel according to the institution code as the data to be batch processed, and store the data to be batch processed in the batch message queue; monitor the batch message queue, batch process the data to be batch processed in the batch message queue, and obtain the processing result.

[0074] Based on the same idea, some embodiments of the present invention also provide a medium corresponding to the above method.

[0075] Some embodiments of the present invention provide a storage medium storing computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the following steps:

[0076] Receive the reinsurance business data, store the reinsurance business data in the sharding interface table of the database after sharding; push the reinsurance business data to the matching message queue while successfully storing the reinsurance business data in the sharding interface table; monitor the matching message queue, match the reinsurance business data in the matching message queue with the business contract, and store the matching result in the sharding interface table according to the institution code included in the corresponding reinsurance business data; extract the reinsurance business data from the sharding interface table in parallel according to the institution code as the data to be batch processed, and store the data to be batch processed in the batch message queue; monitor the batch message queue, batch process the data to be batch processed in the batch message queue, and obtain the processing result.

[0077] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0078] The devices and media provided in the embodiments of the present invention correspond one-to-one to the methods, and therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0079] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or method. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the presence of other identical elements in the process, method, commodity or method including the elements.

[0080] The above are only embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed in the present invention.

Claims

1. A data sharding and parallel data processing method for reinsurance business, characterized in that: include: Receiving reinsurance business data, slicing the reinsurance business data and storing the slicing interface table of the database; When the reinsurance business data is successfully stored in the shard interface table, the reinsurance business data is pushed to a matching message queue; Monitoring the matching message queue, matching the reinsurance business data in the matching message queue with the business contract, and storing the matching result in the slicing interface table according to the institution code included in the corresponding reinsurance business data; Extract the reinsurance business data from the shard interface table in parallel according to the organization code as data to be processed in batches, and store the data to be processed in batches in a batch message queue, including: configuring the corresponding relationship between each batch node and the shard interface table and the execution parameters of each batch node in the database according to the organization code; when the batch program is started, the batch node starts a thread pool according to the execution parameters; each thread in the thread pool extracts data from the shard interface table according to the execution parameters and stores them in the batch message queue; The batch message queue is monitored, and batch processing is performed on the data to be batch processed in the batch message queue to obtain a processing result.

2. The method according to claim 1, characterized in that The slicing of the reinsurance business data specifically includes: Performing a first sharding process according to the first attribute of the reinsurance business data to obtain first-layer sharding data corresponding to each first attribute; Determine the data size of each of the first-layer shard data, and judge whether the data size of each of the first-layer shard data exceeds a preset threshold; If so, the first-layer fragmented data whose data volume exceeds the preset threshold is fragmented again according to its second attribute.

3. The method according to claim 1, characterized in that The execution parameters include the number of master shard interface tables in the shard interface table, the time interval for extracting data from the database, and the number of data items extracted from the shard interface table each time.

4. The method according to claim 1, characterized in that: The shard interface table includes a primary shard interface table and a backup shard interface table; The method further comprises: Determine whether the weight strategy is enabled. If so, only retrieve data from the primary shard interface table with a high weight configuration; If not, poll the primary shard interface table and the backup shard interface table to obtain data.

5. The method according to claim 4, characterized in that The method further comprises: When it is determined that the weight strategy is enabled, it is determined whether the amount of data obtained from the primary shard interface table is less than a preset threshold. If so, the primary shard interface table and the backup shard interface table are polled to obtain data.

6. The method according to claim 1, characterized in that The step of slicing the reinsurance business data and storing it in a slicing interface table of a database specifically includes: Slice the reinsurance business data to obtain multi-layer sliced ​​data, and obtain the institution code corresponding to each sliced ​​data; According to the organization code, a shard interface table having the organization code is generated in a database; Matching the institution code included in the reinsurance business data with the institution code in the slice interface table; The reinsurance business data is stored in a slice interface table having the same institution code as the reinsurance business data.

7. A data sharding and parallel data processing device for reinsurance business, characterized in that: include: The interface data input module is used to receive reinsurance business data, and store the reinsurance business data into a sharding interface table of the database after sharding; When the reinsurance business data is successfully stored in the shard interface table, the reinsurance business data is pushed to a matching message queue; A contract matching module, used for monitoring the matching message queue, matching the reinsurance business data in the matching message queue with the business contract, and storing the matching result in the slicing interface table according to the institution code included in the corresponding reinsurance business data; The batch scheduling module is used to extract the reinsurance business data from the shard interface table in parallel according to the organization code as the data to be processed in batches, and store the data to be processed in batches in the batch message queue, including: configuring the corresponding relationship between each batch node and the shard interface table and the execution parameters of each batch node in the database according to the organization code; when the batch program is started, the batch node starts a thread pool according to the execution parameters; each thread in the thread pool extracts data from the shard interface table according to the execution parameters and stores it in the batch message queue; The batch processing module is used to monitor the batch processing message queue, batch process the data to be batch processed in the batch processing message queue, and obtain the processing result.

8. A data sharding and parallel data processing device for reinsurance business, characterized in that: include: at least one processor; as well as, A memory connected to the at least one processor via a bus; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed to implement: Receiving reinsurance business data, slicing the reinsurance business data and storing the slicing interface table of the database; When the reinsurance business data is successfully stored in the shard interface table, the reinsurance business data is pushed to a matching message queue; Monitoring the matching message queue, matching the reinsurance business data in the matching message queue with the business contract, and storing the matching result in the slicing interface table according to the institution code included in the corresponding reinsurance business data; Extract the reinsurance business data from the shard interface table in parallel according to the organization code as data to be processed in batches, and store the data to be processed in batches in a batch message queue, including: configuring the corresponding relationship between each batch node and the shard interface table and the execution parameters of each batch node in the database according to the organization code; when the batch program is started, the batch node starts a thread pool according to the execution parameters; each thread in the thread pool extracts data from the shard interface table according to the execution parameters and stores them in the batch message queue; The batch message queue is monitored, and batch processing is performed on the data to be batch processed in the batch message queue to obtain a processing result.

9. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions are executed by a processor to implement the following steps: Receiving reinsurance business data, slicing the reinsurance business data and storing the slicing interface table of the database; When the reinsurance business data is successfully stored in the shard interface table, the reinsurance business data is pushed to a matching message queue; Monitoring the matching message queue, matching the reinsurance business data in the matching message queue with the business contract, and storing the matching result in the slicing interface table according to the institution code included in the corresponding reinsurance business data; Extract the reinsurance business data from the shard interface table in parallel according to the organization code as data to be processed in batches, and store the data to be processed in batches in a batch message queue, including: configuring the corresponding relationship between each batch node and the shard interface table and the execution parameters of each batch node in the database according to the organization code; when the batch program is started, the batch node starts a thread pool according to the execution parameters; each thread in the thread pool extracts data from the shard interface table according to the execution parameters and stores them in the batch message queue; The batch message queue is monitored, and batch processing is performed on the data to be batch processed in the batch message queue to obtain a processing result.

Images