Batch data updating method, updating device and batch data updating system

Abstract

The application provides a batch data updating method, an updating device and a batch data updating system. The updating method comprises the following steps: a block dividing step, in which a plurality of predetermined service data are divided into a plurality of service data blocks; an initialization step, in which a transaction is started and a JVM local cache is initialized; a processing step, in which the plurality of predetermined service data in a corresponding service data block are processed based on service logic to obtain a target service block, the plurality of target service data in the target service block are stored in the JVM local cache, the plurality of target service data in the JVM local cache are updated to a target database in batches, and the transaction is closed; and a repeating step, in which the initialization step and the processing step are repeated at least once in sequence until all the service data blocks are processed, so that the problem that the cost of updating batch data in the prior art is high is solved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method, apparatus, computer-readable storage medium, and system for updating batch data. Background Technology

[0002] Currently, batch updates to databases typically use Redis as a cache and establish a mapping with the underlying database such as MySQL. By reading and writing to the Redis database, data in the Redis database is periodically updated to the MySQL database, thereby achieving batch updates to the MySQL database. This reduces the access pressure on the MySQL database and improves transaction efficiency.

[0003] However, Redis, as a solution for distributed cache consistency, plays a crucial role in ensuring cache consistency across different nodes. For parameter data shared by different nodes, storing it in a Redis database can not only improve transaction efficiency but also ensure consistency of cached data accessed by different nodes. However, for updates to data on a single node in a distributed, sharded database scenario, since this data is exclusively accessed by that single node, operations such as updating, inserting, and deleting data will not affect other nodes. Using a Redis cluster as a cache database makes it difficult to fully leverage its cache consistency advantages; the overall deployment cost of Redis databases is high, and it consumes a certain amount of network bandwidth. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, computer-readable storage medium, and system for updating batch data, in order to solve the problem of high cost of updating batch data in the prior art.

[0005] According to one aspect of the present invention, a method for updating batch data is provided, comprising: a block-splitting step, dividing multiple predetermined business data into blocks to obtain multiple business data blocks, wherein one business data block includes multiple predetermined business data; an initialization step, initiating a transaction and initializing a JVM local cache; a processing step, processing multiple predetermined business data in a corresponding business data block based on business logic to obtain a target business block, storing multiple target business data in the target business block in the JVM local cache, and batch updating the multiple target business data in the JVM local cache to a target database and closing the transaction, wherein the target business data is business data after processing the predetermined business data based on the business logic; and a repetition step, repeating the initialization step and the processing step at least once in sequence until all the business data blocks have been processed.

[0006] Optionally, based on business logic, multiple predetermined business data items in a corresponding business data block are processed to obtain a target business block, and multiple target business data items in the target business block are stored in the JVM local cache. This includes: processing each predetermined business data item in a corresponding business data block one by one based on the business logic to obtain multiple target business data items; intercepting the multiple target business data items to prevent them from being updated in the target database; constructing the target business block from the multiple target business data items; and storing the multiple target business data items in the target business block in the JVM local cache.

[0007] Optionally, intercepting multiple pieces of target business data to prevent them from being updated in the target database includes: intercepting MyBatis update operations, MyBatis insert operations, and MyBatis delete operations on multiple pieces of target business data to prevent them from being updated in the target database.

[0008] Optionally, updating multiple pieces of target business data in the JVM local cache to the target database in batch includes: at least invoking a Java database connection to update multiple pieces of target business data in the JVM local cache to the target database in batch.

[0009] Optionally, at least a Java database connection may be invoked to batch update multiple pieces of the target business data in the JVM local cache to the target database, including: invoking a batch update method, a batch insert method, and a batch delete method of the Java database connection to batch update multiple pieces of the target business data in the JVM local cache to the target database.

[0010] Optionally, multiple predetermined business data are divided into blocks to obtain multiple business data blocks, including: upon receiving a business data request, filtering business data that meets the business data request from the target database to obtain multiple predetermined business data; and dividing the multiple predetermined business data into blocks according to predetermined rules to obtain multiple business data blocks, wherein the predetermined rules are rules for dividing the multiple predetermined business data into blocks.

[0011] Optionally, after updating multiple pieces of the target business data in the JVM local cache to the target database in batches, before closing the transaction, the update method further includes: clearing the multiple pieces of the target business data in the JVM local cache.

[0012] According to another aspect of the present invention, a batch data update apparatus is also provided, comprising: a block division unit, configured to perform a block division step, dividing multiple predetermined business data into blocks to obtain multiple business data blocks, wherein one business data block includes multiple predetermined business data; an initialization unit, configured to perform an initialization step, starting a transaction and initializing a JVM local cache; a processing unit, configured to perform a processing step, processing multiple predetermined business data in a corresponding business data block based on business logic to obtain a target business block, storing multiple target business data in the target business block in the JVM local cache, batch updating multiple target business data in the JVM local cache to a target database, and closing the transaction, wherein the target business data is business data after processing the predetermined business data based on the business logic; and a repetition unit, configured to perform a repetition step, repeating the initialization step and the processing step at least once in sequence until all the business data blocks have been processed.

[0013] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes any of the batch data update methods described above.

[0014] According to another aspect of the present invention, a batch data update system is also provided, comprising: a UI interface for sending business data requests to a batch data update device, the batch data update device being communicatively connected to the UI interface and for executing any of the batch data update methods described herein; and a target database communicating with the batch data update device.

[0015] In this embodiment of the invention, the batch data update method first divides multiple predetermined business data into blocks to obtain multiple business data blocks; then, a transaction is started, and the JVM local cache is initialized; subsequently, based on business logic, multiple predetermined business data in a corresponding business data block are processed to obtain a target business block, and multiple target business data in the target business block are stored in the JVM local cache, and the multiple target business data in the JVM local cache are batch updated to the target database; finally, the initialization step and the processing step are executed at least once in sequence until all business data blocks have been processed. Compared with the prior art, which uses Redis caching to batch update multiple target business data after processing based on business logic, this solution is based on JVM local caching and transactions, that is, restricting the JVM local cache to transactions and batch updating multiple target business data in the JVM local cache. This can avoid the problem of data loss in the JVM local cache due to the difficulty in persisting the JVM local cache. Meanwhile, since JVM local caching is lightweight and efficient, and does not require complex deployment or high deployment costs, nor does it consume network bandwidth, the method for updating batch data in this application is simple and low-cost, thus solving the problem of high cost for updating batch data in the prior art. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 A flowchart illustrating a batch data update method according to an embodiment of this application is shown;

[0018] Figure 2 A flowchart illustrating a batch data update method according to another embodiment of this application is shown;

[0019] Figure 3 A schematic diagram of a batch data updating apparatus according to an embodiment of this application is shown;

[0020] Figure 4 A flowchart illustrating a batch data update scheme according to a specific embodiment of this application is shown. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0025] A transaction is a series of database operations that are either all executed or none executed; it is an indivisible unit of work.

[0026] JVM native cache: The cache built into the JVM (Java Virtual Machine), which stores data in memory throughout the entire program's lifecycle.

[0027] As mentioned in the background section, updating batch data in the prior art is costly. To address this issue, in a typical embodiment of this application, a batch data updating method, updating apparatus, computer-readable storage medium, and batch data updating system are provided.

[0028] According to an embodiment of this application, a method for updating batch data is provided.

[0029] Figure 1 This is a flowchart of a batch data update method according to an embodiment of this application. For example... Figure 1 As shown, the update method includes the following steps:

[0030] Step S101, block segmentation step, the multiple predetermined business data are divided into blocks to obtain multiple business data blocks, and one of the above business data blocks includes multiple of the above predetermined business data.

[0031] Step S102, initialization step, start a transaction and initialize the JVM local cache;

[0032] Step S103, processing steps: Based on business logic, multiple pieces of the predetermined business data in a corresponding business data block are processed to obtain a target business block, and multiple pieces of target business data in the target business block are stored in the JVM local cache. The multiple pieces of target business data in the JVM local cache are then batch updated to the target database and the transaction is closed. The target business data is the business data after processing the predetermined business data based on the business logic.

[0033] Step S104: Repeat the steps, repeating the above initialization steps and the above processing steps at least once, until all the above business data blocks have been processed.

[0034] In the aforementioned batch data update method, firstly, multiple pre-defined business data items are divided into blocks to obtain multiple business data blocks; then, a transaction is started, and the JVM local cache is initialized; subsequently, based on business logic, multiple pre-defined business data items in a corresponding business data block are processed to obtain a target business block, and multiple target business data items in the target business block are stored in the JVM local cache, and the multiple target business data items in the JVM local cache are batch-updated to the target database; finally, the initialization and processing steps are executed at least once in sequence until all business data blocks have been processed. Compared with the existing technology that uses Redis caching to batch update multiple target business data items processed based on business logic, this solution is based on JVM local caching and transactions, that is, restricting the JVM local cache to transactions and batch updating multiple target business data items in the JVM local cache. This can avoid the problem of data loss in the JVM local cache due to the difficulty in persisting the JVM local cache. Meanwhile, since JVM local caching is lightweight and efficient, and does not require complex deployment or high deployment costs, nor does it consume network bandwidth, the method for updating batch data in this application is simple and low-cost, thus solving the problem of high cost for updating batch data in the prior art.

[0035] Specifically, in the above embodiments, multiple predetermined business data are divided into blocks to obtain multiple business data blocks. Each business data block is restricted to one transaction; that is, after processing multiple predetermined business data in a business data block, the corresponding target business data obtained after processing is stored in the JVM local cache until all predetermined business data in a business data block has been processed, resulting in a target data block. Then, multiple target business data in a target data block are batch-updated to the target database. If any predetermined business data in a business data block fails to process, all predetermined business data in that business data block are rolled back. When the next task starts, the multiple predetermined business data in the previously failed business data block can be restarted and processed again. Furthermore, the batch update method of this application can also support exception skipping. For example, if the processing of one predetermined business data in a business data block fails, the entire business data block and the entire thread will fail. To avoid this situation, the batch update method of this application can also support skipping specified exceptions to ensure the success rate of data processing.

[0036] Specifically, in the above embodiments, processing multiple predetermined business data items in a corresponding business data block based on business logic can be equivalent to processing multiple predetermined business data items in a single business data block based on business logic. That is, one transaction can correspond to one business data block. However, in practical applications, if the business logic corresponding to two business data blocks is related and multiple predetermined business data items in the business data blocks need to be processed simultaneously, then one transaction can correspond to two or more business data blocks. In other words, this application does not limit the number of business data blocks corresponding to one transaction.

[0037] Specifically, the batch data update method of this application also supports concurrent processing, that is, multiple threads can be used for batch processing. For example, each thread can process one or more pre-defined business data blocks. In practical applications, the number of threads used in the concurrent batch data update process can be set according to the actual business needs. Simultaneously, the size of each business data block and the number of pre-defined business data blocks can also be set according to the actual business needs. For example, if 100,000 pre-defined business data blocks need to be updated in batches, 10 threads can be used for concurrent processing. Each thread can process 10,000 data blocks, so the size of the pre-defined business data blocks in each business data block can be set to 200, meaning each thread processes 50 business data blocks. However, this is not limited to the scenarios listed above; the specific number of business data blocks that each thread can process and the size of the pre-defined business data blocks in each business data block are configurable.

[0038] Specifically, any suitable method in the prior art can be used to batch update multiple pieces of target business data from the JVM local cache to the target database. This application does not limit the specific method used to batch update multiple pieces of target business data from the JVM local cache to the target database.

[0039] In one specific embodiment of this application, the target database is a database that stores multiple predetermined business data entries. The target database can be a MySQL database. However, the target database is not limited to a MySQL database; it can also be any other suitable database of the prior art.

[0040] In practical applications, such as Figure 2 As shown, upon receiving a user's request to read data via the UI (i.e., the business data request mentioned later), multiple pre-defined business data entries can be read from the target database. These read entries are then divided into blocks (blocking is necessary when the amount of pre-defined business data is large; otherwise, it's not mandatory). This results in multiple business data blocks. At this point, a transaction needs to be started, and the JVM local cache initialized. Then, based on business logic, the pre-defined business data entries in one or more corresponding business data blocks are processed (i.e., data processing) to obtain the corresponding target business block. Simultaneously, during the processing of the pre-defined business data entries, MyBatis write-back operations (i.e., MyBatis update, insert, and delete operations) need to be intercepted synchronously, and the target business data in the target business block is stored in the JVM local cache. Finally, the data is written, which involves batch updating multiple pieces of target business data from the JVM local cache to the target database, and then closing the transaction.

[0041] Specifically, in the above embodiments, a business data block may contain only one piece of predetermined business data. That is, in this application, there is no limit to the amount of predetermined business data in a business data block. Specifically, the above business logic can be the logic for processing the predetermined business data. For example, in a banking scenario, it is necessary to calculate interest (settle interest) on a batch of customer deposit data. Then, the above business logic can be the business logic for calculating interest.

[0042] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0043] In one embodiment of this application, based on business logic, multiple predetermined business data items in a corresponding business data block are processed to obtain a target business block, and the multiple target business data items in the target business block are stored in the JVM local cache. This includes: processing each predetermined business data item in a corresponding business data block one by one based on the business logic to obtain multiple target business data items; intercepting the multiple target business data items to prevent them from being updated in the target database; constructing the target business block from the multiple target business data items; and storing the multiple target business data items in the target business block in the JVM local cache. In practical applications, during the process of processing each predetermined business data item in a corresponding business data block based on business logic, the obtained target business data is constantly updated in the target database. This means that the target database needs to be continuously connected to and accessed, resulting in significant connection and access pressure on the target database. Therefore, in order to reduce the pressure on the target database's connection and access, the target business data after processing based on business logic is intercepted. This prevents the target business data after processing based on business logic from being updated in the target database, thus reducing the pressure on the target database's connection and access.

[0044] Specifically, in order to further and more simply prevent target business data from being updated to the target database, in another embodiment of this application, multiple pieces of the aforementioned target business data are intercepted to prevent them from being updated to the target database. This includes intercepting MyBatis update operations, MyBatis insert operations, and MyBatis delete operations on multiple pieces of the aforementioned target business data to prevent them from being updated to the target database.

[0045] In practical applications, it's not limited to using MyBatis write-back operations (i.e., MyBatis update, insert, and delete operations) to update multiple target business data entries to the target database according to their corresponding target business blocks. Any feasible write-back method available in existing technologies can also be used to update the corresponding target business blocks to the target database. Furthermore, if the corresponding write-back operation changes, the corresponding interception operation must also be modified accordingly. The goal is simply to avoid continuous connections and accesses to the target database due to modifications to multiple pre-defined business data entries. In other words, after all pre-defined business data entries within the same business data block have been processed, a single connection to the target database can be established, thus achieving batch updates of multiple target business data entries within a single target business block to the target database.

[0046] In another embodiment of this application, the batch update of the target business data in the JVM local cache to the target database includes: at least invoking a Java Database Connectivity (JDBC) to batch update multiple pieces of the target business data in the JVM local cache to the target database. In this embodiment, by using a Java Database Connectivity to batch update the target business data in the JVM local cache to the target database, a relatively simple and convenient method is achieved to batch update multiple pieces of business data in a target business block to the target database.

[0047] Of course, in practical applications, it's not limited to using a Java database connection to batch update target business data from the JVM local cache to the target database. Any feasible existing method of connecting to the target database can also be used to achieve this.

[0048] To further simplify and facilitate the batch updating of multiple target business data entries in a target business block to the target database, i.e., to further simplify and facilitate the batch writing of multiple target business data entries back to the target database, in another embodiment of this application, at least a Java database connection is invoked to batch update multiple target business data entries in the JVM local cache to the target database. This includes: invoking the batch update method, batch insert method, and batch delete method of the Java database connection to batch update the target business data entries in the JVM local cache to the target database.

[0049] Specifically, in practical applications, if the method of connecting to the target database changes, the specific methods of writing (batch update method, batch insert method, and batch delete method) to the target database also need to be changed accordingly.

[0050] In one embodiment of this application, multiple predetermined business data are divided into blocks to obtain multiple business data blocks. This includes: upon receiving a business data request, filtering business data that meets the business data request from the target database to obtain multiple predetermined business data; and dividing the multiple predetermined business data into blocks according to predetermined rules, where the predetermined rules are the rules for dividing the multiple predetermined business data into blocks. In this embodiment, dividing the multiple predetermined business data into blocks according to predetermined rules simplifies the logic of dividing the multiple predetermined business data, reduces computational load, and makes the batch data update method of this application more lightweight.

[0051] Specifically, the aforementioned pre-defined rules can be divided according to the number of pre-defined business data entries. Of course, the aforementioned pre-defined rules can also be other rules used for segmentation, such as date, amount, etc. That is, this application does not limit the specific content of the aforementioned pre-defined rules, which can be flexibly adjusted according to actual circumstances.

[0052] In one specific embodiment of this application, if a total of 1,000 pieces of pre-defined business data are read, they can be divided into 5 blocks in units of 200 pieces, thus obtaining 5 business data blocks.

[0053] Specifically, in the above embodiments, the process of filtering data that satisfies the business data request from the target database when a business data request is received may further include: filtering predetermined business data that satisfies the business data request according to a business field and an idempotent field. Specifically, the business field may be a field related to the business data request, and the idempotent field may be a field used to mark the state of the predetermined business data or business data block. In a specific embodiment of this application, the predetermined business data is processed based on the corresponding business logic, and the idempotent field of the corresponding predetermined business data may be modified (e.g., modified to "processed"). Of course, in actual applications, the idempotent field can be processed more flexibly. Specifically, it can be designed according to specific business needs. For predetermined business data that only needs to initiate a batch data processing task once, it can be designed as unprocessed or processed; for predetermined business data that needs to be batch processed periodically (e.g., interest settlement every three months in a bank system), it can be designed as the next interest settlement date; when the current date equals the next interest settlement date, a batch task is initiated, and after successful processing, the next interest settlement date is modified to the date three months after the current date, and then the next batch task is waited for again, etc. In other words, this application does not restrict the specific design of the aforementioned idempotent fields, which can be flexibly set according to the characteristics of the corresponding business requirements, etc.

[0054] In practical applications, if the processing of scheduled business data in a business data block fails, a rollback is performed. In this case, the reason for the processing failure can also be displayed on the front-end UI page. After fixing the reason for the processing failure (i.e., ensuring that the scheduled business data that failed to process can be processed successfully next time), the next batch processing task can be initiated. When the next batch task is initiated, filtering is still performed using idempotency fields and business fields. For example, unprocessed scheduled business data with idempotency fields is filtered out until all scheduled business data is processed successfully.

[0055] In another embodiment of this application, after updating the target business data in the JVM local cache to the target database in batches, before closing the transaction, the update method further includes: clearing multiple pieces of the target business data in the JVM local cache, thus ensuring that the occupied JVM local cache can be released in a timely manner and ensuring the high robustness of the overall system.

[0056] This application also provides a batch data updating apparatus. It should be noted that the batch data updating apparatus of this application can be used to execute the batch data updating method provided in this application. The batch data updating apparatus provided in this application is described below.

[0057] Figure 3 This is a schematic diagram of the structure of a batch data updating device according to an embodiment of this application. Figure 3 As shown, the updating device includes:

[0058] Blocking unit 10 is used to perform the blocking step, to divide multiple predetermined business data into blocks, and to obtain multiple business data blocks. One of the aforementioned business data blocks includes multiple of the aforementioned predetermined business data.

[0059] Initialization unit 20 is used to perform initialization steps, start a transaction, and initialize the JVM local cache;

[0060] The processing unit 30 is used to execute processing steps, based on business logic, to process multiple pieces of the predetermined business data in a corresponding business data block to obtain a target business block, and to store multiple pieces of target business data in the target business block in the JVM local cache, and to batch update the multiple pieces of the target business data in the JVM local cache to the target database and close the transaction. The target business data is the business data after processing the predetermined business data based on the business logic.

[0061] The repeating unit 40 is used to perform repeating steps, repeating the above initialization steps and the above processing steps at least once, until all the above business data blocks have been processed.

[0062] In the aforementioned batch data update device, the block division unit is used to divide multiple predetermined business data into blocks to obtain multiple business data blocks; the initialization unit is used to start a transaction and initialize the JVM local cache; the processing unit is used to process multiple predetermined business data in the corresponding business data block based on business logic to obtain target business blocks, and to store multiple target business data in the target business block in the JVM local cache, and to batch update the multiple target business data in the JVM local cache to the target database; the repeating unit is used to execute the initialization step and the processing step at least once in sequence until all business data blocks have been processed. Compared with the prior art, which uses Redis caching to batch update multiple target business data after processing based on business logic, this solution is based on JVM local cache and transactions, that is, restricting JVM local cache to transactions, and batch updating multiple target business data in JVM local cache. This can avoid the problem of data loss in JVM local cache due to the difficulty of JVM local cache persistence. Meanwhile, since JVM local caching is lightweight and efficient, and does not require complex deployment or high deployment costs, nor does it consume network bandwidth, the method for updating batch data in this application is simple and low-cost, thus solving the problem of high cost for updating batch data in the prior art.

[0063] Specifically, in the above embodiments, multiple predetermined business data are divided into blocks to obtain multiple business data blocks. Each business data block is restricted to one transaction; that is, after processing multiple predetermined business data in a business data block, the corresponding target business data obtained after processing is stored in the JVM local cache until all predetermined business data in a business data block has been processed, resulting in a target data block. Then, multiple target business data in a target data block are batch-updated to the target database. If any predetermined business data in a business data block fails to process, all predetermined business data in that business data block are rolled back. When the next task starts, the multiple predetermined business data in the previously failed business data block can be restarted and processed again. Furthermore, the batch update device of this application can also support exception skipping. For example, if the processing of one predetermined business data in a business data block fails, the entire business data block and the entire thread will fail. To avoid this situation, the batch update device of this application can also support skipping specified exceptions to ensure the success rate of data processing.

[0064] Specifically, in the above embodiments, processing multiple predetermined business data items in a corresponding business data block based on business logic can be equivalent to processing multiple predetermined business data items in a single business data block based on business logic. That is, one transaction can correspond to one business data block. However, in practical applications, if the business logic corresponding to two business data blocks is related and multiple predetermined business data items in the business data blocks need to be processed simultaneously, then one transaction can correspond to two or more business data blocks. In other words, this application does not limit the number of business data blocks corresponding to a single transaction.

[0065] Specifically, the batch data update device of this application also supports concurrent processing, that is, multiple threads can be enabled for batch processing. For example, each thread can process one or more blocks of pre-defined business data. In practical applications, the number of threads enabled during concurrent batch data updates can be set according to the actual business needs. Simultaneously, the size of each business data block and the number of pre-defined business data items can also be set according to the actual business needs. For example, if 100,000 pre-defined business data items need to be updated in batches, 10 threads can be enabled for concurrent processing. Each thread can process 10,000 data items, so the size of the pre-defined business data items in each business data block can be set to 200, meaning each thread processes 50 business data blocks. However, this is not limited to the scenarios listed above; the number of business data blocks that each thread can process and the size of the pre-defined business data items in each business data block are both configurable.

[0066] Specifically, any suitable method in the prior art can be used to batch update multiple pieces of target business data from the JVM local cache to the target database. This application does not limit the specific method used to batch update multiple pieces of target business data from the JVM local cache to the target database.

[0067] In one specific embodiment of this application, the target database is a database that stores multiple predetermined business data entries. The target database can be a MySQL database. However, the target database is not limited to a MySQL database; it can also be any other suitable database of the prior art.

[0068] In practical applications, such as Figure 2As shown, upon receiving a user's request to read data via the UI (i.e., the business data request mentioned later), multiple pre-defined business data entries can be read from the target database. These read entries are then divided into blocks (blocking is necessary when the amount of pre-defined business data is large; otherwise, it's not mandatory). This results in multiple business data blocks. At this point, a transaction needs to be started, and the JVM local cache initialized. Then, based on business logic, the pre-defined business data entries in one or more corresponding business data blocks are processed (i.e., data processing) to obtain the corresponding target business block. Simultaneously, during the processing of the pre-defined business data entries, MyBatis write-back operations (i.e., MyBatis update, insert, and delete operations) need to be intercepted synchronously, and the target business data in the target business block is stored in the JVM local cache. Finally, the data is written, which involves batch updating multiple pieces of target business data from the JVM local cache to the target database, and then closing the transaction.

[0069] Specifically, in the above embodiments, a business data block may contain only one piece of predetermined business data. That is, in this application, there is no limit to the amount of predetermined business data in a business data block. Specifically, the above business logic can be the logic for processing the predetermined business data. For example, in a banking scenario, it is necessary to calculate interest (settle interest) on a batch of customer deposit data. Then, the above business logic can be the business logic for calculating interest.

[0070] In one embodiment of this application, the processing unit includes a processing module, an interception module, and a combination module. The processing module processes multiple predetermined business data items in a corresponding business data block one by one based on the business logic to obtain multiple target business data items. The interception module intercepts the multiple target business data items to prevent them from being updated in the target database. The combination module constructs a target business block from the multiple target business data items and stores the target business data in the target business block in the JVM local cache. In practical applications, during the process of processing multiple predetermined business data items in a corresponding business data block one by one based on business logic, the obtained target business data is constantly updated in the target database. This means that the target database needs to be continuously connected to and accessed, resulting in significant connection and access pressure. Therefore, to reduce the connection and access pressure on the target database, the target business data processed based on business logic is intercepted, preventing it from being updated in the target database and thus reducing the connection and access pressure on the target database.

[0071] Specifically, in order to further and more simply prevent the target business data from being updated to the target database, in another embodiment of this application, the interception module includes an interception submodule, which is used to intercept MyBatis update operations, MyBatis insert operations and MyBatis delete operations of multiple pieces of the target business data, so as to prevent multiple pieces of the target business data from being updated to the target database.

[0072] In practical applications, it's not limited to using MyBatis write-back operations (i.e., MyBatis update, insert, and delete operations) to update multiple target business data entries to the target database according to their corresponding target business blocks. Any feasible write-back method available in existing technologies can also be used to update the corresponding target business blocks to the target database. Furthermore, if the corresponding write-back operation changes, the corresponding interception operation must also be modified accordingly. The goal is simply to avoid continuous connections and accesses to the target database due to modifications to multiple pre-defined business data entries. In other words, after all pre-defined business data entries within the same business data block have been processed, a single connection to the target database can be established, thus achieving batch updates of multiple target business data entries within a single target business block to the target database.

[0073] In another embodiment of this application, the processing unit includes an update module, configured to at least invoke a Java Database Connectivity (JDBC) to batch update multiple pieces of the target business data in the JVM local cache to the target database. In this embodiment, by using a Java Database Connectivity to batch update the target business data in the JVM local cache to the target database, a relatively simple and convenient method is achieved for batch updating multiple pieces of business data from a target business block to the target database.

[0074] Of course, in practical applications, it's not limited to using a Java database connection to batch update target business data from the JVM local cache to the target database. Any feasible existing method of connecting to the target database can also be used to achieve this.

[0075] To further simplify and facilitate the batch updating of multiple target business data entries in a target business block to the target database, i.e., to further simplify and facilitate the batch writing of multiple target business data entries back to the target database, in another embodiment of this application, the update module includes an update submodule, which is used to call the batch update method, batch insert method, and batch delete method of the Java database connection to batch update the target business data in the JVM local cache to the target database.

[0076] Specifically, in practical applications, if the method of connecting to the target database changes, the specific methods of writing (batch update method, batch insert method, and batch delete method) to the target database also need to be changed accordingly.

[0077] In one embodiment of this application, the segmentation unit includes a receiving module and a segmentation module. The receiving module is used to, upon receiving a service data request, filter service data that meets the service data request from the target database to obtain multiple pieces of predetermined service data. The segmentation module is used to segment the multiple pieces of predetermined service data into blocks according to predetermined rules to obtain multiple service data blocks. The predetermined rules are rules for segmenting the multiple pieces of predetermined service data. In this embodiment, segmenting the multiple pieces of predetermined service data according to predetermined rules simplifies the logic of segmenting the multiple pieces of predetermined service data, reduces the computational load, and makes the batch data update device of this application more lightweight.

[0078] Specifically, the aforementioned pre-defined rules can be divided according to the number of pre-defined business data entries. Of course, the aforementioned pre-defined rules can also be other rules used for segmentation, such as date, amount, etc. That is, this application does not limit the specific content of the aforementioned pre-defined rules, which can be flexibly adjusted according to actual circumstances.

[0079] In one specific embodiment of this application, if a total of 1,000 pieces of pre-defined business data are read, they can be divided into 5 blocks in units of 200 pieces, thus obtaining 5 business data blocks.

[0080] Specifically, in the above embodiments, the process of filtering data that satisfies the business data request from the target database when a business data request is received may further include: filtering predetermined business data that satisfies the business data request according to a business field and an idempotent field. Specifically, the business field may be a field related to the business data request, and the idempotent field may be a field used to mark the state of the predetermined business data or business data block. In a specific embodiment of this application, the predetermined business data is processed based on the corresponding business logic, and the idempotent field of the corresponding predetermined business data may be modified (e.g., modified to "processed"). Of course, in actual applications, the idempotent field can be processed more flexibly. Specifically, it can be designed according to specific business needs. For predetermined business data that only needs to initiate a batch data processing task once, it can be designed as unprocessed or processed; for predetermined business data that needs to be batch processed periodically (e.g., interest settlement every three months in a bank system), it can be designed as the next interest settlement date; when the current date equals the next interest settlement date, a batch task is initiated, and after successful processing, the next interest settlement date is modified to the date three months after the current date, and then the next batch task is waited for again, etc. In other words, this application does not restrict the specific design of the aforementioned idempotent fields, which can be flexibly set according to the characteristics of the corresponding business requirements, etc.

[0081] In practical applications, if the processing of scheduled business data in a business data block fails, a rollback is performed. In this case, the reason for the processing failure can also be displayed on the front-end UI page. After fixing the reason for the processing failure (i.e., ensuring that the scheduled business data that failed to process can be processed successfully next time), the next batch processing task can be initiated. When the next batch task is initiated, filtering is still performed using idempotency fields and business fields. For example, unprocessed scheduled business data with idempotency fields is filtered out until all scheduled business data is processed successfully.

[0082] In another embodiment of this application, the update device further includes a clearing unit, which is used to clear multiple pieces of the target business data in the JVM local cache after the target business data in the JVM local cache is updated in batch to the target database and before the transaction is closed. This ensures that the occupied JVM local cache can be released in a timely manner, thus ensuring the high robustness of the overall system.

[0083] The aforementioned batch data update device includes a processor and a memory. The aforementioned block unit, initialization unit, processing unit, and repeating unit are all stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0084] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and adjusting kernel parameters can address the high cost of updating batch data in existing technologies.

[0085] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0086] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the above-described method for updating batch data.

[0087] This invention provides a processor for running a program, wherein the program executes the batch data update method during runtime.

[0088] In a typical embodiment of this application, a batch data update system is also provided. The update system includes a user interface, an update device, and a target database. The user interface is used to send business data requests to the batch data update device; the batch data update device is communicatively connected to the user interface and is used to execute any of the batch data update methods described above; the target database communicates with the batch data update device.

[0089] The aforementioned batch data update system includes a UI interface, an update device, and a target database. The UI interface sends business data requests to the batch data update device; the batch data update device is communicatively connected to the UI interface and executes any of the aforementioned batch data update methods; the target database communicates with the batch data update device. In the aforementioned update method, firstly, multiple predetermined business data items are divided into blocks to obtain multiple business data blocks; then, a transaction is initiated, and the JVM local cache is initialized; subsequently, based on business logic, multiple predetermined business data items in a corresponding business data block are processed to obtain a target business block, and multiple target business data items in the target business block are stored in the JVM local cache, and the multiple target business data items in the JVM local cache are batch-updated to the target database; finally, the initialization step and the processing step are executed at least once in sequence until all business data blocks have been processed. Compared to existing technologies that use Redis caching for batch updates of multiple target business data entries based on business logic processing, this solution utilizes JVM local caching and transactions. Specifically, it confines the JVM local cache to transactions, allowing for batch updates of multiple target business data entries within the JVM local cache. This avoids data loss issues caused by the difficulty in persisting JVM local cache entries. Furthermore, because JVM local caching is lightweight and efficient, requiring no complex deployment or high deployment costs, and does not consume network bandwidth, this ensures that the batch data update method in this application is simple and low-cost, thus solving the problem of high costs associated with batch data updates in existing technologies.

[0090] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0091] Step S101, block segmentation step, the multiple predetermined business data are divided into blocks to obtain multiple business data blocks, and one of the above business data blocks includes multiple of the above predetermined business data.

[0092] Step S102, initialization step, start a transaction and initialize the JVM local cache;

[0093] Step S103, processing steps: Based on business logic, multiple pieces of the predetermined business data in a corresponding business data block are processed to obtain a target business block, and multiple pieces of target business data in the target business block are stored in the JVM local cache. The multiple pieces of target business data in the JVM local cache are then batch updated to the target database and the transaction is closed. The target business data is the business data after processing the predetermined business data based on the business logic.

[0094] Step S104: Repeat the steps, repeating the above initialization steps and the above processing steps at least once, until all the above business data blocks have been processed.

[0095] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0096] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0097] Step S101, block segmentation step, the multiple predetermined business data are divided into blocks to obtain multiple business data blocks, and one of the above business data blocks includes multiple of the above predetermined business data.

[0098] Step S102, initialization step, start a transaction and initialize the JVM local cache;

[0099] Step S103, processing steps: Based on business logic, multiple pieces of the predetermined business data in a corresponding business data block are processed to obtain a target business block, and multiple pieces of target business data in the target business block are stored in the JVM local cache. The multiple pieces of target business data in the JVM local cache are then batch updated to the target database and the transaction is closed. The target business data is the business data after processing the predetermined business data based on the business logic.

[0100] Step S104: Repeat the steps, repeating the above initialization steps and the above processing steps at least once, until all the above business data blocks have been processed.

[0101] To enable those skilled in the art to better understand the technical solution of this application, the technical solution and technical effects of this application will be described below in conjunction with specific embodiments.

[0102] Example

[0103] In a typical embodiment of this application, a batch data update scheme is also provided, as detailed in [link to specific implementation]. Figure 4As shown. In this embodiment, the specific steps of the batch data update scheme are as follows:

[0104] Step 1: Configuring the batch update function. In this embodiment, the option to enable or disable the batch update function can also be set. For example, this can be configured in the system's configuration center, or the default method can be used. The default method includes enabling or disabling by default. Of course, the default method can also be configured according to business functions. For example, in a commercial bank system, batch transactions can be enabled and online transactions can be disabled, etc.

[0105] Step Two: Obtain multiple pre-defined business data entries through batch querying (i.e., when receiving business data sent from the UI interface). Then, divide the multiple pre-defined business data entries into blocks according to pre-defined rules to obtain one or more business data blocks.

[0106] Step 3: Start a transaction, initialize the JVM local cache, and obtain the switch to enable or disable batch commit functionality;

[0107] Step 4: Based on business logic, process each of the multiple pre-defined business data entries in the corresponding business data block to obtain multiple target business data entries (i.e., ... Figure 4 The processed data is shown. Furthermore, the batch update method of this application can also perform parallel operations (this embodiment only uses one business data block for explanation);

[0108] Step 5: If the batch update function is enabled, intercept each MyBatis write-back operation (i.e., intercept MyBatis update, insert, and delete operations) for the target business data obtained based on business logic processing. Store the multiple target business data entries obtained after business logic processing in the JVM local cache (i.e.,...). Figure 4 (Store in the JVM local cache), and return a success message;

[0109] Step Six: After writing the data is complete. If the batch update function is enabled, retrieve the multiple target business data records to be updated from the JVM local cache. Then, depending on the write operation type, call the JDBC batch update method, batch insert method, and batch delete method respectively to batch update the multiple target business data records based on the business logic processing to the target database.

[0110] Step 7: After updating multiple pieces of target business data in a target business block, clear the JVM local cache;

[0111] Step 8: Commit the transaction, i.e. close the transaction;

[0112] Step 9: After the processing of multiple scheduled business data in a business data block is completed, repeat steps 3 to 8 above, that is, process multiple scheduled business data in the next business data block, until the processing of multiple scheduled business data in all business data blocks is completed.

[0113] The above-described batch data update scheme can achieve the following results:

[0114] 1) By intercepting MyBatis database write operations (i.e., MyBatis update, insert, and delete operations), and after a transaction, using JDBC to perform batch update, insert, and delete methods to update multiple records of target business data in the target business block to the target database. This reduces the number of operations required to connect to the target database for a single business data block to only 1, reducing the number of interactions with the target database and lowering the access pressure on the target database.

[0115] 2) JVM local caching is lightweight and efficient. Furthermore, batch updates are performed based on the JVM local cache, and a success message is returned immediately after intercepting Mybatis database write operations, improving system response rate.

[0116] 3) By limiting the lifecycle of the JVM local cache to the lifecycle of a transaction, the impact of cache invalidation due to server failure is avoided on the application, thus ensuring the robustness of the system.

[0117] 4) In banking systems, if end-of-day tasks involve numerous operations on single-node databases, and in a sharded database environment, single-table data updates do not require validation of sharding fields or optimistic locking version numbers. JDBC-based batch update methods can avoid Mybatis's validation of technical fields during single-table operations, improving transaction efficiency while ensuring data integrity.

[0118] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units described above can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0120] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0123] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0124] 1) In the batch data update method of this application, firstly, multiple pre-defined business data are divided into blocks to obtain multiple business data blocks; then, a transaction is started, and the JVM local cache is initialized; subsequently, based on business logic, multiple pre-defined business data in a corresponding business data block are processed to obtain a target business block, and multiple target business data in the target business block are stored in the JVM local cache, and the multiple target business data in the JVM local cache are batch updated to the target database; finally, the initialization step and the processing step are executed at least once in sequence until all business data blocks have been processed. Compared with the prior art, which uses Redis caching to batch update multiple target business data after processing based on business logic, this solution is based on JVM local caching and transactions, that is, restricting the JVM local cache to transactions and batch updating multiple target business data in the JVM local cache. This can avoid the problem of data loss in the JVM local cache due to the difficulty in persisting the JVM local cache. Meanwhile, since JVM local caching is lightweight and efficient, and does not require complex deployment or high deployment costs, nor does it consume network bandwidth, the method for updating batch data in this application is simple and low-cost, thus solving the problem of high cost for updating batch data in the prior art.

[0125] 2) In the batch data update apparatus of this application, the block division unit is used to divide multiple predetermined business data into blocks to obtain multiple business data blocks; the initialization unit is used to start a transaction and initialize the JVM local cache; the processing unit is used to process multiple predetermined business data in the corresponding business data block based on business logic to obtain target business blocks, and to store multiple target business data in the target business block in the JVM local cache, and to batch update the multiple target business data in the JVM local cache to the target database; the repeating unit is used to execute the initialization step and the processing step at least once in sequence until all business data blocks have been processed. Compared with the prior art, which uses Redis caching to batch update multiple target business data after processing based on business logic, this solution is based on JVM local cache and transactions, that is, restricting JVM local cache to transactions, and batch updating multiple target business data in JVM local cache. This can avoid the problem of data loss in JVM local cache due to the difficulty of JVM local cache persistence. Meanwhile, since JVM local caching is lightweight and efficient, and does not require complex deployment or high deployment costs, nor does it consume network bandwidth, the method for updating batch data in this application is simple and low-cost, thus solving the problem of high cost for updating batch data in the prior art.

[0126] 3) The batch data update system of this application includes a UI interface, an update device, and a target database. The UI interface is used to send business data requests to the batch data update device; the batch data update device is communicatively connected to the UI interface and is used to execute any of the batch data update methods described above; the target database communicates with the batch data update device. In the update method described above, firstly, multiple predetermined business data items are divided into blocks to obtain multiple business data blocks; then, a transaction is started, and the JVM local cache is initialized; subsequently, based on business logic, multiple predetermined business data items in a corresponding business data block are processed to obtain a target business block, and multiple target business data items in the target business block are stored in the JVM local cache, and the multiple target business data items in the JVM local cache are batch-updated to the target database; finally, the initialization step and the processing step are executed at least once in sequence until all business data blocks have been processed. Compared to existing technologies that use Redis caching for batch updates of multiple target business data entries based on business logic processing, this solution utilizes JVM local caching and transactions. Specifically, it confines the JVM local cache to transactions, allowing for batch updates of multiple target business data entries within the JVM local cache. This avoids data loss issues caused by the difficulty in persisting JVM local cache entries. Furthermore, because JVM local caching is lightweight and efficient, requiring no complex deployment or high deployment costs, and does not consume network bandwidth, this ensures that the batch data update method in this application is simple and low-cost, thus solving the problem of high costs associated with batch data updates in existing technologies.

[0127] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for updating batch data, characterized in that, include: The segmentation step involves dividing multiple pre-defined business data into multiple business data blocks, with each business data block comprising multiple pre-defined business data. The initialization steps include starting a transaction and initializing the JVM local cache. The processing steps are as follows: based on business logic, multiple predetermined business data in a corresponding business data block are processed to obtain a target business block, and multiple target business data in the target business block are stored in the JVM local cache. The multiple target business data in the JVM local cache are then batch updated to the target database, and the transaction is closed. The target business data is the business data after processing the predetermined business data based on the business logic. Repeat the steps, repeating the initialization step and the processing step at least once, until all the business data blocks have been processed; Based on business logic, multiple predetermined business data items in a corresponding business data block are processed to obtain a target business block, and multiple target business data items in the target business block are stored in the JVM local cache. This includes: processing each predetermined business data item in a corresponding business data block one by one based on the business logic to obtain multiple target business data items; intercepting the multiple target business data items to prevent them from being updated in the target database; constructing the target business block from the multiple target business data items; and storing the multiple target business data items in the target business block in the JVM local cache. Intercepting multiple pieces of target business data to prevent them from being updated in the target database includes: intercepting MyBatis update operations, MyBatis insert operations, and MyBatis delete operations on multiple pieces of target business data to prevent them from being updated in the target database.

2. The updating method according to claim 1, characterized in that, Batch update multiple records of the target business data in the JVM local cache to the target database, including: At least a Java database connection is invoked to batch update multiple records of the target business data in the JVM local cache to the target database.

3. The updating method according to claim 2, characterized in that, At least a Java database connection is invoked to batch update multiple records of the target business data in the JVM local cache to the target database, including: The batch update, batch insert, and batch delete methods of the Java database connection are invoked to update multiple records of the target business data in the JVM local cache to the target database in batches.

4. The update method according to any one of claims 1 to 3, characterized in that, Multiple scheduled business data entries are divided into blocks to obtain multiple business data blocks, including: Upon receiving a business data request, the system filters business data that meets the business data request from the target database to obtain multiple pieces of the predetermined business data. According to a predetermined rule, the multiple predetermined business data are divided into blocks to obtain multiple business data blocks. The predetermined rule is the rule for dividing the multiple predetermined business data into blocks.

5. The update method according to any one of claims 1 to 3, characterized in that, After batch updating multiple pieces of the target business data from the JVM local cache to the target database, and before closing the transaction, the update method further includes: The target business data entries in the JVM local cache are cleared.

6. A batch data updating device, characterized in that, include: A segmentation unit is used to perform a segmentation step, segmenting multiple predetermined business data into multiple business data blocks, wherein one business data block includes multiple predetermined business data. The initialization unit is used to perform initialization steps, start a transaction, and initialize the JVM local cache; The processing unit is used to execute processing steps, based on business logic, to process multiple predetermined business data in a corresponding business data block to obtain a target business block, and to store multiple target business data in the target business block in the JVM local cache, and to batch update the multiple target business data in the JVM local cache to the target database and close the transaction, wherein the target business data is the business data after processing the predetermined business data based on the business logic; A repeating unit is used to perform repeating steps, repeating the initialization step and the processing step at least once in sequence until all the business data blocks have been processed. The processing unit includes a processing module, an interception module, and a combination module. The processing module processes multiple predetermined business data items in a corresponding business data block one by one based on the business logic to obtain multiple target business data items. The interception module intercepts the multiple target business data items to prevent them from being updated in the target database. The combination module constructs the target business block from the multiple target business data items and stores the multiple target business data items in the target business block in the JVM local cache. The interception module includes an interception submodule for intercepting MyBatis update operations, MyBatis insert operations, and MyBatis delete operations on multiple pieces of the target business data, so as to prevent multiple pieces of the target business data from being updated in the target database.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program executes the batch data update method according to any one of claims 1 to 5.

8. A batch data update system, characterized in that, include: The UI interface is used to send business data requests to the batch data update device. The batch data updating device is communicatively connected to the UI interface and is used to execute the batch data updating method according to any one of claims 1 to 5. The target database communicates with the batch data update device.

Images