Data synchronization method and apparatus, storage medium, and electronic device
By pre-configuring the maximum number of pagination pages and the minimum number of pages to determine the pagination information set, and creating an asynchronous thread for data synchronization, the problem of low efficiency in batch data synchronization is solved, and more efficient data synchronization is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 吉林亿联银行股份有限公司
- Filing Date
- 2023-06-28
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies suffer from low data synchronization efficiency when synchronizing batches of data.
By pre-configuring the maximum number of pagination pages corresponding to the target business data and the minimum quantity of business data in each page, the pagination information set is determined, and an asynchronous thread is created based on this to achieve data pagination synchronization.
It reduces data synchronization time, improves data synchronization efficiency, and avoids the problem of excessive system pressure.
Smart Images
Figure CN116719889B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of financial technology, and more specifically, to a data synchronization method, apparatus, storage medium, and electronic device. Background Technology
[0002] With the rapid development of computer network technology, more and more enterprises are using networks to collect, store, and share data. To better achieve data sharing, data synchronization is crucial for enterprises. Currently, enterprises typically synchronize data by querying data from a database and then synchronizing all the data into system memory at once. However, as enterprise data increases, synchronizing all data into system memory at once not only easily leads to excessive system load and high memory consumption, but also results in excessively long synchronization times and low data synchronization efficiency.
[0003] There is currently no effective solution to the above problems. Summary of the Invention
[0004] This application provides a data synchronization method, apparatus, storage medium, and electronic device to at least solve the technical problem of low data synchronization efficiency in the prior art when synchronizing batch data.
[0005] According to one aspect of this application, a data synchronization method is provided, comprising: receiving a data synchronization instruction, wherein the data synchronization instruction carries a first parameter value and a second parameter value corresponding to target business data to be synchronized, the first parameter value being used to characterize the maximum number of pages corresponding to the target business data when performing data paging processing on the target business data, the data paging processing being used to split the target business data into at least one page of business data, and the second parameter value being used to characterize the minimum number of business data in each page of business data; determining a paging information set corresponding to the target business data based on the first parameter value and the second parameter value, wherein the paging information set includes at least a data range corresponding to the at least one page of business data and the actual number of business data in the at least one page of business data; creating at least one asynchronous thread based on the data range corresponding to the at least one page of business data; obtaining the actual number of business data from a target database corresponding to a business system through the asynchronous thread, and synchronizing the actual number of business data to a preset storage area.
[0006] Further, determining the pagination information set corresponding to the target business data based on the first parameter value and the second parameter value includes: obtaining the sequence number corresponding to each of the M business data in the first business data table to obtain M sequence numbers, wherein the first business data table is a business data table in the target database, the first business data table is used to store the target business data, and the sequence number corresponding to each business data is used to characterize the order of the business data in the M business data, where M is a positive integer; generating a target data table based on the M sequence numbers; counting the total amount of data in the target data table to obtain the target quantity; and determining the pagination information set corresponding to the target business data based on the target data table, the target quantity, the first parameter value, and the second parameter value.
[0007] Furthermore, before obtaining the sequence number corresponding to each of the M business data items in the first business data table to obtain the M sequence numbers, the method further includes: receiving a data query instruction, wherein the data query instruction carries unique identification information of the first business data table; determining the first business data table from N business data tables in the target database based on the unique identification information, wherein N is a positive integer.
[0008] Further, generating a target data table based on the M serial numbers includes: sorting the M serial numbers to obtain a sorting result; and generating the target data table based on the sorting result.
[0009] Further, determining the pagination information set corresponding to the target business data based on the target data table, the target quantity, the first parameter value, and the second parameter value includes: calculating the ratio of the target quantity to the first parameter value to obtain a first value; determining a second value based on the first value and the second parameter value, wherein the second value is used to characterize the maximum number of business data actually allocated to each page of business data; calculating the ratio of the second value to the target quantity to obtain a third value, wherein the third value is used to characterize the actual number of pages corresponding to the target business data; and determining the pagination information set corresponding to the target business data based on the target data table, the second value, and the third value.
[0010] Further, determining the pagination information set corresponding to the target business data based on the target data table, the second value, and the third value includes: performing the data pagination processing on the serial number data in the target data table based on the second value and the third value to obtain at least one page of serial number data; obtaining the data range corresponding to the at least one page of serial number data and the actual number of serial number data in the at least one page of serial number data; and determining the pagination information set based on the data range corresponding to the at least one page of serial number data and the actual number of serial number data in the at least one page of serial number data.
[0011] Further, determining the pagination information set based on the data range corresponding to the at least one page of serial number data and the actual number of serial number data in the at least one page of serial number data includes: taking the data range corresponding to the at least one page of serial number data as the data range corresponding to the business data to obtain the data range corresponding to the at least one page of business data; taking the actual number of serial number data in the at least one page as the actual number of business data to obtain the actual number of business data in the at least one page of business data; and determining the pagination information set based on the data range corresponding to the at least one page of business data and the actual number of business data in the at least one page of business data.
[0012] According to another aspect of this application, a data synchronization apparatus is also provided, comprising: a receiving module, configured to receive a data synchronization instruction, wherein the data synchronization instruction carries a first parameter value and a second parameter value corresponding to target business data to be synchronized, the first parameter value being used to characterize the maximum number of pages corresponding to the target business data when performing data paging processing on the target business data, the data paging processing being used to split the target business data into at least one page of business data, and the second parameter value being used to characterize the minimum number of business data in each page of business data; a determining module, configured to determine a paging information set corresponding to the target business data based on the first parameter value and the second parameter value, wherein the paging information set includes at least a data range corresponding to the at least one page of business data and the actual number of business data in the at least one page of business data; a creating module, configured to create at least one asynchronous thread based on the data range corresponding to the at least one page of business data; and a synchronization module, configured to obtain the actual number of business data from a target database corresponding to the business system through the asynchronous thread, and synchronize the actual number of business data to a preset storage area.
[0013] According to another aspect of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to execute the above-described data synchronization method at runtime.
[0014] According to another aspect of this application, an electronic device is also provided, the electronic device including one or more processors; a memory for storing one or more programs, which, when executed by one or more processors, cause one or more processors to perform the program, wherein the program is configured to execute the above-described data synchronization method during runtime.
[0015] In this application, a data synchronization instruction is first received. This instruction carries a first parameter value and a second parameter value corresponding to the target business data to be synchronized. The first parameter value represents the maximum number of pages corresponding to the target business data during data paging processing, whereby the data paging processing splits the target business data into at least one page of business data. The second parameter value represents the minimum number of business data items in each page. Then, a paging information set corresponding to the target business data is determined based on the first and second parameter values. This paging information set includes at least the data range corresponding to the at least one page of business data and the actual number of business data items in that page. Next, at least one asynchronous thread is created based on the data range corresponding to the at least one page of business data. Finally, the actual number of business data items is retrieved from the target database corresponding to the business system through the asynchronous thread and synchronized to a preset storage area.
[0016] In the above process, the pagination information set is determined by the maximum number of pages corresponding to the target business data (i.e., the first parameter value) and the minimum number of business data in each page (i.e., the second parameter value) in the pre-configured target business data. At least one asynchronous thread is created based on the pagination information set, and the actual number of business data is synchronized to the preset storage area through the asynchronous thread. This realizes the pagination synchronization of the target business data through the asynchronous thread, reduces the data synchronization time, and thus achieves the technical effect of improving the data synchronization efficiency. This solves the technical problem of low data synchronization efficiency in the existing technology when synchronizing batch data. Attached Figure Description
[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0018] Figure 1 This is a flowchart of an optional data synchronization method according to an embodiment of this application;
[0019] Figure 2 This is a schematic diagram of an optional data synchronization device according to an embodiment of this application;
[0020] Figure 3 This is a schematic diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation
[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0023] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
[0024] Example 1
[0025] According to an embodiment of this application, an optional data synchronization method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0026] Figure 1 This is a flowchart of an optional data synchronization method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:
[0027] Step S101: Receive a data synchronization instruction, wherein the data synchronization instruction carries a first parameter value and a second parameter value corresponding to the target business data to be synchronized.
[0028] In one optional embodiment, a data synchronization system can serve as the execution entity for the data synchronization method in this application. For ease of description, the data synchronization system will be referred to simply as the system. The data synchronization system includes a data loading module and a data processing module. The data loading module is used to determine the paging information set, and the data processing module is used to create an asynchronous thread and synchronize business data to a preset storage area through the asynchronous thread.
[0029] In step S101, the first parameter value is used to characterize the maximum number of pages corresponding to the target business data when performing data paging processing on the target business data. Data paging processing is used to split the target business data into at least one page of business data. The second parameter value is used to characterize the minimum number of business data in each page of business data.
[0030] Step S102: Determine the set of pagination information corresponding to the target business data based on the first parameter value and the second parameter value.
[0031] In step S102, the pagination information set includes at least the data range corresponding to at least one page of business data and the actual number of business data in at least one page of business data. The data range corresponding to the business data includes the start sequence number and end sequence number corresponding to each page of business data. The sequence number is the unique identification information of each piece of business data, such as the identifier ID.
[0032] Step S103: Create at least one asynchronous thread based on the data range corresponding to at least one page of business data.
[0033] Step S104: Obtain the actual amount of business data from the target database corresponding to the business system through an asynchronous thread, and synchronize the actual amount of business data to the preset storage area.
[0034] In steps S103-S104, the system can limit the data range corresponding to the business data through an asynchronous thread, and then retrieve the actual amount of business data from the target database through the asynchronous thread. The specific SQL statements corresponding to the system executing the above process are as follows:
[0035] with TMP_TAB(key)as(#querySQL#)select TMP_TAB.key from TMP_TAB.
[0036] where TMP_TAB.key between:START_KEY and:END_KEY.
[0037] Optionally, the target database can be an Oracle database.
[0038] Optionally, in this embodiment, the values of the first and second parameters are flexible and variable, and can be adjusted according to the actual production data. By adjusting the values of the first parameter (MAX_PAGE_NUM) and the second parameter (MIN_PAGE_SIZE), the total amount of data queried by each thread can be effectively controlled, the system memory usage can be controlled, and the pressure of parallel processing can be effectively controlled, avoiding the problem of excessive system pressure. Specifically, the smaller the second parameter value and the larger the first parameter value, the smaller the initial capacity (MIN_PAGE_SIZE) of the thread query. This fully utilizes system resources, improves data synchronization efficiency, and solves the technical problem of low data synchronization efficiency in existing technologies when synchronizing batch data.
[0039] Based on the scheme defined in steps S101 to S104 above, it can be understood that in this application, a data synchronization instruction is first received. This instruction carries a first parameter value and a second parameter value corresponding to the target business data to be synchronized. The first parameter value represents the maximum number of pages corresponding to the target business data when performing data paging processing. Data paging processing is used to split the target business data into at least one page of business data. The second parameter value represents the minimum number of business data items in each page of business data. Then, a paging information set corresponding to the target business data is determined based on the first and second parameter values. This paging information set includes at least the data range corresponding to at least one page of business data and the actual number of business data items in at least one page of business data. Next, at least one asynchronous thread is created based on the data range corresponding to at least one page of business data. Finally, the actual number of business data items is obtained from the target database corresponding to the business system through the asynchronous thread, and the actual number of business data items is synchronized to a preset storage area.
[0040] It should be noted that in the above process, the pagination information set is determined by the maximum number of pagination pages corresponding to the target business data (i.e., the first parameter value) and the minimum number of business data in each page (i.e., the second parameter value) in the pre-configured target business data. At least one asynchronous thread is created based on the pagination information set, and the actual number of business data is synchronized to the preset storage area through the asynchronous thread. This realizes the pagination synchronization of the target business data through the asynchronous thread, reduces the data synchronization time, and thus achieves the technical effect of improving the data synchronization efficiency. This solves the technical problem of low data synchronization efficiency in the existing technology when synchronizing batch data.
[0041] Optionally, in the data synchronization method provided in this application embodiment, determining the pagination information set corresponding to the target business data based on the first parameter value and the second parameter value includes: the system can obtain the sequence number corresponding to each of the M business data in the first business data table to obtain M sequence numbers, wherein the first business data table is a business data table in the target database, the first business data table is used to store the target business data, and the sequence number corresponding to each business data is used to characterize the order of the business data in the M business data, where M is a positive integer; then, a target data table is generated based on the M sequence numbers; then, the total amount of data in the target data table is counted to obtain the target quantity; finally, the pagination information set corresponding to the target business data is determined based on the target data table, the target quantity, the first parameter value, and the second parameter value.
[0042] Optionally, in the data synchronization method provided in this application embodiment, before obtaining the sequence number corresponding to each of the M business data in the first business data table and obtaining the M sequence numbers, the method further includes: the system can receive a data query instruction, wherein the data query instruction carries the unique identification information of the first business data table; and then the first business data table is determined from N business data tables in the target database based on the unique identification information, wherein N is a positive integer.
[0043] In this embodiment, the system can receive a data query instruction and determine the first business data table from N business data tables in the target database based on the unique identifier information of the first business data table carried in the data query instruction. Then, the system can obtain the sequence number corresponding to each of the M business data in the first business data table. The sequence number corresponding to each business data can be used to characterize the order of the business data in the M business data. The system generates a target data table based on the M sequence numbers. Then, the system can count the total amount of data in the target data table to obtain the target quantity, and determine the pagination information set corresponding to the target business data based on the target data table, the target quantity, the first parameter value, and the second parameter value.
[0044] It should be noted that the first business data table can be accurately identified from N business data tables in the target database by using its unique identifier. By obtaining the sequence number corresponding to each of the M business data records in the first business data table and generating the target data table based on the M sequence numbers, the pagination information set corresponding to the target business data can be determined based on the target data table, the total amount of data in the target data table (i.e., the target quantity), the first parameter value, and the second parameter value. This eliminates the need to obtain all the target business data to determine the data range and the actual quantity of the business data, reducing the time required to determine the pagination information set and thus improving data synchronization efficiency.
[0045] Optionally, in the data synchronization method provided in this application embodiment, generating a target data table based on M sequence numbers includes: the system can sort the M sequence numbers to obtain a sorting result; and then generate a target data table based on the sorting result.
[0046] In this embodiment, the system can sort the M sequence numbers to obtain a sorting result. For example, the system can sort the M sequence numbers from largest to smallest to obtain a sorting result. Then, the system can generate a target data table based on the sorting result. The SQL statements corresponding to the system executing the above process are as follows:
[0047] querySQL: select TRANSFLOW_ID from CORE_TRANSFLOW_DETAIL order byTRANSFLOW_ID.
[0048] It should be noted that the system sorts the M sequence numbers to obtain the sorting result, and generates a target data table based on the sorting result, providing data preparation for subsequently determining the data range of business data.
[0049] Optionally, in the data synchronization method provided in this application embodiment, determining the pagination information set corresponding to the target business data based on the target data table, the target quantity, the first parameter value, and the second parameter value includes: the system can calculate the ratio of the target quantity to the first parameter value to obtain a first value; then determine a second value based on the first value and the second parameter value, wherein the second value is used to characterize the maximum number of business data actually allocated to each page of business data; then calculate the ratio of the second value to the target quantity to obtain a third value, wherein the third value is used to characterize the actual number of pages corresponding to the target business data; finally, determine the pagination information set corresponding to the target business data based on the target data table, the second value, and the third value.
[0050] In this embodiment, the system calculates the ratio of the total amount of data in the target data table (i.e., the target quantity, TOTAL) to the value of a first parameter to obtain a first value. If the calculated first value is not an integer, the system can round it up to obtain the first value. Then, the system determines the maximum number of business data actually allocated to each page of business data (i.e., the second value) by comparing the first value with the value of a second parameter. Specifically, when the first value is greater than the second parameter value, the first value is used as the second value; when the first value is less than the second parameter value, the second parameter value is used as the second value. Then, the system can calculate the ratio of the second value to the target quantity to obtain the actual number of pages corresponding to the target business data (i.e., the third value), where the third value is less than or equal to the first parameter value. Based on the target data table, the second value, and the third value, the system determines the set of pagination information corresponding to the target business data. The SQL statements corresponding to the above process are as follows:
[0051] withTMP_TAB(key)as(#querySQL#),\nTTL_TABas(selectcount(1)totalfromTMP_TA B)select.
[0052] MIN(TMP_TAB.key)startKey,MAX(TMP_TAB.key)endKey,COUNT(1)quantityfromTMP_TAB,TTL_TAB.
[0053] groupBYTRUNC((ROWNUM-1) / greatest(TTL_TAB.total / :maxPageNum,:minPageSize))orderbystartKey.
[0054] It should be noted that by calculating the ratio of the target quantity to the first parameter value, a first value is obtained. Based on the first value and the second parameter value, a second value is determined, which can accurately determine the maximum number of business data actually allocated to each page of the target business data. By calculating the ratio of the second value to the target quantity, a third value is obtained, which can accurately determine the actual number of pages corresponding to the target business data. It is not necessary to obtain all the target business data to determine the data range and the actual quantity of business data, which reduces the time for determining the pagination information set and thus improves data synchronization efficiency.
[0055] Optionally, in the data synchronization method provided in this application embodiment, determining the pagination information set corresponding to the target business data based on the target data table, the second value, and the third value includes: the system can perform data pagination processing on the sequence number data in the target data table based on the second value and the third value to obtain at least one page of sequence number data; then obtain the data range corresponding to the at least one page of sequence number data and the actual number of sequence number data in the at least one page of sequence number data; and then determine the pagination information set based on the data range corresponding to the at least one page of sequence number data and the actual number of sequence number data in the at least one page of sequence number data.
[0056] Optionally, in the data synchronization method provided in this application embodiment, determining the paging information set based on the data range corresponding to at least one page of sequence number data and the actual quantity of sequence number data in at least one page of sequence number data includes: the system can take the data range corresponding to at least one page of sequence number data as the data range corresponding to business data to obtain the data range corresponding to at least one page of business data; then take the actual quantity of at least one page of sequence number data as the actual quantity of business data to obtain the actual quantity of business data in at least one page of business data; finally, determine the paging information set based on the data range corresponding to at least one page of business data and the actual quantity of business data in at least one page of business data.
[0057] In this embodiment, the system can split the serial number data in the target data table into at least one page of serial number data using a second and a third numerical value, and obtain the data range corresponding to at least one page of serial number data and the actual number of serial number data in at least one page of serial number data. Then, the system can use the data range corresponding to at least one page of serial number data as the data range corresponding to the business data to obtain the data range corresponding to at least one page of business data, and use the actual number of serial number data in at least one page as the actual number of business data to obtain the actual number of business data in at least one page of business data. Finally, the system determines the pagination information set based on the data range corresponding to at least one page of business data and the actual number of business data in at least one page of business data.
[0058] It should be noted that by taking the data range corresponding to at least one page of sequence number data in the target data table as the data range corresponding to the business data, and taking the actual quantity of at least one page of sequence number data in the target data table as the actual quantity of the business data, it is not necessary to obtain all the target business data to determine the data range and the actual quantity of the business data. This reduces the time required to determine the pagination information set and thus improves data synchronization efficiency.
[0059] Therefore, the data synchronization method of this application can determine the pagination information set by pre-configuring the maximum number of pages corresponding to the target business data (i.e., the first parameter value) and the minimum number of business data in each page (i.e., the second parameter value). Based on this pagination information set, at least one asynchronous thread is created. This asynchronous thread synchronizes the actual number of business data to a preset storage area, achieving pagination synchronization of the target business data through asynchronous threads. This reduces data synchronization time and improves data synchronization efficiency, thus solving the technical problem of low data synchronization efficiency in existing technologies when synchronizing batch data. By adjusting the values of the first and second parameters, the total amount of data queried by each thread can be effectively controlled, controlling the system's memory usage and effectively managing the pressure of parallel processing, avoiding excessive system pressure.
[0060] Example 2
[0061] According to an embodiment of this application, an embodiment of a data synchronization device is provided, wherein... Figure 2 This is a schematic diagram of an optional data synchronization device according to an embodiment of this application, such as... Figure 2 As shown, the device includes:
[0062] The receiving module 201 is used to receive a data synchronization instruction, wherein the data synchronization instruction carries a first parameter value and a second parameter value corresponding to the target business data to be synchronized. The first parameter value is used to characterize the maximum number of pages corresponding to the target business data when performing data paging processing on the target business data. Data paging processing is used to split the target business data into at least one page of business data. The second parameter value is used to characterize the minimum number of business data in each page of business data.
[0063] The determining module 202 is used to determine the pagination information set corresponding to the target business data based on the first parameter value and the second parameter value, wherein the pagination information set includes at least the data range corresponding to at least one page of business data and the actual number of business data in at least one page of business data;
[0064] Create module 203 to create at least one asynchronous thread based on the data range corresponding to at least one page of business data;
[0065] The synchronization module 204 is used to obtain the actual amount of business data from the target database corresponding to the business system through an asynchronous thread, and synchronize the actual amount of business data to the preset storage area.
[0066] It should be noted that the above-mentioned receiving module 201, determining module 202, creating module 203 and synchronizing module 204 correspond to steps S101 to S104 in the above embodiment 1. The four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in the above embodiment 1.
[0067] Optionally, the determining module includes: a first acquisition unit, configured to acquire the sequence number corresponding to each of the M business data items in the first business data table, thereby obtaining M sequence numbers, wherein the first business data table is a business data table in the target database, the first business data table is used to store target business data, and the sequence number corresponding to each business data item is used to characterize the order of the business data item in the M business data items, where M is a positive integer; a first generation unit, configured to generate a target data table based on the M sequence numbers; a statistics unit, configured to count the total amount of data in the target data table to obtain the target quantity; and a first determining unit, configured to determine the pagination information set corresponding to the target business data based on the target data table, the target quantity, the first parameter value, and the second parameter value.
[0068] Optionally, the data synchronization device further includes: a first receiving module, configured to receive a data query instruction before obtaining the sequence number corresponding to each of the M business data in the first business data table and obtaining the M sequence numbers, wherein the data query instruction carries unique identification information of the first business data table; and a first determining module, configured to determine the first business data table from N business data tables in the target database based on the unique identification information, wherein N is a positive integer.
[0069] Optionally, the first generation unit includes: a sorting unit for sorting the M sequence numbers to obtain a sorting result; and a second generation unit for generating a target data table based on the sorting result.
[0070] Optionally, the first determining unit includes: a first calculation unit, used to calculate the ratio of the target quantity to the first parameter value to obtain a first value; a second determining unit, used to determine a second value based on the first value and the second parameter value, wherein the second value is used to characterize the maximum number of business data actually allocated to each page of business data; a second calculation unit, used to calculate the ratio of the second value to the target quantity to obtain a third value, wherein the third value is used to characterize the actual number of pages corresponding to the target business data; and a third determining unit, used to determine the set of pagination information corresponding to the target business data based on the target data table, the second value, and the third value.
[0071] Optionally, the third determining unit includes: a processing unit, configured to perform data paging processing on the serial number data in the target data table based on the second value and the third value, to obtain at least one page of serial number data; a second obtaining unit, configured to obtain the data range corresponding to at least one page of serial number data and the actual number of serial number data in at least one page of serial number data; and a fourth determining unit, configured to determine the paging information set based on the data range corresponding to at least one page of serial number data and the actual number of serial number data in at least one page of serial number data.
[0072] Optionally, the fourth determining unit includes: a fifth determining unit, used to take the data range corresponding to at least one page of serial number data as the data range corresponding to business data, to obtain the data range corresponding to at least one page of business data; a sixth determining unit, used to take the actual quantity of at least one page of serial number data as the actual quantity of business data, to obtain the actual quantity of business data in at least one page of business data; and a seventh determining unit, used to determine the pagination information set based on the data range corresponding to at least one page of business data and the actual quantity of business data in at least one page of business data.
[0073] Example 3
[0074] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to execute the above-described data synchronization method at runtime.
[0075] Example 4
[0076] According to another aspect of the embodiments of this application, an electronic device is also provided, wherein, Figure 3 This is a schematic diagram of an optional electronic device according to an embodiment of this application, such as... Figure 3 As shown, the electronic device includes one or more processors; and a memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to run the programs, wherein the programs are configured to execute the aforementioned data synchronization method during runtime.
[0077] like Figure 3 As shown, this application embodiment provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps:
[0078] The system receives a data synchronization instruction, which carries a first parameter value and a second parameter value corresponding to the target business data to be synchronized. The first parameter value represents the maximum number of pages corresponding to the target business data when performing data paging processing on the target business data. Data paging processing is used to split the target business data into at least one page of business data. The second parameter value represents the minimum number of business data in each page of business data. Based on the first parameter value and the second parameter value, the system determines a paging information set corresponding to the target business data, wherein the paging information set includes at least the data range corresponding to at least one page of business data and the actual number of business data in at least one page of business data. Based on the data range corresponding to at least one page of business data, the system creates at least one asynchronous thread. The asynchronous thread retrieves the actual number of business data from the target database corresponding to the business system and synchronizes the actual number of business data to a preset storage area.
[0079] Optionally, the processor, when executing the program, further implements the following steps: obtaining the sequence number corresponding to each of the M business data items in the first business data table, resulting in M sequence numbers, wherein the first business data table is a business data table in the target database, the first business data table is used to store target business data, and the sequence number corresponding to each business data item is used to characterize the order of the business data item in the M business data items, where M is a positive integer; generating a target data table based on the M sequence numbers; calculating the total amount of data in the target data table to obtain the target quantity; and determining the paging information set corresponding to the target business data based on the target data table, the target quantity, the first parameter value, and the second parameter value.
[0080] Optionally, the processor may further implement the following steps when executing the program: before obtaining the sequence number corresponding to each of the M business data in the first business data table and obtaining the M sequence numbers, a data query instruction is received, wherein the data query instruction carries the unique identification information of the first business data table; the first business data table is determined from the N business data tables in the target database based on the unique identification information, wherein N is a positive integer.
[0081] Optionally, the processor may also perform the following steps when executing the program: sorting the M sequence numbers to obtain a sorting result; and generating a target data table based on the sorting result.
[0082] Optionally, when the processor executes the program, it further performs the following steps: calculating the ratio of the target quantity to the first parameter value to obtain a first value; determining a second value based on the first value and the second parameter value, wherein the second value is used to characterize the maximum number of business data actually allocated to each page of business data; calculating the ratio of the second value to the target quantity to obtain a third value, wherein the third value is used to characterize the actual number of pages corresponding to the target business data; and determining the paging information set corresponding to the target business data based on the target data table, the second value, and the third value.
[0083] Optionally, when the processor executes the program, it also performs the following steps: performing data paging processing on the serial numbers in the target data table based on the second value and the third value to obtain at least one page of serial number data; obtaining the data range corresponding to the at least one page of serial number data and the actual number of serial number data in the at least one page of serial number data; and determining the paging information set based on the data range corresponding to the at least one page of serial number data and the actual number of serial number data in the at least one page of serial number data.
[0084] Optionally, when the processor executes the program, it also performs the following steps: taking the data range corresponding to at least one page of sequence number data as the data range corresponding to the business data, to obtain the data range corresponding to at least one page of business data; taking the actual quantity of at least one page of sequence number data as the actual quantity of business data, to obtain the actual quantity of business data in at least one page of business data; and determining the paging information set based on the data range corresponding to at least one page of business data and the actual quantity of business data in at least one page of business data.
[0085] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0086] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0087] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.
[0088] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0091] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A data synchronization method, characterized in that, include: A data synchronization instruction is received, wherein the data synchronization instruction carries a first parameter value and a second parameter value corresponding to the target business data to be synchronized, the first parameter value is used to characterize the maximum number of pages corresponding to the target business data when performing data paging processing on the target business data, the data paging processing is used to split the target business data into at least one page of business data, and the second parameter value is used to characterize the minimum number of business data in each page of business data; Based on the first parameter value and the second parameter value, a pagination information set corresponding to the target business data is determined, wherein the pagination information set includes at least the data range corresponding to the at least one page of business data and the actual number of business data in the at least one page of business data; Create at least one asynchronous thread based on the data range corresponding to at least one page of business data; The asynchronous thread retrieves the actual amount of business data from the target database corresponding to the business system and synchronizes the actual amount of business data to the preset storage area. The step of determining the pagination information set corresponding to the target business data based on the first parameter value and the second parameter value includes: obtaining the sequence number corresponding to each of the M business data items in the first business data table to obtain M sequence numbers, wherein the first business data table is a business data table in the target database, the first business data table is used to store the target business data, and the sequence number corresponding to each business data item is used to represent the order of the business data item in the M business data items, where M is a positive integer; generating a target data table based on the M sequence numbers; counting the total amount of data in the target data table to obtain a target quantity; and determining the pagination information set corresponding to the target business data based on the target data table, the target quantity, the first parameter value, and the second parameter value. The process of determining the pagination information set corresponding to the target business data based on the target data table, the target quantity, the first parameter value, and the second parameter value includes: calculating the ratio of the target quantity to the first parameter value to obtain a first value; determining a second value based on the first value and the second parameter value, wherein the second value is used to characterize the maximum number of business data actually allocated to each page of business data; calculating the ratio of the second value to the target quantity to obtain a third value, wherein the third value is used to characterize the actual number of pages corresponding to the target business data; and determining the pagination information set corresponding to the target business data based on the target data table, the second value, and the third value.
2. The method according to claim 1, characterized in that, Before obtaining the sequence number corresponding to each of the M business data entries in the first business data table, and thus obtaining the M sequence numbers, the method further includes: Receive a data query instruction, wherein the data query instruction carries the unique identifier information of the first business data table; The first business data table is determined from N business data tables in the target database based on the unique identifier information, where N is a positive integer.
3. The method according to claim 1, characterized in that, Generate a target data table based on the M sequence numbers, including: Sort the M sequence numbers to obtain the sorting result; The target data table is generated based on the sorting results.
4. The method according to claim 1, characterized in that, Determining the pagination information set corresponding to the target business data based on the target data table, the second value, and the third value includes: Based on the second value and the third value, the serial number data in the target data table is paginated to obtain at least one page of serial number data. Obtain the data range corresponding to the at least one page of serial number data and the actual number of serial number data in the at least one page of serial number data; The pagination information set is determined based on the data range corresponding to the at least one page of serial number data and the actual number of serial number data in the at least one page of serial number data.
5. The method according to claim 4, characterized in that, The pagination information set is determined based on the data range corresponding to the at least one page of serial number data and the actual number of serial number data in the at least one page of serial number data, including: The data range corresponding to the at least one page of serial number data is used as the data range corresponding to the business data to obtain the data range corresponding to the at least one page of business data; The actual number of the serial number data in the at least one page is taken as the actual number of the business data to obtain the actual number of business data in the at least one page of business data; The pagination information set is determined based on the data range corresponding to the at least one page of business data and the actual number of business data in the at least one page of business data.
6. A data synchronization device, characterized in that, include: A receiving module is used to receive a data synchronization instruction, wherein the data synchronization instruction carries a first parameter value and a second parameter value corresponding to the target business data to be synchronized, the first parameter value is used to characterize the maximum number of pages corresponding to the target business data when performing data paging processing on the target business data, the data paging processing is used to split the target business data into at least one page of business data, and the second parameter value is used to characterize the minimum number of business data in each page of business data; The determining module is used to determine the pagination information set corresponding to the target business data based on the first parameter value and the second parameter value, wherein the pagination information set includes at least the data range corresponding to the at least one page of business data and the actual number of business data in the at least one page of business data; A module is created to create at least one asynchronous thread based on the data range corresponding to the at least one page of business data. The synchronization module is used to obtain the actual amount of business data from the target database corresponding to the business system through the asynchronous thread, and synchronize the actual amount of business data to the preset storage area; The determining module includes: a first acquisition unit, configured to acquire the sequence number corresponding to each of the M business data items in a first business data table, thereby obtaining M sequence numbers, wherein the first business data table is a business data table in the target database, the first business data table is used to store the target business data, and the sequence number corresponding to each business data item is used to characterize the order of the business data item in the M business data items, where M is a positive integer; a first generation unit, configured to generate a target data table based on the M sequence numbers; a statistics unit, configured to count the total amount of data in the target data table to obtain a target quantity; and a first determining unit, configured to determine the pagination information set corresponding to the target business data based on the target data table, the target quantity, the first parameter value, and the second parameter value. The first determining unit includes: a first calculation unit, used to calculate the ratio of the target quantity to the first parameter value to obtain a first value; a second determining unit, used to determine a second value based on the first value and the second parameter value, wherein the second value is used to characterize the maximum number of business data actually allocated to each page of business data; a second calculation unit, used to calculate the ratio of the second value to the target quantity to obtain a third value, wherein the third value is used to characterize the actual number of pages corresponding to the target business data; and a third determining unit, used to determine the pagination information set corresponding to the target business data based on the target data table, the second value, and the third value.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the data synchronization method according to any one of claims 1 to 5 when it is run.
8. An electronic device, characterized in that, The electronic device includes one or more processors; A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to be configured to run the programs, wherein the programs are configured to execute the data synchronization method of any one of claims 1 to 5 at runtime.