A data synchronization method and apparatus, an electronic device, and a computer readable medium

Abstract

The application discloses a data synchronization method and device, electronic equipment and a computer readable medium, relates to the technical field of big data data exchange, and a specific embodiment comprises receiving a data synchronization request, obtaining corresponding end-of-day batch processing data; calling a database archiving tool to generate corresponding database archiving files based on the end-of-day batch processing data and store the database archiving files; reading the database archiving files and dividing the database archiving files into pieces according to a set number to obtain piece files; and concurrently executing a synchronization process on each piece file. Large amounts of data can improve data transmission efficiency through large file transmission, meet non-functional requirements by controlling the concurrent processing speed of files, and reduce the pressure on the database; and the data processing stack customizes the processing mode of file records, providing flexibility for data in different code formats. The piece files set the concurrency degree according to non-functional requirements, ensure that the pressure on the database is controllable, and improve the generality and extreme TPS data processing performance.

Description

Technical Field

[0001] This application relates to the field of big data exchange technology, and in particular to a data synchronization method, apparatus, electronic device and computer-readable medium. Background Technology

[0002] Currently, core banking systems operate in two transaction modes: online and batch processing. For online transactions, transaction data can be synchronized to the distributed system via message queues (MQ). However, for batch processing, some application scenarios experience massive data changes within a short period, such as interest settlement batch processing. In such cases, using message synchronization would create significant concurrency pressure and substantial synchronization delays, affecting normal online transaction data synchronization. Directly using binlog parsing and message sending to synchronize data changes would generate a huge volume of concurrent messages, placing immense pressure on the message center, causing significant data synchronization delays, and impacting normal online message synchronization. This approach lacks versatility and has poor performance in handling extreme TPS (transactions per second) data volumes. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data synchronization method, apparatus, electronic device, and computer-readable medium, which can solve the problems of existing data synchronization methods that put great pressure on the message center, cause a large amount of data synchronization delay, and have poor data synchronization processing performance.

[0004] To achieve the above objectives, according to one aspect of the embodiments of this application, a data synchronization method is provided, comprising:

[0005] Receive data synchronization requests and obtain the corresponding end-of-day batch processing data;

[0006] The database archive tool is invoked to generate and store the corresponding database archive files based on the end-of-day batch processing data.

[0007] Read the database archive file and split the database archive file into fragments according to the set number of records to obtain fragmented files;

[0008] The synchronization process is executed concurrently on each file segment.

[0009] Optionally, read the database archive file, including:

[0010] Set up a scheduled task to execute at a set time to read the stored database archive files from the distributed file system.

[0011] Optionally, a synchronization process is executed concurrently on each file shard, including:

[0012] At preset time points, the processing stack is invoked to simultaneously perform transformation processing on each record in each shard file and update the corresponding database for data synchronization requests.

[0013] Optionally, at preset time points, the processing stack is invoked to simultaneously perform transformation processing on each record within each fragment file, including:

[0014] Determine the database architecture corresponding to the data synchronization request, and call the corresponding code conversion table and special field conversion table based on the database architecture, so as to simultaneously perform code conversion and special field conversion on each record in each shard file at a preset time point.

[0015] Optionally, before calling the corresponding character set conversion table and special field conversion table based on the database schema, the method may also include:

[0016] Based on the database architecture, display the corresponding custom conversion options;

[0017] Obtain user input regarding their selection of custom conversion options, and generate a code conversion table and a special field conversion table based on the selection.

[0018] Optionally, generate the corresponding database archive file, including:

[0019] Determine the target database for the data synchronization request;

[0020] Obtain the file type of the target database configuration, and call the conversion program corresponding to the file type in the database archive tool to convert the end-of-day batch data into a database archive file corresponding to the file type.

[0021] Optionally, a synchronization process is executed concurrently on each file shard, including:

[0022] In response to a failure to synchronize a file segment, a timed retry task is generated based on the failed file segment. The timed retry task is executed at a preset time until the corresponding file segment is successfully synchronized.

[0023] In addition, this application also provides a data synchronization device, including:

[0024] The receiving unit is configured to receive data synchronization requests and obtain the corresponding end-of-day batch processing data.

[0025] The file generation unit is configured to call the database archive tool to generate and store the corresponding database archive file based on the end-of-day batch processing data.

[0026] The reading unit is configured to read the database archive file and split the database archive file into fragments according to a set number of records to obtain fragmented files;

[0027] The concurrent execution unit is configured to execute synchronous processes concurrently on each file segment.

[0028] Optionally, the reading unit is further configured to:

[0029] Set up a scheduled task to execute at a set time to read the stored database archive files from the distributed file system.

[0030] Optionally, the concurrent execution unit is further configured to:

[0031] At preset time points, the processing stack is invoked to simultaneously perform transformation processing on each record in each shard file and update the corresponding database for data synchronization requests.

[0032] Optionally, the concurrent execution unit is further configured to:

[0033] Determine the database architecture corresponding to the data synchronization request, and call the corresponding code conversion table and special field conversion table based on the database architecture, so as to simultaneously perform code conversion and special field conversion on each record in each shard file at a preset time point.

[0034] Optionally, the apparatus further includes a table generation unit, configured to:

[0035] Based on the database architecture, display the corresponding custom conversion options;

[0036] Obtain user input regarding their selection of custom conversion options, and generate a code conversion table and a special field conversion table based on the selection.

[0037] Optionally, the file generation unit is further configured to:

[0038] Determine the target database for the data synchronization request;

[0039] Obtain the file type of the target database configuration, and call the conversion program corresponding to the file type in the database archive tool to convert the end-of-day batch data into a database archive file corresponding to the file type.

[0040] Optionally, the data synchronization device further includes a retry unit, configured to:

[0041] In response to a failure to synchronize a file segment, a timed retry task is generated based on the failed file segment. The timed retry task is executed at a preset time until the corresponding file segment is successfully synchronized.

[0042] In addition, this application also provides a data synchronization electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the data synchronization method described above.

[0043] In addition, this application also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data synchronization method described above.

[0044] To achieve the above objectives, according to another aspect of the embodiments of this application, a computer program product is provided.

[0045] A computer program product according to an embodiment of this application includes a computer program that, when executed by a processor, implements the data synchronization method provided in an embodiment of this application.

[0046] One embodiment of the above invention has the following advantages or beneficial effects: This application receives a data synchronization request to obtain the corresponding end-of-day batch processing data; it calls a database archiving tool to generate and store the corresponding database archiving file based on the end-of-day batch processing data; it reads the database archiving file and segments it according to a set number of records to obtain segmented files; and it concurrently executes a synchronization process on each segmented file. Large data transfers via large files can improve data transmission efficiency. Controlling the concurrent processing speed of files can meet non-functional requirements and reduce pressure on the database. Custom processing methods for file records can be defined in the data processing stack, providing flexibility for data compatible with different encoding formats. The concurrency level of segmented files can be set according to non-functional requirements, ensuring controllable pressure on the database and improving versatility and performance in handling extreme TPS data volumes.

[0047] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0048] The accompanying drawings are provided to better understand this application and do not constitute an undue limitation thereof. Wherein:

[0049] Figure 1 This is a schematic diagram illustrating the main flow of a data synchronization method according to an embodiment of this application;

[0050] Figure 2 This is a schematic diagram illustrating the main flow of a data synchronization method according to an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the main flow of a data synchronization method according to an embodiment of this application;

[0052] Figure 4This is a schematic diagram of the main units of a data synchronization device according to an embodiment of this application;

[0053] Figure 5 This is an exemplary system architecture diagram to which embodiments of this application can be applied;

[0054] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers in the embodiments of this application. Detailed Implementation

[0055] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the collection, analysis, use, transmission, and storage of user personal information involved in the technical solutions of this application comply with relevant laws and regulations, are used for legitimate and reasonable purposes, are not shared, disclosed, or sold outside of these legitimate uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken to prevent unauthorized access to such personal information data, ensure that personnel authorized to access personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once this user personal information data is no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data.

[0056] When used, including in certain relevant applications, data is deidentified to protect user privacy, for example by removing specific identifiers, controlling the amount or specificity of stored data, controlling how data is stored, and / or other methods.

[0057] Figure 1 This is a schematic diagram illustrating the main flow of a data synchronization method according to an embodiment of this application, as shown below. Figure 1 As shown, data synchronization methods include:

[0058] Step S101: Receive a data synchronization request and obtain the corresponding end-of-day batch processing data.

[0059] In this embodiment, the execution entity of the data synchronization method (e.g., a server) can receive data synchronization requests via wired or wireless connections. The data synchronization request can be a request to synchronize heterogeneous data. The execution entity can obtain the end-of-day batch processing data corresponding to the data synchronization request. For example, the end-of-day batch processing data could be end-of-day interest batch processing data or end-of-day deposit batch processing data; this embodiment does not specifically limit the end-of-day batch processing data.

[0060] Step S102: Call the database archive tool to generate and store the corresponding database archive file based on the end-of-day batch processing data.

[0061] Database archiving tools, such as Word and Excel, convert the daily batch processing data into data files in a format acceptable to the target database for data synchronization; these are called database archiving files. A distributed system is then invoked to upload these database archiving files to a distributed file system, specifically the HDFS file system, using multiple threads.

[0062] Specifically, generating the corresponding database archive file includes: determining the target database corresponding to the data synchronization request; obtaining the file type configured for the target database, such as whether the target database can only receive Word files, XML files, or Excel files, etc. This embodiment does not specifically limit the file type configured for the target database. The conversion program corresponding to the file type in the database archive tool is called to convert the end-of-day batch processing data into a database archive file corresponding to the file type. For example, the type of the database archive file can be Word, XML, or Excel corresponding to the target database, or it can be a file in other user-defined preset formats. This embodiment does not specifically limit the type of the database archive file.

[0063] Step S103: Read the database archive file and split the database archive file into fragments according to the set number of records to obtain fragmented files.

[0064] First, the executing entity can determine the set number of records corresponding to the database archive file. For example, the set number of records corresponding to the database archive file can be determined as follows: The database archive file is segmented based on contextual semantics; the number of segments is used as the set number of records to divide the database archive file into fragments, resulting in individual fragment files. This reduces the file size occupied by a single synchronization and improves file synchronization efficiency by dividing the database archive file into fragment files.

[0065] Step S104: Perform a synchronization process concurrently on each file segment.

[0066] Specifically, the synchronization process is executed concurrently for each file segment, including: in response to a file segment synchronization failure, a timed retry task is generated based on the failed file segment, and the timed retry task is executed at a preset execution time until the corresponding file segment is successfully synchronized.

[0067] For files that fail to be processed, a scheduled task will periodically scan these files and process them again to improve the success rate of data synchronization.

[0068] This embodiment receives a data synchronization request to obtain the corresponding end-of-day batch processing data; it calls a database archiving tool to generate and store the corresponding database archiving file based on the end-of-day batch processing data; it reads the database archiving file and splits it into fragments according to a set number of records to obtain fragmented files; and it concurrently executes the synchronization process on each fragmented file. Transferring large amounts of data via large files can improve data transmission efficiency. Controlling the concurrent processing speed of files can meet non-functional requirements and reduce the pressure on the database. The processing method for file records can be customized in the data processing stack, providing flexibility for data with different encoding formats. The concurrency level of fragmented files can be set according to non-functional requirements to ensure controllable pressure on the database, improve versatility and performance in handling extreme TPS data volumes.

[0069] Figure 2 This is a schematic diagram of the main flow of a data synchronization method according to an embodiment of this application, as shown below. Figure 2 As shown, data synchronization methods include:

[0070] Step S201: Receive data synchronization request and obtain the corresponding end-of-day batch processing data.

[0071] Step S202: Call the database archive tool to generate and store the corresponding database archive file based on the end-of-day batch processing data.

[0072] Specifically, a distributed system can use multiple threads to store database archive files into the HDFS file system.

[0073] Step S203: Set up a scheduled task to execute at a set time to read the stored database archive file from the distributed file system and split the database archive file into fragments according to a set number of records to obtain fragmented files.

[0074] A scheduled task invokes a distributed system to read database archive files from a distributed file system (HDFS). The specified number of rows can be determined based on user configuration data. After obtaining the database archive files, the executing entity can partition the archive files into fragments according to the configured number of rows, resulting in fragment files corresponding to the specified number of rows.

[0075] Step S204: At a preset time point, the processing stack is invoked to simultaneously perform transformation processing on each record in each shard file and update the corresponding database of the data synchronization request.

[0076] At preset time points, such as 6 AM and 12 PM daily, the processing stack is invoked to simultaneously and in parallel perform transformation processing on each record within the sharded file. Based on the transformation results, the database to which the data to be synchronized is updated according to the data synchronization request.

[0077] Specifically, at a preset time point, the processing stack is invoked to simultaneously perform conversion processing on each record in each shard file, including: determining the database architecture corresponding to the data synchronization request, and invoking the corresponding code conversion table and special field conversion table based on the database architecture, so as to simultaneously perform code conversion and special field conversion on each record in each shard file at the preset time point.

[0078] The database architecture corresponding to the data synchronization request can be the database architecture of the database to which the data is to be synchronized. In this embodiment, the database architecture has corresponding file types that can be received. Specifically, a preset mapping table between data architectures and file types can be called to determine the corresponding file type based on the database architecture corresponding to the data synchronization request.

[0079] Specifically, before calling the corresponding character set conversion table and special field conversion table based on the database architecture, the method also includes: displaying the corresponding custom conversion options based on the database architecture; obtaining the user's selection operation on the custom conversion options; and generating the character set conversion table and special field conversion table based on the selection operation. The custom options are used to characterize the processing method for file records that need to be synchronized, thereby enabling customization of the processing method for file records in the data processing stack, providing flexibility for data compatible with different character set formats.

[0080] Figure 3This is a schematic diagram of the main flow of a data synchronization method according to an embodiment of this application. The data synchronization method of this embodiment can be applied to file-based heterogeneous data synchronization scenarios. During the end-of-day batch processing stage, the application component generates a file archive for the host database, and the distributed system uploads the archive to the HDFS file system via multi-threading. A scheduled task is set to read the file from the HDFS file system, sharding the file according to a set number of records, and processing multiple shards concurrently. For each file shard, the file records are read one by one, processed using a processing stack, and a custom data processing program is executed to complete operations such as encoding conversion and special field processing, processing the data into data conforming to the format and content of the distributed business database. After processing, the processed data is updated to the business database. For files where processing fails, a scheduled task periodically scans these files for reprocessing to ensure the success rate of data synchronization. Using files as the medium for heterogeneous data synchronization, and leveraging file concurrency and a custom processing stack mode, this method eliminates the high dependence on the technical characteristics of the database product itself in general heterogeneous data synchronization methods, making it universally applicable across database products with different architectures. For scenarios involving massive daily batch data synchronization that do not require high real-time performance, large-scale data transfer via large files can improve data transmission efficiency. Controlling file concurrency processing speed can meet non-functional requirements and reduce pressure on the database. Custom processing methods for file records can be defined in the data processing stack, providing flexibility for data with different encoding formats. The method proposed in this application does not rely on database log analysis or the architecture and technical characteristics of specific database products, thus possessing universality for different database products at both ends of heterogeneous data synchronization. The system can perform customized operations on database data with different architectures, shielding the databases at both ends of the synchronization from data differences and providing flexibility for data with different encoding formats. Therefore, it can be widely applied to various database synchronization scenarios with different architectures. Data updates do not require sending messages, avoiding real-time concurrent processing pressure on the message center and not affecting the synchronization of other online transaction data. The concurrency level for file processing can be set according to non-functional requirements, ensuring controllable pressure on the database.

[0081] Figure 4 This is a schematic diagram of the main units of a data synchronization device according to an embodiment of this application. Figure 4 As shown, the data synchronization device 400 includes a receiving unit 401, a file generation unit 402, a reading unit 403, and a concurrent execution unit 404.

[0082] The receiving unit 401 is configured to receive data synchronization requests and obtain the corresponding end-of-day batch processing data.

[0083] The file generation unit 402 is configured to call the database archive tool to generate and store the corresponding database archive file based on the end-of-day batch processing data.

[0084] The reading unit 403 is configured to read the database archive file and split the database archive file into fragments according to a set number of records to obtain fragmented files.

[0085] Concurrent execution unit 404 is configured to concurrently execute synchronous processes on each fragment file.

[0086] In some embodiments, the reading unit 403 is further configured to: set a scheduled task to execute the scheduled task at a set time to read the stored database archive files from the distributed file system.

[0087] In some embodiments, the concurrent execution unit 404 is further configured to: simultaneously execute the transformation processing of each record in each shard file by calling the processing stack at a preset time point, and update the database corresponding to the data synchronization request.

[0088] In some embodiments, the concurrent execution unit 404 is further configured to: determine the database architecture corresponding to the data synchronization request, and call the corresponding code conversion table and special field conversion table based on the database architecture, so as to simultaneously perform code conversion and special field conversion on each record in each shard file at a preset time point.

[0089] In some embodiments, the data synchronization device further includes Figure 4 The table generation unit (not shown) is configured to: display corresponding custom conversion options based on the database architecture; obtain user input on the selection of custom conversion options; and generate a code conversion table and a special field conversion table based on the selection.

[0090] In some embodiments, the file generation unit 402 is further configured to: determine the target database corresponding to the data synchronization request; obtain the file type configured in the target database; and call the conversion program corresponding to the file type in the database archive tool to convert the end-of-day batch processing data into a database archive file corresponding to the file type.

[0091] In some embodiments, the data synchronization device further includes Figure 4 The retry unit, not shown, is configured to: in response to a fragment file synchronization failure, generate a timed retry task based on the failed fragment file, and execute the timed retry task at a preset execution time until the corresponding fragment file is successfully synchronized.

[0092] It should be noted that the data synchronization method and data synchronization device in this application are related in terms of specific implementation, so repeated content will not be described again.

[0093] Figure 5 An exemplary system architecture 500 is shown that can be applied to the data synchronization method or data synchronization apparatus of the embodiments of this application.

[0094] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0095] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, and 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0096] Terminal devices 501, 502, and 503 can be various electronic devices with a data synchronization processing screen and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0097] Server 505 can be a server providing various services, such as a backend management server supporting data synchronization requests submitted by users using terminal devices 501, 502, and 503 (for example only). The backend management server can receive data synchronization requests, obtain the corresponding end-of-day batch data; call the database archiving tool to generate and store corresponding database archiving files based on the end-of-day batch data; read the database archiving files and shard them according to a set number of records to obtain sharded files; and concurrently execute the synchronization process on each sharded file. Large data transfers via large files can improve data transfer efficiency. Controlling the concurrent processing speed of files can meet non-functional requirements and reduce pressure on the database. Custom processing methods for file records can be defined in the data processing stack, providing flexibility for data with different encoding formats. The concurrency level of sharded files can be set according to non-functional requirements, ensuring controllable pressure on the database and improving versatility and performance in handling extreme TPS data volumes.

[0098] It should be noted that the data synchronization method provided in this application embodiment is generally executed by server 505, and correspondingly, the data synchronization device is generally set in server 505.

[0099] It should be understood that Figure 5The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0100] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing a terminal device according to the embodiments of this application. Figure 6 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0101] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0102] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0103] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this application.

[0104] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be located in a processor; for example, a processor can be described as including a receiving unit, a file generation unit, a reading unit, and a concurrent execution unit. The names of these units do not necessarily limit the specific unit itself.

[0107] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to receive a data synchronization request, obtain corresponding end-of-day batch processing data; call a database archiving tool to generate and store corresponding database archiving files based on the end-of-day batch processing data; read the database archiving files and divide them into fragments according to a set number of entries to obtain fragmented files; and concurrently execute a synchronization process on each fragmented file.

[0108] The computer program product of this application includes a computer program that, when executed by a processor, implements the data synchronization method in the embodiments of this application.

[0109] According to the technical solution of this application embodiment, large-scale data transmission via large files can improve data transmission efficiency. Non-functional requirements can be met by controlling the concurrent processing speed of files, reducing pressure on the database. Custom processing methods for file records can be defined in the data processing stack, providing flexibility for data compatible with different encoding formats. File fragments can have their concurrency set according to non-functional requirements, ensuring controllable pressure on the database and improving versatility and performance in handling extreme TPS data volumes.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data synchronization method, characterized in that, include: Receive data synchronization requests and obtain the corresponding end-of-day batch processing data; The database archive tool is invoked to generate and store the corresponding database archive file based on the end-of-day batch processing data; Read the database archive file and split the database archive file into fragments according to a set number of records to obtain fragmented files; The synchronization process is executed concurrently for each of the aforementioned shard files, including: simultaneously executing the conversion processing of each record in each of the aforementioned shard files by calling the processing stack at a preset time point, including: determining the database architecture corresponding to the data synchronization request, calling the corresponding code conversion table and special field conversion table based on the database architecture, so as to simultaneously execute the code conversion and special field conversion of each record in each of the aforementioned shard files at a preset time point; and updating the database corresponding to the data synchronization request; wherein, the database file is segmented based on context semantics, and the number of segments obtained is used as a set number of entries to segment the database file, resulting in each shard file.

2. The method according to claim 1, characterized in that, The reading of the database archive file includes: Set up a scheduled task to execute at a set time to read the stored database archive files from the distributed file system.

3. The method according to claim 1, characterized in that, Before calling the corresponding code conversion table and special field conversion table based on the database architecture, the method further includes: Based on the aforementioned database architecture, the corresponding custom conversion options are displayed; Obtain the user's input selection operation for the custom conversion option, and generate a code conversion table and a special field conversion table based on the selection operation.

4. The method according to claim 1, characterized in that, The generation of the corresponding database archive file includes: Determine the target database corresponding to the data synchronization request; Obtain the file type of the target database configuration, and call the conversion program corresponding to the file type in the database archive tool to convert the end-of-day batch processing data into a database archive file corresponding to the file type.

5. The method according to claim 1, characterized in that, The concurrent execution of the synchronization process for each of the aforementioned file segments includes: In response to a failure to synchronize a file segment, a timed retry task is generated based on the failed file segment. The timed retry task is executed at a preset execution time until the corresponding file segment is successfully synchronized.

6. A data synchronization device, characterized in that, include: The receiving unit is configured to receive data synchronization requests and obtain the corresponding end-of-day batch processing data. The file generation unit is configured to call the database archive tool to generate and store the corresponding database archive file based on the end-of-day batch processing data. The reading unit is configured to read the database archive file and split the database archive file into fragments according to a set number of records to obtain fragmented files; The concurrent execution unit is configured to concurrently execute a synchronization process on each of the said fragment files; The concurrent execution unit is further configured to: call the processing stack at a preset time point to simultaneously perform the conversion processing of each record in each of the shard files, and update the database corresponding to the data synchronization request; wherein, the database file is segmented based on context semantics, and the number of segments obtained is used as a set number to segment the database file to obtain each shard file; The database architecture corresponding to the data synchronization request is determined, and the corresponding code conversion table and special field conversion table are called based on the database architecture to simultaneously perform code conversion and special field conversion on each record in each of the shard files at a preset time point.

7. The apparatus according to claim 6, characterized in that, The reading unit is further configured to: Set up a scheduled task to execute at a set time to read the stored database archive files from the distributed file system.

8. The apparatus according to claim 6, characterized in that, The device further includes a table generation unit, configured to: Based on the aforementioned database architecture, the corresponding custom conversion options are displayed; Obtain the user's input selection operation for the custom conversion option, and generate a code conversion table and a special field conversion table based on the selection operation.

9. The apparatus according to claim 6, characterized in that, The file generation unit is further configured to: Determine the target database corresponding to the data synchronization request; Obtain the file type of the target database configuration, and call the conversion program corresponding to the file type in the database archive tool to convert the end-of-day batch processing data into a database archive file corresponding to the file type.

10. A data synchronization electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images