Data synchronization method, device and system under distributed architecture

Abstract

The application provides a data synchronization method, device and system under a distributed architecture, the method comprising: in the case of obtaining a target copy file, selecting a target parser from a plurality of parsers to parse the target copy file according to a first file format to obtain a target file, the target parser being used for parsing a file in the first file format, the first file format being a file format of the target copy file; performing first predetermined processing on the target file to obtain a plurality of subfiles, the first predetermined processing comprising splitting processing, and the data in each subfile being target data; storing each target data in a corresponding temporary table of a distributed unit, the distributed unit comprising a formal table and a temporary table, one formal table corresponding to at least one temporary table, and one distributed unit corresponding to at least one target data; and storing at least part of the target data in each temporary table in a corresponding formal table. The problem of poor generality of a data synchronization mechanism is solved.

Description

Technical Field

[0001] This application relates to the field of data synchronization technology in banking systems, and more specifically, to a data synchronization method, apparatus, computer-readable storage medium, processor, and data synchronization system under a distributed architecture. Background Technology

[0002] In a bank's distributed deployment architecture, most related systems need to share data. However, due to network limitations, frequent cross-system remote data access calls cannot be initiated. Therefore, it is necessary to copy and synchronize data from related systems to the local system for access. Because banks have numerous related systems, the methods of file transfer, file formats, file content formats, data quality, file data validity periods, and file relationships are relatively complex. Existing data synchronization frameworks are limited in functionality and scalability, providing only single-file processing and not supporting dynamic data adjustments. Summary of the Invention

[0003] The main objective of this application is to provide a data synchronization method, apparatus, computer-readable storage medium, processor, and data synchronization system under a distributed architecture, so as to at least solve the problem that the data synchronization mechanism between related systems in the prior art only supports single file processing and has poor versatility.

[0004] To achieve the above objectives, according to one aspect of this application, a data synchronization method under a distributed architecture is provided, comprising: upon obtaining a target copy file, selecting a target parser from multiple parsers to parse the target copy file according to a first file format to obtain a target file, wherein the target copy file is a copy file sent by an associated system, the target parser is used to parse files of the first file format, the first file format being the file format of the target copy file, and the multiple parsers being used to parse files of different file formats; performing a first predetermined processing on the target file to obtain multiple sub-files, the first predetermined processing including splitting processing, wherein the data in each sub-file is target data; storing each target data into a temporary table of a corresponding distributed unit, wherein the distributed unit includes a formal table and a temporary table, one formal table corresponding to at least one temporary table, and one distributed unit corresponding to at least one target data; and storing at least a portion of the target data in each temporary table into the corresponding formal table.

[0005] Optionally, the target file undergoes a first predetermined processing to obtain multiple sub-files, including: splitting the target file according to preset rules to obtain multiple intermediate files, each intermediate file corresponding to a temporary table, the preset rules including at least one of the following: splitting according to the customer number of the data in the target file, splitting according to the database to which the data in the target file belongs, splitting according to whether the target file is a customer-related file; and sharding each intermediate file according to the corresponding number of threads to obtain multiple sub-files corresponding to each intermediate file, the number of sub-files being the same as the corresponding number of threads.

[0006] Optionally, storing each of the target data into a temporary table of the corresponding distributed unit includes: configuring each of the target data corresponding to the intermediate file as a thread; executing multiple threads corresponding to each intermediate file in parallel to store multiple target data corresponding to each intermediate file into the corresponding temporary table in parallel.

[0007] Optionally, the type of the copy file includes single-table files and general-purpose table files. The single-table file indicates that the copy file corresponds one-to-one with the official table, and the general-purpose table file indicates that a file group corresponds one-to-one with the official table. The file group includes multiple copy files. At least a portion of the target data in each temporary table is stored into the corresponding official table, including: a first processing step, in the case that the type of the target copy file is a single-table file, storing a portion of the target data in all the temporary tables corresponding to the target copy file into the corresponding official table; a determination step, in the case that the type of the target copy file is a general-purpose table file, determining whether the data of other copy files in the target file group is stored into the corresponding temporary table, where the target file group is the file group to which the target copy file belongs; and a second processing step, in the case that the data of other copy files is stored into the corresponding temporary table, storing a portion of the target data in all the temporary tables corresponding to the file group into the corresponding official table.

[0008] Optionally, the first processing step includes: performing a second predetermined processing on the target data in all the temporary tables to obtain final data; and storing the final data in the corresponding formal table, wherein the second predetermined processing includes at least one of data splitting, data combining, and deletion.

[0009] Optionally, if the data of other copy files is not stored in the corresponding temporary table, the method further includes: performing the determining step and the second processing step at least once in sequence until a portion of the target data in all the temporary tables corresponding to the file group is stored in the corresponding formal table.

[0010] Optionally, before performing the first predetermined processing on the target file to obtain multiple sub-files, after selecting a target parser from multiple parsers according to the first file format to parse the target copy file and obtain the target file, the method further includes: performing file verification on the target file, and performing the first predetermined processing on the target file to obtain multiple sub-files, including: if the target file verification is successful, performing the first predetermined processing on the target file to obtain multiple sub-files.

[0011] Optionally, after storing at least a portion of the target data from each of the temporary tables into the corresponding formal table, the method further includes at least one of the following: performing a third predetermined process on at least a portion of the data in the formal table, the third predetermined process including deletion and modification; modifying the data in the associated table associated with the formal table based on the data in the formal table; and invoking a software development kit to update the version of the formal table to trigger a cache refresh.

[0012] According to another aspect of this application, a data synchronization device under a distributed architecture is provided, comprising: a selection unit, configured to, upon obtaining a target copy file, select a target parser from a plurality of parsers to parse the target copy file according to a first file format to obtain a target file, wherein the target copy file is a copy file sent by an associated system, the target parser is configured to parse files of the first file format, the first file format being the file format of the target copy file, and the plurality of parsers being configured to parse files of different file formats; a first processing unit, configured to perform a first predetermined processing on the target file to obtain a plurality of sub-files, the first predetermined processing including splitting processing, wherein the data in each of the sub-files is target data; a first storage unit, configured to store each of the target data into a temporary table of a corresponding distributed unit, wherein the distributed unit includes a formal table and a temporary table, one formal table corresponding to at least one of the temporary tables, and one distributed unit corresponding to at least one of the target data; and a second storage unit, configured to store at least a portion of the target data in each of the temporary tables into the corresponding formal table.

[0013] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the methods described.

[0014] According to another aspect of this application, a processor is provided for running a program, wherein the program, when running, performs any of the methods described.

[0015] According to another aspect of this application, a data synchronization system is also provided, comprising: an association system for issuing a copy of a source file; and a target system connected to the association system, the target system for receiving the copy, the target system including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any one of the methods described.

[0016] By applying the technical solution of this application, when the target copy file is obtained, a target parser that can parse the file format is selected from multiple parsers according to the file format of the target copy file, and the copy file is parsed. This achieves the effect of parsing and processing files of different formats, and can be applied to data synchronization of various file formats, ensuring good universality. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a mobile terminal for implementing a data synchronization method under a distributed architecture, according to an embodiment of this application, is shown.

[0019] Figure 2 A flowchart illustrating a data synchronization method under a distributed architecture according to an embodiment of this application is shown.

[0020] Figure 3 A schematic diagram of a data synchronization process according to an embodiment of this application is shown;

[0021] Figure 4 A schematic diagram of the first-stage task flow according to an embodiment of this application is shown;

[0022] Figure 5 A schematic diagram of the second-stage task flow according to an embodiment of this application is shown;

[0023] Figure 6 A structural block diagram of a data synchronization device under a distributed architecture provided according to an embodiment of this application is shown;

[0024] Figure 7 A schematic diagram of the architecture of a target system according to an embodiment of this application is shown.

[0025] The above figures include the following reference numerals:

[0026] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] As described in the background section, existing data synchronization mechanisms between related systems only support single file processing and have poor versatility. To address the above problems, embodiments of this application provide a data synchronization method, apparatus, computer-readable storage medium, processor, and data synchronization system under a distributed architecture.

[0031] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0032] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a data synchronization method under a distributed architecture according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal. For example, the mobile terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the device information display method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the method described. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of such networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] This embodiment provides a data synchronization method running on a distributed architecture of a mobile terminal, computer terminal or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although the logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0035] Figure 2 This is a flowchart of a data synchronization method under a distributed architecture according to an embodiment of this application. The method can be applied to a target system, which is connected to an associated system for data synchronization with the associated system.

[0036] like Figure 2 As shown, the method includes the following steps:

[0037] Step S201: When the target copy file is obtained, a target parser is selected from multiple parsers to parse the target copy file according to the first file format to obtain the target file. The target copy file is a copy file sent by the associated system. The target parser is used to parse files of the first file format. The first file format is the file format of the target copy file. The multiple parsers are used to parse files of different file formats.

[0038] Specifically, the copy file is a redundant file of the source file in the associated system. In a distributed database, introducing copy files ensures the reliability of data import, while the source file is the data file in the associated system where data changes have occurred. The parser is used to parse the copy file into a file format recognizable by the target system. Since copy files have various formats, such as XML, TXT, CSV, GBK, or UTF8, multiple parsers are used to parse copy files of different formats. The associated system can be a customer information system or a user management system, and the target system is a data processing system.

[0039] In one specific embodiment, the associated system sends the copy file to a file server. The copy file can be an incremental file or a full file. The incremental file and the full file are stored on different file servers or in different regions of the file servers. The method of this application includes: retrieving the incremental file from the file server, or retrieving the full file from the file server. Compared with the prior art where the processing time of incremental and full files depends on the associated system sending them and cannot be controlled independently, the method of this application has the associated system sending the incremental file and the full file to the file server, and the target system then retrieving them from the file server. This allows for dynamic control over whether to process the incremental file or the full file, further ensuring dynamic and flexible processing of copy files and data synchronization.

[0040] The file server can be an SFTP (SSH File Transfer Protocol) server, an EDB (Enterprise Data Bus) server, or other types of servers.

[0041] Step S202: Perform a first predetermined process on the target file to obtain multiple sub-files. The first predetermined process includes splitting. The data in each sub-file is the target data.

[0042] Specifically, the splitting process is a method of splitting a large target file into multiple smaller files to facilitate subsequent data processing.

[0043] In one optional approach, the target file undergoes a first predetermined processing to obtain multiple sub-files, including: splitting the target file according to preset rules to obtain multiple intermediate files, each intermediate file corresponding to a temporary table, the preset rules including at least one of the following: splitting according to the customer ID of the data in the target file, splitting according to the database to which the data in the target file belongs, and splitting according to whether the target file is a customer-related file; and sharding each intermediate file according to the corresponding number of threads to obtain multiple sub-files corresponding to each intermediate file, the number of sub-files being the same as the corresponding number of threads.

[0044] In the described embodiment, the target file is first split according to rules such as customer number, database, and whether it is related to a customer, to obtain multiple sets of intermediate files. Then, the intermediate files are sharded according to the required number of threads, splitting them into sub-files with the same number of threads. One sub-file corresponds to one thread. Subsequently, multiple threads corresponding to the intermediate file are executed in parallel. Compared with the existing technology, which does not support dynamic file splitting and requires splitting file data based on the partitioning test of the entity table and the partitioning business fields, and placing the corresponding data into the corresponding partitioned table, this application realizes flexible control and dynamic adjustment of task concurrency.

[0045] In cases where the target file is a customer-related file, the data in the target file will have a corresponding customer number, representing the customer to which the data belongs.

[0046] Furthermore, after splitting the target file to obtain multiple intermediate files, the method further includes uploading the multiple intermediate files to HDFS (Hadoop Distributed File System). This facilitates other related systems or distributed units to retrieve these intermediate files from HDFS when needed.

[0047] To further ensure file integrity, specifically, before performing the first predetermined processing on the target file to obtain multiple sub-files, after selecting a target parser from multiple parsers according to a first file format to parse the target copy file and obtain the target file, the method further includes: performing file verification on the target file, and performing the first predetermined processing on the target file to obtain multiple sub-files, including: if the target file verification is successful, performing the first predetermined processing on the target file to obtain multiple sub-files. After parsing the target copy file into the target file, the target file is first verified to determine whether the target file is complete. Only if the target file is complete is the first predetermined processing, such as splitting, performed on the target file to further ensure the integrity of the subsequently obtained target data.

[0048] Step S203: Store each of the target data into a temporary table of the corresponding distributed unit. The distributed unit includes a formal table and a temporary table. One formal table corresponds to at least one of the temporary tables, and one distributed unit corresponds to at least one of the target data.

[0049] Specifically, the temporary table is a temporary table within the distributed unit, while the formal table is a formal table within the distributed unit. When dealing with large amounts of data in the operating table and needing to query other tables, or when the data being operated on is temporary and involves many DML operations within a short period, or when queries require joining multiple tables, directly manipulating the database tables would be inefficient. Using temporary tables can improve data processing efficiency. Furthermore, the formal table only stores a portion of the data, while the temporary table can store the original data parsed from the replica file. When the distributed unit needs to use this original data, there is no need to re-parse and split the replica file; the data can be directly read from the temporary table. The temporary table implements breakpoint resumption functionality. The distributed unit is a fundamental element of logical unitized services, comprising a set of application and database services. It provides business services for a specific, fixed set of data objects, and data is independent between different distributed units. The number of formal tables included in a distributed unit can be flexibly set according to actual needs.

[0050] To further enable flexible control over task concurrency, one option involves storing each target data item in a temporary table of the corresponding distributed unit, including: configuring each target data item corresponding to the intermediate file as a thread; and executing multiple threads corresponding to each intermediate file in parallel to store multiple target data items corresponding to each intermediate file in parallel into the corresponding temporary table.

[0051] Step S204: Store at least a portion of the target data from each of the temporary tables into the corresponding formal table.

[0052] Specifically, the part of the target data referred to in the "partial target data" can be set according to actual needs. Alternatively, it can be configured to store all the target data from the temporary table into the corresponding formal table. The formal table corresponding to the temporary table can be determined by reading the SQL ID information of the temporary table in the parameter configuration.

[0053] In one exemplary embodiment, the types of the copy files include single-table files and general-purpose table files. The single-table files represent a one-to-one correspondence between the copy files and the official tables, and the general-purpose table files represent a one-to-one correspondence between a file group and the official tables. Each file group includes multiple copy files. At least a portion of the target data from each temporary table is stored in the corresponding official table. This includes: a first processing step, where, if the target copy file is a single-table file, a portion of the target data from all temporary tables corresponding to the target copy file is stored in the corresponding official table; a determination step, where, if the target copy file is a general-purpose table file, it is determined whether the data from other copy files in the target file group is stored in the corresponding temporary table, where the target file group is the file group to which the target copy file belongs; and a second processing step, where, if the data from other copy files is stored in the corresponding temporary table, a portion of the target data from all temporary tables corresponding to the file group is stored in the corresponding official table. Compared to existing data synchronization mechanisms that do not support the simultaneous activation and rollback of a group of associated files, the embodiments described in this application, when the target copy file is a single-table file, do not require attention to the issue of simultaneous activation and rollback. In this case, the necessary data in all temporary tables corresponding to the target copy file is directly stored into the corresponding formal table. However, when the target copy file is a general-purpose file, considering the simultaneous activation mechanism, it is necessary to consider whether all the data in the entire file group has been stored in the temporary table. When all the data in the entire file group has been stored in the temporary table, the simultaneous activation condition is met. In this case, the necessary data in the file group is stored into the formal table, thus achieving the effect of multiple files activating simultaneously in the case of multiple file associations.

[0054] It should be noted that when the copy file transmitted by the associated system is a single-table file, this copy file only operates on that single table in the receiving system, i.e., the target system. This copy file has no association or dependency with other copy files. When the copy file transmitted by the associated system is a general-purpose table file, this copy file needs to be committed to the database at the same time as the other files transmitted by the associated system, and take effect or be rolled back if it fails.

[0055] Specifically, after completing the step of storing each target data into the temporary table of the corresponding distributed unit, this application records execution traces in the execution trace table. The determination step is specifically implemented as follows: based on the grouping information configured on the predetermined platform, it is determined whether there are execution traces for each copy file in the file group in the execution trace table, so as to determine whether the data of other copy files in the target file group are stored into the corresponding temporary table. The predetermined platform is a platform that stores the grouping information, and the grouping information indicates which copy files constitute a file group. The predetermined platform can be the Apollo platform. Furthermore, to avoid repeatedly executing the second processing step on the same file, after the second processing step, the execution trace of writing some data from the temporary table into the corresponding formal table is registered in the execution trace table.

[0056] To further ensure the flexibility and controllability of data storage, the first processing step further includes: performing a second predetermined processing on the target data in all the temporary tables to obtain final data; and storing the final data in the corresponding formal table. The second predetermined processing includes at least one of data splitting, data combining, and deletion. Those skilled in the art can set the specific content of the second predetermined processing according to actual needs to process the data required in the temporary tables and store the processed data in the formal table.

[0057] To further ensure that multiple associated files can take effect and roll back simultaneously, according to an optional embodiment, if the data of other copy files is not stored in the corresponding temporary table, the method further includes: performing the determining step and the second processing step at least once in sequence until a portion of the target data in all the temporary tables corresponding to the file group is stored in the corresponding formal table.

[0058] The solution described in this application proposes a relatively universal data synchronization framework that can parse, verify, split, shard, and store in tables multiple file types of copy files. The table storage process enables simultaneous operation of multiple tables under multi-file association and triggering of two-stage tasks within the same group. In addition to this universal data synchronization framework, those skilled in the art can perform personalized and additional function configurations. In one specific embodiment, after storing at least a portion of the target data from each of the temporary tables into the corresponding formal table, the method further includes at least one of the following: performing a third predetermined process on at least a portion of the data in the formal table, the third predetermined process including deletion and modification; modifying the data in the associated tables related to the formal table based on the data in the formal table; and calling the Software Development Kit (SDK) to update the version of the formal table to trigger a cache refresh.

[0059] The method described in this application embodiment also includes an exception management mechanism, which flexibly controls exceptions that occur during file processing, promptly registers abnormal data, and sends abnormal data caused by data quality issues to the associated system. This flexible exception management mechanism handles exceptions during data synchronization and manages the overall task results.

[0060] In this embodiment, upon obtaining a target copy file sent by the associated system, firstly, based on the first file format of the target copy file, a target parser corresponding to the first file format is selected from multiple parsers to parse the target copy file, obtaining the target file; then, the target file undergoes a first predetermined process including splitting, resulting in multiple sub-files; subsequently, the target data in each sub-file is stored in a temporary table of the corresponding distributed unit; finally, at least a portion of the target data in each temporary table is stored in the corresponding formal table. Compared to the existing data synchronization mechanism between associated systems, which only supports single file processing and suffers from poor versatility, this application, upon obtaining the target copy file, selects a target parser capable of parsing that file format from multiple parsers based on the file format of the target copy file, parses the copy file, and achieves the effect of parsing and processing files of different formats. This makes it applicable to data synchronization of various file formats, ensuring good versatility.

[0061] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the data synchronization method under the distributed architecture of this application will be described in detail below with reference to specific embodiments.

[0062] This embodiment relates to a specific data synchronization method under a distributed architecture, applicable to data synchronization when the replica file type is a single-table file. The replica file transmitted by the associated system only operates on the single table of the receiving party, and this replica file has no association or dependency relationship with other replica files. Figure 3 As shown, the specific process of data synchronization includes the following steps:

[0063] Step S1: Parse and split the copy file: Download the copy file (supports XML, TXT and other file formats) from the file server (SFTP / EDB), parse and verify the copy file according to the splitting rules (by customer number, by database, by whether it is related to the customer, which can be further refined into groups) to obtain the intermediate file; in addition, upload the split and verified intermediate file to HDFS, and pass the intermediate file into the batch framework context to facilitate subsequent steps to obtain and split the file;

[0064] Step S2: Store copy file data into a temporary table: Use a custom sharder (by customer number, by database, by whether it is related to a customer) to shard the intermediate file, thereby determining the number of threads that will be executed concurrently in the step of storing data into the temporary table; each shard is processed to obtain a single split sub-file, the data of the sub-file is read, and it is stored in the temporary table in batch (Batch Insert).

[0065] Step S3: Data from replica files is written to the official table: Based on the SQL ID of the temporary table read according to the parameter configuration, the data processing and storage into the official table are implemented by each replica component (for some replica files with large amounts of data, the Merge Into syntax is used to improve performance in scenarios with existing data).

[0066] Step S4: Custom extension of replica files: Steps for each replica component to customize extensions, such as cleaning up certain data, modifying related data in the associated tables of the original table, etc.

[0067] Step S5: Update the cache of the copy file: After the data of the copy file of the parameter library is stored in the official table, the software development kit of the parameter service needs to be called to update the version of the current parameter table and trigger the parameter cache refresh.

[0068] This embodiment relates to a specific data synchronization method under a distributed architecture, applicable to data synchronization when the replica file type is a general table file. Multiple files transmitted by the associated system need to be successfully written to the database or rolled back upon failure at the same time. The files sent by the associated system are divided into two batch task processes. The specific data processing process includes the following steps:

[0069] The first phase of the task flow stores the data in a temporary table, such as... Figure 4 As shown:

[0070] Step S1', Parse and split the copy file: Download the copy file (supports XML, TXT and other file formats) from the file server (SFTP / EDB), parse and verify the copy file according to the splitting rules (by customer number, by database, by whether it is related to the customer, which can be further refined into groups) to obtain the intermediate file; in addition, upload the split and verified intermediate file to HDFS, and pass the intermediate file into the batch framework context to facilitate subsequent steps to obtain and split the file;

[0071] Step S2', Copy file data is stored in a temporary table: A custom sharder (by customer number, by database, by whether it is related to a customer) is used to shard the intermediate file, thereby determining the number of threads that will be executed concurrently in the step of storing the data in the temporary table; each shard is processed to obtain a single split sub-file, the data of the sub-file is read, and it is stored in the temporary table in batch (Batch Insert).

[0072] Step S3': Check if the second phase scheduling has been triggered: Register the execution trace of the first phase of the current replica file. According to the group information configured on Apollo, check the records in the first phase execution trace table to confirm whether all replica files in this group have been executed successfully that day. If there are still unsuccessful ones, ignore the logic of triggering the second phase task and wait for the subsequent replica files to be executed successfully before triggering the second phase. If all replica files under the group have been executed successfully, trigger the second phase task and register the second phase trigger trace to avoid repeatedly triggering the second phase task.

[0073] The second phase of the task process involves transferring the data from the temporary tables corresponding to multiple copy files to the corresponding official tables, such as... Figure 5 As shown:

[0074] Step S4': Data from the replica file is stored in the official table: Based on the SQL ID of the temporary table read according to the parameter configuration, the data processing and storage into the official table are implemented by each replica component (for some replica files with large amounts of data, the Merge Into syntax is used to improve performance in scenarios with existing data).

[0075] Step S5', Custom Extension of Copy Files: Steps for each copy component to customize extensions, such as cleaning up certain data, modifying related data in the associated tables of the original table, etc.

[0076] Step S6', Copy file update cache: After the data of the copy file of the parameter library is stored in the official table, the software development kit of the parameter service needs to be called to update the version of the current parameter table and trigger the parameter cache refresh.

[0077] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0078] This application also provides a data synchronization device under a distributed architecture. It should be noted that the data synchronization device under a distributed architecture in this application can be used to execute the data synchronization method for a distributed architecture provided in this application. This device is used to implement the embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0079] The following describes the data synchronization device under the distributed architecture provided in the embodiments of this application.

[0080] Figure 6 This is a schematic diagram of a data synchronization device under a distributed architecture according to an embodiment of this application. The device can be applied to a target system, which is connected to an associated system for data synchronization with the associated system.

[0081] like Figure 6 As shown, the device includes:

[0082] Selection unit 10 is used to select a target parser from multiple parsers to parse the target copy file according to a first file format when a target copy file is obtained, so as to obtain a target file. The target copy file is a copy file sent by the associated system. The target parser is used to parse files of the first file format. The first file format is the file format of the target copy file. The multiple parsers are used to parse files of different file formats.

[0083] Specifically, the copy file is a redundant file of the source file in the associated system. In a distributed database, introducing copy files ensures the reliability of data import, while the source file is the data file in the associated system where data changes have occurred. The parser is used to parse the copy file into a file format recognizable by the target system. Since copy files have various formats, such as XML, TXT, CSV, GBK, or UTF8, multiple parsers are used to parse copy files of different formats. The associated system can be a customer information system or a user management system, and the target system is a data processing system.

[0084] In one specific embodiment, the association system sends the copy file to a file server. The copy file can be an incremental file or a full file. The incremental file and the full file are stored on different file servers or in different regions of the file servers. The apparatus of this application includes an acquisition unit, used to acquire the incremental file or the full file from the file server. Compared with the prior art where the processing time of incremental and full files depends on the association system and cannot be controlled independently, the apparatus of this application allows the association system to send the incremental and full files to the file server, and the target system to acquire them from the file server. This allows for dynamic control over whether to process the incremental file or the full file, further ensuring dynamic and flexible processing of copy files and data synchronization.

[0085] The file server can be an SFTP server, an EDB server, or other types of servers.

[0086] The first processing unit 20 is used to perform a first predetermined processing on the target file to obtain multiple sub-files. The first predetermined processing includes splitting processing, and the data in each of the sub-files is the target data.

[0087] Specifically, the splitting process is a method of splitting a large target file into multiple smaller files to facilitate subsequent data processing.

[0088] In one optional embodiment, the first processing unit includes: a splitting module, configured to split the target file according to preset rules to obtain multiple intermediate files, each intermediate file corresponding to a temporary table, the preset rules including at least one of the following: splitting according to the customer number of the data in the target file, splitting according to the database to which the data in the target file belongs, and splitting according to whether the target file is a customer-related file; and a sharding module, configured to shard each intermediate file according to the corresponding number of threads, thereby obtaining multiple sub-files corresponding to each intermediate file, the number of sub-files being the same as the corresponding number of threads.

[0089] In the described embodiment, the target file is first split according to rules such as customer number, database, and whether it is related to a customer, to obtain multiple sets of intermediate files. Then, the intermediate files are sharded according to the required number of threads, splitting them into sub-files with the same number of threads. One sub-file corresponds to one thread. Subsequently, multiple threads corresponding to the intermediate file are executed in parallel. Compared with the existing technology, which does not support dynamic file splitting and requires splitting file data based on the partitioning test of the entity table and the partitioning business fields, and placing the corresponding data into the corresponding partitioned table, this application realizes flexible control and dynamic adjustment of task concurrency.

[0090] In cases where the target file is a customer-related file, the data in the target file will have a corresponding customer number, representing the customer to which the data belongs.

[0091] Furthermore, the apparatus also includes an upload unit, configured to upload multiple intermediate files to HDFS after splitting the target file to obtain multiple intermediate files. This facilitates other related systems or distributed units in retrieving these intermediate files from HDFS when needed.

[0092] To further ensure file integrity, the apparatus further includes: a verification unit, configured to, before performing the first predetermined processing on the target file to obtain multiple sub-files, select a target parser from multiple parsers according to a first file format to parse the target copy file, and after obtaining the target file, perform file verification on the target file. The first processing unit includes: a first processing module, configured to, if the target file verification is successful, perform the first predetermined processing on the target file to obtain multiple sub-files. After parsing the target copy file into the target file, the target file is first verified to determine whether the target file is complete. Only if the target file is complete will the first predetermined processing, such as splitting, continue to be performed on the target file, further ensuring the integrity of the subsequently obtained target data.

[0093] The first storage unit 30 is used to store each of the target data into a temporary table of the corresponding distributed unit. The distributed unit includes a formal table and a temporary table. One formal table corresponds to at least one of the temporary tables, and one distributed unit corresponds to at least one of the target data.

[0094] Specifically, the temporary table is a temporary table within the distributed unit, while the formal table is a formal table within the distributed unit. When dealing with large amounts of data in the operating table and needing to query other tables, or when the data being operated on is temporary and involves many DML operations within a short period, or when queries require joining multiple tables, directly manipulating the database tables would be inefficient. Using temporary tables can improve data processing efficiency. Furthermore, the formal table only stores a portion of the data, while the temporary table can store the original data parsed from the replica file. When the distributed unit needs to use this original data, there is no need to re-parse and split the replica file; the data can be directly read from the temporary table. The temporary table implements breakpoint resumption functionality. The distributed unit is a fundamental element of logical unitized services, comprising a set of application and database services. It provides business services for a specific, fixed set of data objects, and data is independent between different distributed units. The number of formal tables included in a distributed unit can be flexibly set according to actual needs.

[0095] To further enable flexible control over task concurrency, in one option, the first storage unit includes: a configuration module, configured to configure each target data corresponding to the intermediate file as a thread; and an execution module, configured to execute multiple threads corresponding to each intermediate file in parallel, so as to store multiple target data corresponding to each intermediate file in parallel into the corresponding temporary table.

[0096] The second storage unit 40 is used to store at least a portion of the target data from each of the temporary tables into the corresponding formal table.

[0097] Specifically, the part of the target data referred to in the "partial target data" can be set according to actual needs. Alternatively, it can be configured to store all the target data from the temporary table into the corresponding formal table. The formal table corresponding to the temporary table can be determined by reading the SQL ID information of the temporary table in the parameter configuration.

[0098] In one exemplary embodiment, the types of the copy files include single-table files and general-purpose table files. The single-table files represent a one-to-one correspondence between the copy files and the official tables, and the general-purpose table files represent a one-to-one correspondence between a file group and the official tables. The file group includes multiple copy files. The second storage unit includes: a second processing module, used in a first processing step, where, if the target copy file is of the single-table type, a portion of the target data from all the temporary tables corresponding to the target copy file is stored into the corresponding official table; a determining module, used in a determining step, where, if the target copy file is of the general-purpose table type, it determines whether the data of other copy files in the target file group is stored into the corresponding temporary tables, where the target file group is the file group to which the target copy file belongs; and a third processing module, used in a second processing step, where, if the data of other copy files is stored into the corresponding temporary tables, a portion of the target data from all the temporary tables corresponding to the file group is stored into the corresponding official table. Compared to existing data synchronization mechanisms that do not support the simultaneous activation and rollback of a group of associated files, the embodiments described in this application, when the target copy file is a single-table file, do not require attention to the issue of simultaneous activation and rollback. In this case, the necessary data in all temporary tables corresponding to the target copy file is directly stored into the corresponding formal table. However, when the target copy file is a general-purpose file, considering the simultaneous activation mechanism, it is necessary to consider whether all the data in the entire file group has been stored in the temporary table. When all the data in the entire file group has been stored in the temporary table, the simultaneous activation condition is met. In this case, the necessary data in the file group is stored into the formal table, thus achieving the effect of multiple files activating simultaneously in the case of multiple file associations.

[0099] It should be noted that when the copy file transmitted by the associated system is a single-table file, this copy file only operates on that single table in the receiving system, i.e., the target system. This copy file has no association or dependency with other copy files. When the copy file transmitted by the associated system is a general-purpose table file, this copy file needs to be committed to the database at the same time as the other files transmitted by the associated system, and take effect or be rolled back if it fails.

[0100] Specifically, after completing the step of storing each target data into the temporary table of the corresponding distributed unit, this application records execution traces in the execution trace table. The determining module specifically includes a determining submodule, used to determine whether there are execution traces for each copy file in the file group in the execution trace table based on the grouping information configured on the predetermined platform, so as to determine whether the data of other copy files in the target file group are stored into the corresponding temporary table. The predetermined platform is a platform that stores the grouping information, and the grouping information indicates which copy files constitute a file group. The predetermined platform can be the Apollo platform. Furthermore, to avoid repeatedly executing the second processing step on the same file, after the second processing step, the execution traces of writing some data from the temporary table into the corresponding formal table are registered in the execution trace table.

[0101] To further ensure the flexibility and controllability of data storage, the second processing module further includes: a processing submodule, used to perform a second predetermined processing on the target data in all the temporary tables to obtain final data; and a storage submodule, used to store the final data into the corresponding formal table. The second predetermined processing includes at least one of data splitting, data combining, and deletion. Those skilled in the art can set the specific content of the second predetermined processing according to actual needs to process the data required in the temporary tables and store the processed data into the formal table.

[0102] To further ensure that multiple associated files can take effect and roll back simultaneously, according to an optional embodiment, the apparatus further includes: an execution unit, configured to, when the data of other copy files has not been stored in the corresponding temporary table, sequentially execute the determining step and the second processing step at least once, until a portion of the target data in all the temporary tables corresponding to the file group is stored in the corresponding formal table.

[0103] The solution described in this application proposes a relatively universal data synchronization framework that can parse, verify, split, shard, and store multiple file types of copies in tables. The table storage process enables simultaneous operation of multiple tables under multi-file association and triggering of two-stage tasks within the same group. In addition to this universal data synchronization framework, those skilled in the art can perform personalized and additional function configurations. In one specific embodiment, the device further includes at least one of the following: a second processing unit, configured to perform a third predetermined processing on at least a portion of the data in the formal table after storing at least a portion of the target data from each of the temporary tables into the corresponding formal table; the third predetermined processing including deletion and modification; a modification unit, configured to modify the data in the associated tables related to the formal table based on the data in the formal table; and a calling unit, configured to call a software development kit to update the version of the formal table to trigger a cache refresh.

[0104] The device described in this application embodiment also includes an exception management mechanism, which flexibly manages exceptions that occur during the file processing process, promptly registers abnormal data, and sends abnormal data caused by data quality issues to the associated system. This flexible exception management mechanism handles exceptions during the data synchronization process and manages the overall task results.

[0105] In this embodiment, upon obtaining a target copy file sent by the associated system, a selection unit selects a target parser corresponding to the first file format from multiple parsers to parse the target copy file, obtaining a target file. A first processing unit performs a first predetermined process, including splitting, on the target file to obtain multiple sub-files. A first storage unit stores the target data from each sub-file into a temporary table of the corresponding distributed unit. A second storage unit stores at least a portion of the target data from each temporary table into the corresponding formal table. Compared to existing data synchronization mechanisms between associated systems that only support single-file processing, resulting in poor versatility, this application, upon obtaining a target copy file, selects a target parser capable of parsing that file format from multiple parsers, parses the copy file, and achieves the effect of parsing and processing files of different formats. This makes it applicable to data synchronization of various file formats, ensuring good versatility.

[0106] The data synchronization device under the distributed architecture includes a processor and a memory. The selection unit, the first processing unit, the first storage unit, and the second storage unit are all stored as program units in the memory. The processor executes the program units stored in the memory to implement the corresponding functions. All modules are located in the same processor; or, the modules are located in different processors in any combination.

[0107] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, the problem of existing data synchronization mechanisms between related systems only supporting single-file processing and lacking versatility can be addressed.

[0108] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0109] This invention provides a computer-readable storage medium including a stored program, wherein the program, when running, controls the device where the computer-readable storage medium is located to execute a data synchronization method under the distributed architecture.

[0110] Specifically, data synchronization methods in a distributed architecture include:

[0111] Step S201: When the target copy file is obtained, a target parser is selected from multiple parsers to parse the target copy file according to the first file format to obtain the target file. The target copy file is a copy file sent by the associated system. The target parser is used to parse files of the first file format. The first file format is the file format of the target copy file. The multiple parsers are used to parse files of different file formats.

[0112] Specifically, the copy file is a redundant file of the source file in the associated system. In a distributed database, introducing copy files ensures the reliability of data import, while the source file is the data file in the associated system where data changes have occurred. The parser is used to parse the copy file into a file format recognizable by the target system. Since copy files have various formats, such as XML, TXT, CSV, GBK, or UTF8, multiple parsers are used to parse copy files of different formats. The associated system can be a customer information system or a user management system, and the target system is a data processing system.

[0113] Step S202: Perform a first predetermined process on the target file to obtain multiple sub-files. The first predetermined process includes splitting. The data in each sub-file is the target data.

[0114] Specifically, the splitting process is a method of splitting a large target file into multiple smaller files to facilitate subsequent data processing.

[0115] Step S203: Store each of the target data into a temporary table of the corresponding distributed unit. The distributed unit includes a formal table and a temporary table. One formal table corresponds to at least one of the temporary tables, and one distributed unit corresponds to at least one of the target data.

[0116] Specifically, the temporary table is a temporary table within the distributed unit, while the formal table is a formal table within the distributed unit. When dealing with large amounts of data in the operating table and needing to query other tables, or when the data being operated on is temporary and involves many DML operations within a short period, or when queries require joining multiple tables, directly manipulating the database tables would be inefficient. Using temporary tables can improve data processing efficiency. Furthermore, the formal table only stores a portion of the data, while the temporary table can store the original data parsed from the replica file. When the distributed unit needs to use this original data, there is no need to re-parse and split the replica file; the data can be directly read from the temporary table. The temporary table implements breakpoint resumption functionality. The distributed unit is a fundamental element of logical unitized services, comprising a set of application and database services. It provides business services for a specific, fixed set of data objects, and data is independent between different distributed units. The number of formal tables included in a distributed unit can be flexibly set according to actual needs.

[0117] Step S204: Store at least a portion of the target data from each of the temporary tables into the corresponding formal table.

[0118] Specifically, the part of the target data referred to in the "partial target data" can be set according to actual needs. Alternatively, it can be configured to store all the target data from the temporary table into the corresponding formal table. The formal table corresponding to the temporary table can be determined by reading the SQL ID information of the temporary table in the parameter configuration.

[0119] Optionally, the target file undergoes a first predetermined processing to obtain multiple sub-files, including: splitting the target file according to preset rules to obtain multiple intermediate files, each intermediate file corresponding to a temporary table, the preset rules including at least one of the following: splitting according to the customer number of the data in the target file, splitting according to the database to which the data in the target file belongs, splitting according to whether the target file is a customer-related file; and sharding each intermediate file according to the corresponding number of threads to obtain multiple sub-files corresponding to each intermediate file, the number of sub-files being the same as the corresponding number of threads.

[0120] Optionally, storing each of the target data into a temporary table of the corresponding distributed unit includes: configuring each of the target data corresponding to the intermediate file as a thread; executing multiple threads corresponding to each intermediate file in parallel to store multiple target data corresponding to each intermediate file into the corresponding temporary table in parallel.

[0121] Optionally, the type of the copy file includes single-table files and general-purpose table files. The single-table file indicates that the copy file corresponds one-to-one with the official table, and the general-purpose table file indicates that a file group corresponds one-to-one with the official table. The file group includes multiple copy files. At least a portion of the target data in each temporary table is stored into the corresponding official table, including: a first processing step, in the case that the type of the target copy file is a single-table file, storing a portion of the target data in all the temporary tables corresponding to the target copy file into the corresponding official table; a determination step, in the case that the type of the target copy file is a general-purpose table file, determining whether the data of other copy files in the target file group is stored into the corresponding temporary table, where the target file group is the file group to which the target copy file belongs; and a second processing step, in the case that the data of other copy files is stored into the corresponding temporary table, storing a portion of the target data in all the temporary tables corresponding to the file group into the corresponding official table.

[0122] Optionally, the first processing step includes: performing a second predetermined processing on the target data in all the temporary tables to obtain final data; and storing the final data in the corresponding formal table, wherein the second predetermined processing includes at least one of data splitting, data combining, and deletion.

[0123] Optionally, if the data of other copy files is not stored in the corresponding temporary table, the method further includes: performing the determining step and the second processing step at least once in sequence until a portion of the target data in all the temporary tables corresponding to the file group is stored in the corresponding formal table.

[0124] Optionally, before performing the first predetermined processing on the target file to obtain multiple sub-files, after selecting a target parser from multiple parsers according to the first file format to parse the target copy file and obtain the target file, the method further includes: performing file verification on the target file, and performing the first predetermined processing on the target file to obtain multiple sub-files, including: if the target file verification is successful, performing the first predetermined processing on the target file to obtain multiple sub-files.

[0125] Optionally, after storing at least a portion of the target data from each of the temporary tables into the corresponding formal table, the method further includes at least one of the following: performing a third predetermined process on at least a portion of the data in the formal table, the third predetermined process including deletion and modification; modifying the data in the associated table associated with the formal table based on the data in the formal table; and invoking a software development kit to update the version of the formal table to trigger a cache refresh.

[0126] This invention provides a processor for running a program, wherein the program executes a data synchronization method under the distributed architecture.

[0127] Specifically, data synchronization methods in a distributed architecture include:

[0128] Step S201: When the target copy file is obtained, a target parser is selected from multiple parsers to parse the target copy file according to the first file format to obtain the target file. The target copy file is a copy file sent by the associated system. The target parser is used to parse files of the first file format. The first file format is the file format of the target copy file. The multiple parsers are used to parse files of different file formats.

[0129] Specifically, the copy file is a redundant file of the source file in the associated system. In a distributed database, introducing copy files ensures the reliability of data import, while the source file is the data file in the associated system where data changes have occurred. The parser is used to parse the copy file into a file format recognizable by the target system. Since copy files have various formats, such as XML, TXT, CSV, GBK, or UTF8, multiple parsers are used to parse copy files of different formats. The associated system can be a customer information system or a user management system, and the target system is a data processing system.

[0130] Step S202: Perform a first predetermined process on the target file to obtain multiple sub-files. The first predetermined process includes splitting. The data in each sub-file is the target data.

[0131] Specifically, the splitting process is a method of splitting a large target file into multiple smaller files to facilitate subsequent data processing.

[0132] Step S203: Store each of the target data into a temporary table of the corresponding distributed unit. The distributed unit includes a formal table and a temporary table. One formal table corresponds to at least one of the temporary tables, and one distributed unit corresponds to at least one of the target data.

[0133] Specifically, the temporary table is a temporary table within the distributed unit, while the formal table is a formal table within the distributed unit. When dealing with large amounts of data in the operating table and needing to query other tables, or when the data being operated on is temporary and involves many DML operations within a short period, or when queries require joining multiple tables, directly manipulating the database tables would be inefficient. Using temporary tables can improve data processing efficiency. Furthermore, the formal table only stores a portion of the data, while the temporary table can store the original data parsed from the replica file. When the distributed unit needs to use this original data, there is no need to re-parse and split the replica file; the data can be directly read from the temporary table. The temporary table implements breakpoint resumption functionality. The distributed unit is a fundamental element of logical unitized services, comprising a set of application and database services. It provides business services for a specific, fixed set of data objects, and data is independent between different distributed units. The number of formal tables included in a distributed unit can be flexibly set according to actual needs.

[0134] Step S204: Store at least a portion of the target data from each of the temporary tables into the corresponding formal table.

[0135] Specifically, the part of the target data referred to in the "partial target data" can be set according to actual needs. Alternatively, it can be configured to store all the target data from the temporary table into the corresponding formal table. The formal table corresponding to the temporary table can be determined by reading the SQL ID information of the temporary table in the parameter configuration.

[0136] Optionally, the target file undergoes a first predetermined processing to obtain multiple sub-files, including: splitting the target file according to preset rules to obtain multiple intermediate files, each intermediate file corresponding to a temporary table, the preset rules including at least one of the following: splitting according to the customer number of the data in the target file, splitting according to the database to which the data in the target file belongs, splitting according to whether the target file is a customer-related file; and sharding each intermediate file according to the corresponding number of threads to obtain multiple sub-files corresponding to each intermediate file, the number of sub-files being the same as the corresponding number of threads.

[0137] Optionally, storing each of the target data into a temporary table of the corresponding distributed unit includes: configuring each of the target data corresponding to the intermediate file as a thread; executing multiple threads corresponding to each intermediate file in parallel to store multiple target data corresponding to each intermediate file into the corresponding temporary table in parallel.

[0138] Optionally, the type of the copy file includes single-table files and general-purpose table files. The single-table file indicates that the copy file corresponds one-to-one with the official table, and the general-purpose table file indicates that a file group corresponds one-to-one with the official table. The file group includes multiple copy files. At least a portion of the target data in each temporary table is stored into the corresponding official table, including: a first processing step, in the case that the type of the target copy file is a single-table file, storing a portion of the target data in all the temporary tables corresponding to the target copy file into the corresponding official table; a determination step, in the case that the type of the target copy file is a general-purpose table file, determining whether the data of other copy files in the target file group is stored into the corresponding temporary table, where the target file group is the file group to which the target copy file belongs; and a second processing step, in the case that the data of other copy files is stored into the corresponding temporary table, storing a portion of the target data in all the temporary tables corresponding to the file group into the corresponding official table.

[0139] Optionally, the first processing step includes: performing a second predetermined processing on the target data in all the temporary tables to obtain final data; and storing the final data in the corresponding formal table, wherein the second predetermined processing includes at least one of data splitting, data combining, and deletion.

[0140] Optionally, if the data of other copy files is not stored in the corresponding temporary table, the method further includes: performing the determining step and the second processing step at least once in sequence until a portion of the target data in all the temporary tables corresponding to the file group is stored in the corresponding formal table.

[0141] Optionally, before performing the first predetermined processing on the target file to obtain multiple sub-files, after selecting a target parser from multiple parsers according to the first file format to parse the target copy file and obtain the target file, the method further includes: performing file verification on the target file, and performing the first predetermined processing on the target file to obtain multiple sub-files, including: if the target file verification is successful, performing the first predetermined processing on the target file to obtain multiple sub-files.

[0142] Optionally, after storing at least a portion of the target data from each of the temporary tables into the corresponding formal table, the method further includes at least one of the following: performing a third predetermined process on at least a portion of the data in the formal table, the third predetermined process including deletion and modification; modifying the data in the associated table associated with the formal table based on the data in the formal table; and invoking a software development kit to update the version of the formal table to trigger a cache refresh.

[0143] This invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0144] Step S201: When the target copy file is obtained, a target parser is selected from multiple parsers to parse the target copy file according to the first file format to obtain the target file. The target copy file is a copy file sent by the associated system. The target parser is used to parse files of the first file format. The first file format is the file format of the target copy file. The multiple parsers are used to parse files of different file formats.

[0145] Step S202: Perform a first predetermined process on the target file to obtain multiple sub-files. The first predetermined process includes splitting. The data in each sub-file is the target data.

[0146] Step S203: Store each of the target data into a temporary table of the corresponding distributed unit. The distributed unit includes a formal table and a temporary table. One formal table corresponds to at least one of the temporary tables, and one distributed unit corresponds to at least one of the target data.

[0147] Step S204: Store at least a portion of the target data from each of the temporary tables into the corresponding formal table.

[0148] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0149] Optionally, the target file undergoes a first predetermined processing to obtain multiple sub-files, including: splitting the target file according to preset rules to obtain multiple intermediate files, each intermediate file corresponding to a temporary table, the preset rules including at least one of the following: splitting according to the customer number of the data in the target file, splitting according to the database to which the data in the target file belongs, splitting according to whether the target file is a customer-related file; and sharding each intermediate file according to the corresponding number of threads to obtain multiple sub-files corresponding to each intermediate file, the number of sub-files being the same as the corresponding number of threads.

[0150] Optionally, storing each of the target data into a temporary table of the corresponding distributed unit includes: configuring each of the target data corresponding to the intermediate file as a thread; executing multiple threads corresponding to each intermediate file in parallel to store multiple target data corresponding to each intermediate file into the corresponding temporary table in parallel.

[0151] Optionally, the type of the copy file includes single-table files and general-purpose table files. The single-table file indicates that the copy file corresponds one-to-one with the official table, and the general-purpose table file indicates that a file group corresponds one-to-one with the official table. The file group includes multiple copy files. At least a portion of the target data in each temporary table is stored into the corresponding official table, including: a first processing step, in the case that the type of the target copy file is a single-table file, storing a portion of the target data in all the temporary tables corresponding to the target copy file into the corresponding official table; a determination step, in the case that the type of the target copy file is a general-purpose table file, determining whether the data of other copy files in the target file group is stored into the corresponding temporary table, where the target file group is the file group to which the target copy file belongs; and a second processing step, in the case that the data of other copy files is stored into the corresponding temporary table, storing a portion of the target data in all the temporary tables corresponding to the file group into the corresponding official table.

[0152] Optionally, the first processing step includes: performing a second predetermined processing on the target data in all the temporary tables to obtain final data; and storing the final data in the corresponding formal table, wherein the second predetermined processing includes at least one of data splitting, data combining, and deletion.

[0153] Optionally, if the data of other copy files is not stored in the corresponding temporary table, the method further includes: performing the determining step and the second processing step at least once in sequence until a portion of the target data in all the temporary tables corresponding to the file group is stored in the corresponding formal table.

[0154] Optionally, before performing the first predetermined processing on the target file to obtain multiple sub-files, after selecting a target parser from multiple parsers according to the first file format to parse the target copy file and obtain the target file, the method further includes: performing file verification on the target file, and performing the first predetermined processing on the target file to obtain multiple sub-files, including: if the target file verification is successful, performing the first predetermined processing on the target file to obtain multiple sub-files.

[0155] Optionally, after storing at least a portion of the target data from each of the temporary tables into the corresponding formal table, the method further includes at least one of the following: performing a third predetermined process on at least a portion of the data in the formal table, the third predetermined process including deletion and modification; modifying the data in the associated table associated with the formal table based on the data in the formal table; and invoking a software development kit to update the version of the formal table to trigger a cache refresh.

[0156] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0157] Step S201: When the target copy file is obtained, a target parser is selected from multiple parsers to parse the target copy file according to the first file format to obtain the target file. The target copy file is a copy file sent by the associated system. The target parser is used to parse files of the first file format. The first file format is the file format of the target copy file. The multiple parsers are used to parse files of different file formats.

[0158] Step S202: Perform a first predetermined process on the target file to obtain multiple sub-files. The first predetermined process includes splitting. The data in each sub-file is the target data.

[0159] Step S203: Store each of the target data into a temporary table of the corresponding distributed unit. The distributed unit includes a formal table and a temporary table. One formal table corresponds to at least one of the temporary tables, and one distributed unit corresponds to at least one of the target data.

[0160] Step S204: Store at least a portion of the target data from each of the temporary tables into the corresponding formal table.

[0161] Optionally, the target file undergoes a first predetermined processing to obtain multiple sub-files, including: splitting the target file according to preset rules to obtain multiple intermediate files, each intermediate file corresponding to a temporary table, the preset rules including at least one of the following: splitting according to the customer number of the data in the target file, splitting according to the database to which the data in the target file belongs, splitting according to whether the target file is a customer-related file; and sharding each intermediate file according to the corresponding number of threads to obtain multiple sub-files corresponding to each intermediate file, the number of sub-files being the same as the corresponding number of threads.

[0162] Optionally, storing each of the target data into a temporary table of the corresponding distributed unit includes: configuring each of the target data corresponding to the intermediate file as a thread; executing multiple threads corresponding to each intermediate file in parallel to store multiple target data corresponding to each intermediate file into the corresponding temporary table in parallel.

[0163] Optionally, the type of the copy file includes single-table files and general-purpose table files. The single-table file indicates that the copy file corresponds one-to-one with the official table, and the general-purpose table file indicates that a file group corresponds one-to-one with the official table. The file group includes multiple copy files. At least a portion of the target data in each temporary table is stored into the corresponding official table, including: a first processing step, in the case that the type of the target copy file is a single-table file, storing a portion of the target data in all the temporary tables corresponding to the target copy file into the corresponding official table; a determination step, in the case that the type of the target copy file is a general-purpose table file, determining whether the data of other copy files in the target file group is stored into the corresponding temporary table, where the target file group is the file group to which the target copy file belongs; and a second processing step, in the case that the data of other copy files is stored into the corresponding temporary table, storing a portion of the target data in all the temporary tables corresponding to the file group into the corresponding official table.

[0164] Optionally, the first processing step includes: performing a second predetermined processing on the target data in all the temporary tables to obtain final data; and storing the final data in the corresponding formal table, wherein the second predetermined processing includes at least one of data splitting, data combining, and deletion.

[0165] Optionally, if the data of other copy files is not stored in the corresponding temporary table, the method further includes: performing the determining step and the second processing step at least once in sequence until a portion of the target data in all the temporary tables corresponding to the file group is stored in the corresponding formal table.

[0166] Optionally, before performing the first predetermined processing on the target file to obtain multiple sub-files, after selecting a target parser from multiple parsers according to the first file format to parse the target copy file and obtain the target file, the method further includes: performing file verification on the target file, and performing the first predetermined processing on the target file to obtain multiple sub-files, including: if the target file verification is successful, performing the first predetermined processing on the target file to obtain multiple sub-files.

[0167] Optionally, after storing at least a portion of the target data from each of the temporary tables into the corresponding formal table, the method further includes at least one of the following: performing a third predetermined process on at least a portion of the data in the formal table, the third predetermined process including deletion and modification; modifying the data in the associated table associated with the formal table based on the data in the formal table; and invoking a software development kit to update the version of the formal table to trigger a cache refresh.

[0168] This invention provides a data synchronization system, comprising: an association system for issuing a copy of a source file; and a target system connected to the association system, the target system for receiving the copy, the target system including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the processors, when executing the programs, implement at least the following steps:

[0169] Step S201: When the target copy file is obtained, a target parser is selected from multiple parsers to parse the target copy file according to the first file format to obtain the target file. The target copy file is a copy file sent by the associated system. The target parser is used to parse files of the first file format. The first file format is the file format of the target copy file. The multiple parsers are used to parse files of different file formats.

[0170] Step S202: Perform a first predetermined process on the target file to obtain multiple sub-files. The first predetermined process includes splitting. The data in each sub-file is the target data.

[0171] Step S203: Store each of the target data into a temporary table of the corresponding distributed unit. The distributed unit includes a formal table and a temporary table. One formal table corresponds to at least one of the temporary tables, and one distributed unit corresponds to at least one of the target data.

[0172] Step S204: Store at least a portion of the target data from each of the temporary tables into the corresponding formal table.

[0173] Optionally, the target file undergoes a first predetermined processing to obtain multiple sub-files, including: splitting the target file according to preset rules to obtain multiple intermediate files, each intermediate file corresponding to a temporary table, the preset rules including at least one of the following: splitting according to the customer number of the data in the target file, splitting according to the database to which the data in the target file belongs, splitting according to whether the target file is a customer-related file; and sharding each intermediate file according to the corresponding number of threads to obtain multiple sub-files corresponding to each intermediate file, the number of sub-files being the same as the corresponding number of threads.

[0174] Optionally, storing each of the target data into a temporary table of the corresponding distributed unit includes: configuring each of the target data corresponding to the intermediate file as a thread; executing multiple threads corresponding to each intermediate file in parallel to store multiple target data corresponding to each intermediate file into the corresponding temporary table in parallel.

[0175] Optionally, the type of the copy file includes single-table files and general-purpose table files. The single-table file indicates that the copy file corresponds one-to-one with the official table, and the general-purpose table file indicates that a file group corresponds one-to-one with the official table. The file group includes multiple copy files. At least a portion of the target data in each temporary table is stored into the corresponding official table, including: a first processing step, in the case that the type of the target copy file is a single-table file, storing a portion of the target data in all the temporary tables corresponding to the target copy file into the corresponding official table; a determination step, in the case that the type of the target copy file is a general-purpose table file, determining whether the data of other copy files in the target file group is stored into the corresponding temporary table, where the target file group is the file group to which the target copy file belongs; and a second processing step, in the case that the data of other copy files is stored into the corresponding temporary table, storing a portion of the target data in all the temporary tables corresponding to the file group into the corresponding official table.

[0176] Optionally, the first processing step includes: performing a second predetermined processing on the target data in all the temporary tables to obtain final data; and storing the final data in the corresponding formal table, wherein the second predetermined processing includes at least one of data splitting, data combining, and deletion.

[0177] Optionally, if the data of other copy files is not stored in the corresponding temporary table, the method further includes: performing the determining step and the second processing step at least once in sequence until a portion of the target data in all the temporary tables corresponding to the file group is stored in the corresponding formal table.

[0178] Optionally, before performing the first predetermined processing on the target file to obtain multiple sub-files, after selecting a target parser from multiple parsers according to the first file format to parse the target copy file and obtain the target file, the method further includes: performing file verification on the target file, and performing the first predetermined processing on the target file to obtain multiple sub-files, including: if the target file verification is successful, performing the first predetermined processing on the target file to obtain multiple sub-files.

[0179] Optionally, after storing at least a portion of the target data from each of the temporary tables into the corresponding formal table, the method further includes at least one of the following: performing a third predetermined process on at least a portion of the data in the formal table, the third predetermined process including deletion and modification; modifying the data in the associated table associated with the formal table based on the data in the formal table; and invoking a software development kit to update the version of the formal table to trigger a cache refresh.

[0180] In one specific embodiment, the architecture of the target system is as follows: Figure 7 As shown, the main file controller receives file synchronization messages from associated systems and controls the overall processing of the entire file. Batch scheduling controls the entire task flow, determining the number of subtasks and their execution order based on pre-configuration. Batch master control determines which physical execution machine each task node runs on. File download provides a multi-channel file download mechanism. File backup allows for timely backup of source files and files requiring backup. File parsing supports parsing and validation of XML, TXT, and CSV formats. File splitting splits data based on business attributes. Data sharding, file task grouping, and concurrent control of group task triggering are also included. Exception skipping, exception data registration, and exception data distribution provide an exception management mechanism, flexibly managing exceptions occurring during file processing and promptly registering exception data. Exception data caused by data quality issues is distributed to associated systems. The temporary file table manages a group of associated files, ensuring simultaneous effectiveness of the group and providing a cache refresh mechanism for the parameter cache table.

[0181] It will be apparent to those skilled in the art that the modules or steps of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using device-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.

[0182] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0184] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0186] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0187] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0188] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0189] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0190] As can be seen from the above description, the embodiments described in this application achieve the following technical effects:

[0191] 1) In the data synchronization method under the distributed architecture described in this application, upon obtaining a target copy file sent by the associated system, firstly, based on the first file format of the target copy file, a target parser corresponding to the first file format is selected from multiple parsers to parse the target copy file, obtaining a target file; then, the target file undergoes a first predetermined process including splitting to obtain multiple sub-files; subsequently, the target data in each sub-file is stored in a temporary table of the corresponding distributed unit; finally, at least a portion of the target data in each temporary table is stored in the corresponding formal table. Compared to the existing data synchronization mechanism between associated systems that only supports single file processing, resulting in poor versatility, this application, upon obtaining a target copy file, selects a target parser capable of parsing that file format from multiple parsers based on the file format of the target copy file, parses the copy file, and achieves the effect of parsing and processing different file formats. This makes it applicable to data synchronization of various file formats, ensuring good versatility.

[0192] 2) In the data synchronization device under the distributed architecture described in this application, when a target copy file sent by the associated system is obtained, the selection unit selects a target parser corresponding to the first file format from multiple parsers according to the first file format of the target copy file, and parses the target copy file to obtain a target file; the first processing unit performs a first predetermined processing, including splitting, on the target file to obtain multiple sub-files; the first storage unit stores the target data in each sub-file into a temporary table of the corresponding distributed unit; and the second storage unit stores at least a portion of the target data in each temporary table into the corresponding formal table. Compared with the existing data synchronization mechanism between associated systems that only supports single file processing, resulting in poor versatility, this application, when obtaining a target copy file, selects a target parser that can parse the file format from multiple parsers according to the file format of the target copy file, and parses the copy file, achieving the effect of parsing and processing different file formats. It can be applied to data synchronization of various file formats, ensuring good versatility.

[0193] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data synchronization method under a distributed architecture, characterized in that, include: If a target copy file is obtained, a target parser is selected from multiple parsers according to a first file format to parse the target copy file and obtain a target file. The target copy file is a copy file sent by the associated system. The target parser is used to parse files of the first file format, where the first file format is the file format of the target copy file. The multiple parsers are used to parse files of different file formats. The target file is subjected to a first predetermined process to obtain multiple sub-files. The first predetermined process includes a splitting process, and the data in each of the sub-files is the target data. Each of the target data is stored in a temporary table of the corresponding distributed unit. The distributed unit includes a formal table and a temporary table. One formal table corresponds to at least one of the temporary tables, and one distributed unit corresponds to at least one of the target data. At least a portion of the target data in each of the temporary tables shall be stored in the corresponding formal table; The first processing step is to store a portion of the target data in all the temporary tables corresponding to the target copy file into the corresponding formal table when the type of the target copy file is a single table file. The type of the copy file includes single table files and general table files. The single table file indicates that the copy file corresponds one-to-one with the formal table, and the general table file indicates that the file group corresponds one-to-one with the formal table. The file group includes multiple copy files. The determination step involves determining whether the data of other copy files in the target file group is stored in the corresponding temporary table if the type of the target copy file is the general table file. The target file group is the file group to which the target copy file belongs. The second processing step involves storing a portion of the target data from all the temporary tables corresponding to the file group into the corresponding formal table, provided that the data from the other copy files is stored in the corresponding temporary table. The determination step and the second processing step are executed at least once in sequence until a portion of the target data in all the temporary tables corresponding to the file group is stored into the corresponding formal table.

2. The method according to claim 1, characterized in that, The target file undergoes a first predetermined process to obtain multiple sub-files, including: According to preset rules, the target file is split to obtain multiple intermediate files, and each intermediate file corresponds to a temporary table. The preset rules include at least one of the following: splitting according to the customer number of the data in the target file, splitting according to the database to which the data in the target file belongs, and splitting according to whether the target file is a customer-related file. Based on the corresponding number of threads, each intermediate file is segmented into multiple sub-files corresponding to each intermediate file, and the number of sub-files is the same as the corresponding number of threads.

3. The method according to claim 2, characterized in that, The target data is stored in the temporary table of the corresponding distributed unit, including: Configure each target data corresponding to the intermediate file as a thread; Multiple threads corresponding to each intermediate file are executed in parallel to store multiple target data corresponding to each intermediate file into the corresponding temporary table in parallel.

4. The method according to claim 1, characterized in that, The first processing step includes: The target data in all the temporary tables is subjected to a second predetermined process to obtain the final data; The final data is stored in the corresponding formal table, and the second predetermined process includes at least one of data splitting, data combination, and deletion.

5. The method according to claim 1, characterized in that, Before performing the first predetermined processing on the target file to obtain multiple sub-files, after selecting a target parser from multiple parsers according to a first file format to parse the target copy file and obtain the target file, the method further includes: performing file verification on the target file. The target file is subjected to a first predetermined process to obtain multiple sub-files, including: if the target file is successfully verified, the target file is subjected to the first predetermined process to obtain multiple sub-files.

6. The method according to claim 1, characterized in that, After storing at least a portion of the target data from each of the temporary tables into the corresponding formal table, the method further includes at least one of the following: At least a portion of the data in the formal table is subjected to a third pre-processing, which includes deletion and modification; Based on the data in the official table, modify the data in the associated table that is linked to the official table; The software development kit is invoked to update the version of the official table, triggering a cache refresh.

7. A data synchronization device under a distributed architecture, characterized in that, include: The selection unit is used to select a target parser from multiple parsers to parse the target copy file according to a first file format when a target copy file is obtained, so as to obtain a target file. The target copy file is a copy file sent by the associated system. The target parser is used to parse files of the first file format, where the first file format is the file format of the target copy file. The multiple parsers are used to parse files of different file formats. A first processing unit is configured to perform a first predetermined processing on the target file to obtain multiple sub-files. The first predetermined processing includes splitting processing, and the data in each of the sub-files is the target data. The first storage unit is used to store each of the target data into a temporary table of the corresponding distributed unit. The distributed unit includes a formal table and a temporary table. One formal table corresponds to at least one of the temporary tables, and one distributed unit corresponds to at least one of the target data. The second storage unit is used to store at least a portion of the target data from each of the temporary tables into the corresponding formal table; The second processing module is used in the first processing step. When the type of the target copy file is a single table file, it stores part of the target data in all the temporary tables corresponding to the target copy file into the corresponding formal table. The type of the copy file includes single table files and general table files. The single table file indicates that the copy file and the formal table correspond one-to-one. The general table file indicates that the file group and the formal table correspond one-to-one. The file group includes multiple copy files. The determination module is used to determine the step of whether, when the type of the target copy file is the general table file, the data of other copy files in the target file group is stored in the corresponding temporary table, wherein the target file group is the file group to which the target copy file belongs; The third processing module is used in the second processing step to store a portion of the target data from all the temporary tables corresponding to the file group into the corresponding formal table, provided that the data of the other copy files is stored in the corresponding temporary table. An execution unit is configured to execute the determining step and the second processing step at least once in sequence until a portion of the target data in all the temporary tables corresponding to the file group is stored in the corresponding formal table.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 6.

9. A processor, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 6 when it runs.

10. A data synchronization system, characterized in that, include: The associated system is used to issue copies of the source file. A target system, connected to the associated system, the target system being used to receive the copy file, the target system including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any one of claims 1 to 6.

Images