Cross-datacenter file disaster recovery synchronization method and device

Abstract

This application discloses a cross-data center file disaster recovery synchronization method and apparatus, relating to the field of information security technology and also applicable to the financial field. The method includes: scanning the network storage directory every first preset time interval to obtain newly generated files; the newly generated files include end-of-day batch files and real-time synchronized files that failed to be transmitted; searching the file MD5 of the newly generated files in the file to be sent table; if the search fails, querying the file object of the previous file version corresponding to the business ID in the total file table based on the business ID of the newly generated file; if the query result is a query failure, identifying the newly generated file as a scheduled synchronization file; if the query result is a query success, performing incremental file segmentation based on the newly generated file and the file object to generate a scheduled synchronization file; and transmitting the scheduled synchronization file to a temporary directory in the second data center, thereby achieving real-time and accurate file resource synchronization, meeting disaster recovery requirements, and ensuring high availability in the face of failures.

Description

Technical Field

[0001] This application relates to the field of information security technology, and can also be used in the financial field, particularly to a method and apparatus for cross-data center file disaster recovery synchronization. Background Technology

[0002] Mobile banking is an internet application product that provides customers with a wide range of business services. Its technical implementation heavily utilizes static file resource packages. The mobile client obtains the CDN download address of the file through a three-center, multi-site active-active mobile server and then sends a request to the CDN node to retrieve the corresponding file. Once the CDN node receives the request, it checks if the cache has been hit; if not, it needs to retrieve the file from the origin server. CDN stands for Content Delivery Network, used to provide edge acceleration services for user access.

[0003] To achieve high availability for client access to CDN file resource packages, the links between the client and CDN nodes, as well as between CDN nodes and the origin server, are crucial. The high availability of these two links determines the overall high availability of the entire network. The link between the client and CDN nodes involves the ISP's DNS domain name resolution, the number of nodes deployed by the CDN provider, and the CDN provider's traffic scheduling strategy, leaving limited room for optimization. Regarding the link between CDN nodes and the origin server, while there are currently two logical origin servers, there is only one physically, indicating room for optimization.

[0004] Currently, to ensure high availability for CDN origin server deployment, the origin server has applied for two CDN domains: one primary and one backup. These domains resolve to two different web server clusters within the same data center. Web requests are forwarded to the same NAS file server via load balancing. Normally, the primary domain primarily provides resource access services to clients. If the primary domain experiences multiple access failures, clients will attempt to access resources through the backup domain, ensuring uninterrupted business operations.

[0005] The drawbacks of the above method are that it cannot meet the business requirements of high availability across data centers. When deploying static resource files, although two different web clusters are deployed in the same data center, in extreme cases where the entire data center is unavailable and the CDN cache fails, it is impossible to access files from the origin server. It cannot provide file download transaction services when CDN access is abnormal, and it cannot provide data center-level horizontal scaling for download transactions. In addition, this method uses two domain names, and the client obtains resources through the backup domain name when the primary domain name fails. The calling logic is complex, the timing of the call is uncertain, and the cost of applying for two domain names simultaneously is too high.

[0006] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0007] This application provides a cross-data center file disaster recovery synchronization method, applied to a first data center, for performing file disaster recovery synchronization. The method includes:

[0008] The network storage directory is scanned every first preset time interval to obtain newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that failed to be transmitted.

[0009] Find the file MD5 hash of the newly generated file in the list of files to be sent;

[0010] If the search fails, query the file object of the previous file version corresponding to the business ID in the master file table based on the business ID of the newly generated file;

[0011] If the query result is a failure, the newly generated file will be designated as a scheduled synchronization file;

[0012] If the query result is successful, incremental file segmentation is performed based on the newly generated file and the file object to generate a timed synchronization file;

[0013] The timed synchronization file is transferred to a temporary directory in the second data center.

[0014] Further, the step of incrementally splitting the file based on the newly generated file and the file object to generate a timed synchronization file includes:

[0015] The newly generated file and the file object are divided into multiple file blocks.

[0016] The file blocks of the newly generated file are used to perform sliding difference detection on the file blocks of the file object in order to generate a timed synchronization file.

[0017] Furthermore, the step of transferring the timed synchronization file to a temporary directory in the second data center includes:

[0018] The file information of the scheduled synchronized files is written into the file to be sent table and the total file table; the fields of the file to be sent table include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time, and original data center; the fields of the total file table include: file ID, service ID, file path, file symbolic link path, file MD5, file version, whether the file is deployed on CDN, creation time, creator, update time, updater, resource type, and file size;

[0019] Every second preset time interval, the list of files to be sent is scanned, and timed synchronization files with transmission status of not transmitted or transmission failure are transmitted to the temporary directory of the second data center.

[0020] Furthermore, it also includes:

[0021] When a real-time synchronization file is generated, a thread pool is used to transmit the real-time synchronization file to the second data center; the real-time synchronization file includes production release files and business configuration files.

[0022] Furthermore, the step of using a thread pool to transfer the real-time synchronized file to the second data center includes:

[0023] The task of transmitting the real-time synchronized file is encapsulated as a thread pool task;

[0024] The thread pool task is submitted to the thread pool to transfer the real-time synchronized file to the second data center using a file transfer tool.

[0025] This application provides a cross-data center file disaster recovery synchronization method, applied to a second data center, for performing file disaster recovery synchronization. The method includes:

[0026] Every third preset time interval, the temporary directory is scanned to obtain the timed synchronization files transmitted from the first data center;

[0027] Based on the file symbolic link path of the scheduled synchronization file, the corresponding business ID of the scheduled synchronization file is found in the master file table;

[0028] Based on the service ID of the scheduled synchronization file, query the file object of the previous file version corresponding to the service ID in the master file table;

[0029] If the query result is a failure, the scheduled synchronization file will be designated as the official file.

[0030] If the query result is successful, incremental file merging is performed based on the scheduled synchronization file and the file object to generate a complete file, and the complete file is verified to be an official file;

[0031] Copy the official file to the official directory and write the file information of the official file into the received file table and the total file table.

[0032] Furthermore, verifying whether the complete document is an official document includes:

[0033] The MD5 hash of the complete file is calculated, and the MD5 hash of the current version file is found in the total file table.

[0034] If the MD5 hash of the complete file matches the MD5 hash of the current version file, the complete file is identified as the official file.

[0035] Furthermore, it also includes:

[0036] At a preset time, the received file table and the obtained file table to be sent from the first data center are compared to determine the file synchronization status; the fields of the received file table include: transmission task ID, file ID, transmission source, target data center, reception status, creation time, update time and original data center.

[0037] Furthermore, it also includes:

[0038] The file receiving tool is used to receive real-time synchronized files transmitted by the first data center using a thread pool; the real-time synchronized files include production release files and business configuration files.

[0039] Furthermore, the step of receiving the real-time synchronized file transmitted by the first data center using a thread pool via a file receiving tool includes:

[0040] The file receiving tool is used to listen to a preset port in order to obtain file transfer requests from the first data center;

[0041] When a file transfer request is received from the first data center, the real-time synchronized file is received.

[0042] This application embodiment also provides a cross-data center file disaster recovery synchronization device for performing file disaster recovery synchronization, the device comprising:

[0043] The first file acquisition unit is used to scan the network storage directory every first preset time interval to acquire newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that failed to be transmitted.

[0044] The first file search unit is used to search the file MD5 of the newly generated file in the file to be sent table;

[0045] The first file query unit is used to query the file object of the previous file version corresponding to the business ID in the total file table if the search fails;

[0046] The first file determination unit is used to determine the newly generated file as a timed synchronization file if the query result is a query failure; and to perform incremental file segmentation based on the newly generated file and the file object to generate a timed synchronization file if the query result is a query success.

[0047] The first file transfer unit is used to transfer the timed synchronization file to the temporary directory of the second data center.

[0048] This application embodiment also provides a cross-data center file disaster recovery synchronization device for performing file disaster recovery synchronization, the device comprising:

[0049] The second file acquisition unit is used to scan the temporary directory every third preset time interval to acquire the timed synchronization file transmitted from the first data center.

[0050] The second file lookup unit is used to look up the business ID corresponding to the timed synchronization file in the master file table according to the file soft link path of the timed synchronization file.

[0051] The second file query unit is used to query the file object of the previous file version corresponding to the service ID in the master file table according to the service ID of the timed synchronization file;

[0052] The second file determination unit is used to determine the timed synchronization file as a formal file if the query result is a query failure; if the query result is a query success, it performs incremental file merging based on the timed synchronization file and the file object to generate a complete file, and verifies whether the complete file is a formal file.

[0053] The file copying unit is used to copy the official file to the official directory and write the file information of the official file into the received file table and the total file table.

[0054] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described cross-data center file disaster recovery synchronization method.

[0055] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described cross-data center file disaster recovery synchronization method.

[0056] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described cross-data center file disaster recovery synchronization method.

[0057] The cross-data center file disaster recovery synchronization method and apparatus provided in this application embodiment scans the network storage directory at a first preset time interval in a first data center to obtain newly generated files. These newly generated files include end-of-day batch files and real-time synchronization files that failed to transmit. The method searches the file MD5 hash of the newly generated files in a file-to-send table. If the search fails, it queries the file object of the previous file version corresponding to the service ID in the total file table based on the service ID of the newly generated file. If the query result is also unsuccessful, the newly generated file is identified as a scheduled synchronization file. If the query result is successful, incremental file segmentation is performed based on the newly generated file and the file object to generate a scheduled synchronization file. The scheduled synchronization file is then transmitted to a temporary directory in a second data center, and the second data center scans the temporary directory at a third preset time interval. The system retrieves the timed synchronization file transmitted from the first data center; it searches the master file table for the corresponding service ID of the timed synchronization file based on the file symbolic link path of the timed synchronization file; it then searches the master file table for the file object of the previous file version corresponding to the service ID of the timed synchronization file; if the query fails, the timed synchronization file is identified as a formal file; if the query succeeds, incremental file merging is performed based on the timed synchronization file and the file object to generate a complete file, and the complete file is verified as a formal file; the formal file is copied to the formal directory, and the file information of the formal file is written to the received file table and the master file table. This achieves real-time and accurate file resource synchronization, meets the requirements of disaster recovery deployment, and ensures high availability in the face of failures.

[0058] The strategy combines high-real-time file synchronization with end-of-day full file synchronization, effectively collecting and synchronizing changed files, ensuring the timeliness and integrity of file synchronization, and guaranteeing the business requirements for real-time files. Temporary directory transfers prevent the mistransmission and overwriting of large batches of files. Incremental file splitting significantly reduces intranet bandwidth consumption. The design of the file table structure further improves reliability and flexibility. Single-domain calls reduce implementation costs and simplify the calling logic. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0060] Figure 1A flowchart illustrating a cross-data center file disaster recovery synchronization method provided in an embodiment of this application;

[0061] Figure 2 A flowchart illustrating a cross-data center file disaster recovery synchronization method provided in an embodiment of this application;

[0062] Figure 3 A flowchart illustrating a cross-data center file disaster recovery synchronization method provided in an embodiment of this application;

[0063] Figure 4 A flowchart illustrating a cross-data center file disaster recovery synchronization method provided in an embodiment of this application;

[0064] Figure 5 A schematic diagram illustrating the production release and file synchronization process provided in one embodiment of this application;

[0065] Figure 6 A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0066] Figure 7 A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0067] Figure 8 A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0068] Figure 9 A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0069] Figure 10 A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0070] Figure 11 A flowchart illustrating a cross-data center file disaster recovery synchronization method provided in an embodiment of this application;

[0071] Figure 12 A flowchart illustrating a cross-data center file disaster recovery synchronization method provided in an embodiment of this application;

[0072] Figure 13 A flowchart illustrating a cross-data center file disaster recovery synchronization method provided in an embodiment of this application;

[0073] Figure 14 A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0074] Figure 15A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0075] Figure 16 A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0076] Figure 17 A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0077] Figure 18 A schematic diagram of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application;

[0078] Figure 19 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of this application are used to explain this application, but are not intended to limit this application.

[0080] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0081] The acquisition, transmission, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0082] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0083] Storing files in a single data center poses a single point of failure risk. Therefore, this application provides a disaster recovery deployment for storing file resource packages in multiple data centers. In case of an anomaly, the disaster recovery center issues an origin server switch command. The first data center serves as the primary data center for file resource publishing and CDN origin pull for production deployment and business configuration, while the second data center serves as a backup data center. In one embodiment, a third data center serves as another backup data center. This application does not limit the number of data centers.

[0084] The primary and backup data centers are configured in different server rooms. The client access to the CDN configuration has been changed from using two primary and backup domain names to using only one domain name, which reduces implementation costs and simplifies the calling logic.

[0085] Figure 1 A flowchart illustrating a cross-datacenter file disaster recovery synchronization method provided in an embodiment of this application is shown below. Figure 1 As shown, the cross-datacenter file disaster recovery synchronization method provided in this application includes:

[0086] S101: Scan the network storage directory every first preset time interval to obtain newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that failed to be transmitted;

[0087] S102: Search the file MD5 of the newly generated file in the file to be sent table;

[0088] S103: If the search fails, query the file object of the previous file version corresponding to the business ID in the master file table according to the business ID of the newly generated file;

[0089] S104: If the query result is a query failure, the newly generated file is determined as a timed synchronization file; if the query result is a query success, incremental file segmentation is performed based on the newly generated file and the file object to generate a timed synchronization file;

[0090] S105: Transfer the timed synchronization file to the temporary directory of the second data center.

[0091] from Figure 1 As shown in the flowchart, the cross-data center file disaster recovery synchronization method provided in this application scans the network storage directory every first preset time interval to obtain newly generated files. These newly generated files include end-of-day batch files and real-time synchronized files that failed to transmit. The method then searches the file MD5 hash of the newly generated files in the file-to-send table. If the search fails, it queries the file object of the previous file version corresponding to the business ID in the overall file table. If the query result is also unsuccessful, the newly generated file is identified as a scheduled synchronization file. If the query result is successful, incremental file segmentation is performed based on the newly generated file and the file object to generate a scheduled synchronization file. The scheduled synchronization file is then transmitted to a temporary directory in the second data center. This achieves real-time and accurate file resource synchronization, meets the requirements of disaster recovery deployment, and ensures high availability in the face of failures.

[0092] The following section uses the first data center as the execution entity to explain each step in detail.

[0093] S101: Scan the network storage directory every first preset time interval to obtain newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that failed to be transmitted;

[0094] Specifically, the first data center starts a file update listener to monitor and scan file changes in the NAS directory. It scans the NAS directory every preset time interval to check for new file creation or updates, ensuring that newly created or updated files are detected and recorded in a timely manner to avoid missing any important file updates and to prepare for subsequent file synchronization and transfer.

[0095] In one embodiment, the file update listener startup process of the first data center is named "bye". It uses the Linux find command to listen to the NAS directory, executes a scan command, and searches for files that have been newly generated or updated in a specified directory in the recent period.

[0096] In one embodiment, the preset time interval can be 5 minutes, but this application is not limited thereto.

[0097] In one embodiment, in an emergency, a file scanning task can be manually triggered to ensure that file updates can be detected and processed in a timely manner even under special circumstances. This increases the system's flexibility and ability to respond to emergencies, ensuring that the effectiveness and accuracy of file synchronization can be maintained under any circumstances.

[0098] S102: Search the file MD5 of the newly generated file in the file to be sent table;

[0099] Specifically, for newly generated files detected by scanning, the first data center checks whether a file record with the same MD5 hash exists in the file to be sent table (FT_IDC_SYN_FILE_SEND). If not found and the file's trigger type is "real-time synchronization" or "scheduled synchronization," the data center queries the master file table for the file object of the previous file version corresponding to the business ID of the newly generated file.

[0100] S103: If the search fails, query the file object of the previous file version corresponding to the business ID in the master file table according to the business ID of the newly generated file;

[0101] Specifically, each resource file is generated after code compilation. In most business applications, the code deployed in this production run differs slightly from the previous code baseline. However, each deployment generates a complete resource package, and even modifying a single line of content will generate a complete resource package of several megabytes in size. Transmitting complete packages during file synchronization would waste internal network bandwidth. Therefore, regardless of file type, to save bandwidth during scheduled unified transmission, files are differentiated during transmission and then merged.

[0102] If there is no file record with the same MD5 in the file to be sent table, before file synchronization, the first data center first searches for the file object with the same business ID (BUSINESS_ID) and the previous file version timestamp through the total file table (FT_BASE_FILE). By searching for the previous version file, the first data center can determine whether to perform full file synchronization or incremental synchronization, thus avoiding unnecessary file transfers.

[0103] S104: If the query result is a query failure, the newly generated file is determined as a timed synchronization file; if the query result is a query success, incremental file segmentation is performed based on the newly generated file and the file object to generate a timed synchronization file;

[0104] Specifically, the first data center determines the scheduled synchronization files that need to be transmitted to the second data center based on the query results of the file object corresponding to the previous file version timestamp of the business ID in the master file table.

[0105] If this business ID is being deployed for the first time, meaning that the query result for the file object with the previous file version timestamp corresponding to the business ID in the master file table fails, the first data center will directly synchronize the complete large file of this deployment to the second data center as a scheduled synchronization file.

[0106] If a previous version of the file exists (i.e., the query result for the file object with the previous file version timestamp corresponding to the business ID in the master file table is successful), the first data center uses a file processing tool to compare the file being deployed in production with the file from the previous version, calculates and generates an incremental file (Delta) containing only the differences between the old and new versions. By generating incremental files, the amount of data transmission is significantly reduced, optimizing the use of internal network bandwidth and avoiding bandwidth waste caused by transmitting complete file packets for minor changes.

[0107] In one embodiment, the file processing tool is rsync4j, but this application is not limited thereto.

[0108] Figure 2 A flowchart illustrating a cross-datacenter file disaster recovery synchronization method provided in an embodiment of this application is shown below. Figure 2 As shown, S104 includes:

[0109] S201: Divide the newly generated file and the file object into multiple file blocks;

[0110] Specifically, the first data center divides the files of the previous version and the current production version of the same business ID (BUSSINESS_ID) into blocks according to a preset size for subsequent comparison.

[0111] In one embodiment, the first data center calculates a strong digest (e.g., MD5 and SHA hash values) and a weak digest (e.g., checksum) for each file block, and concatenates the strong and weak digests of all blocks to form a signature string, which represents the block digest information of the previous version of the file.

[0112] S202: Use the file blocks of the file object to perform sliding difference detection on the file blocks of the newly generated file in order to generate a timed synchronization file.

[0113] Specifically, the first data center performs sliding window difference detection on the current version file to generate incremental files for periodic synchronization. Sliding window difference detection is then applied to the current production version file, comparing blocks one by one with the previous version file to identify which blocks are identical and which are different. For identical blocks, only their sequence numbers within the current file are recorded; for different blocks, their contents are recorded.

[0114] The first data center concatenates the sequence numbers of identical blocks with the contents of different blocks to form a Delta incremental file. This incremental file contains the changed parts and their corresponding sequence numbers, representing the differences between the current version and the previous version.

[0115] S105: Transfer the timed synchronization file to the temporary directory of the second data center.

[0116] Specifically, the first data center starts a file sending timer, which will periodically transmit synchronized files to the temporary directory of the second data center at a preset time, and update the file transmission status after successful file transmission.

[0117] In one embodiment, in an emergency, the first data center allows the background to manually trigger file transfer tasks to deal with situations where the automatic scheduled transfer tasks cannot be executed normally. This provides additional flexibility and ensures that file transfer tasks can still be executed in a timely manner in special or emergency situations, without affecting the continuity of the overall business.

[0118] In one embodiment, timed synchronous file transfer can be achieved by calling the NFTSA tool, a new generation of file transfer tool based on Security Components (SA), similar to FTP tools. The transfer direction includes the client sending files to the server or the client pulling files from the server.

[0119] Figure 3A flowchart illustrating a cross-datacenter file disaster recovery synchronization method provided in an embodiment of this application is shown below. Figure 3 As shown, S105 includes:

[0120] S301: Write the file information of the scheduled synchronized file into the file to be sent table and the total file table; the fields of the file to be sent table include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time, and original data center; the fields of the total file table include: file ID, service ID, file path, file soft link path, file MD5, file version, whether the file is deployed on CDN, creation time, creator, update time, updater, resource type, and file size;

[0121] Specifically, information about files to be synchronized at scheduled intervals is written to the "Files to be Sent" table (FT_IDC_SYN_FILE_SEND) and the "Master Files" table (FT_BASE_FILE). The recorded information includes the file's ID, path, MD5 value, version timestamp, and other crucial details. This ensures that all files requiring synchronization are accurately recorded and provides necessary data support for subsequent file transfers, guaranteeing the accuracy and consistency of the synchronization. The fields in the "Files to be Sent" table include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time, and original data center. The fields in the "Master Files" table include: file ID, business ID, file path, file symbolic link path, file MD5, file version, whether the file is deployed via CDN, creation time, creator, update time, updater, resource type, and file size.

[0122] In one embodiment, the table structure of the total file table is shown in Table 1, and the table structure of the file to be sent table is shown in Table 2. By designing the table structure, new backup computer rooms can be quickly expanded, further improving reliability and flexibility.

[0123] Table 1

[0124]

[0125]

[0126] Table 2

[0127]

[0128]

[0129] S302: Scan the file table to be sent every second preset time interval, and transmit the timed synchronization files with transmission status of not transmitted or transmission failure to the temporary directory of the second data center.

[0130] Specifically, the first data center starts a file sending timer, which queries the file task table to be sent every preset time interval, filtering out records with a status of "not sent" or "sending failed" and whose sending count has not exceeded the preset number. To prevent reading too many records at once, a preset number of records are read in batches each time, which improves processing efficiency and ensures the smooth execution of each transmission task, avoiding excessive load due to reading too many records.

[0131] In one embodiment, the preset time interval is 10 minutes, the preset number of times is 100, and the preset quantity is 100 items, but this application is not limited thereto.

[0132] In one embodiment, the user who starts the file sending timer in the first data center is "bye". The process of the "bye" user calls the process service of the "nftsa" user, passing in the corresponding parameters (such as the IP address and port of the second data center NFTSA, the local directory address, and the target directory address), and transmits the file to be sent to the temporary directory of the second data center. The incremental file (Delta) is a transitional file that needs to be temporarily stored in the temporary directory and then merged to generate a usable file. This ensures that even if problems occur during transmission, the business files in the official directory will not be affected, thus improving the security of file transmission.

[0133] After a file is successfully transferred, the first data center will update the STATUS field in the file to be sent table to "1" to indicate that the transfer was successful, and increment the sending count (SEND_NUMBER) by 1 to ensure that the progress of the transfer task can be tracked in real time and to provide a basis for subsequent exception handling and retransmission tasks.

[0134] In one embodiment, the cross-datacenter file disaster recovery synchronization method further includes:

[0135] When a real-time synchronization file is generated, a thread pool is used to transmit the real-time synchronization file to the second data center; the real-time synchronization file includes production release files and business configuration files.

[0136] Specifically, typical generation scenarios for new file resource packages in mobile banking include release on the production launch day, generation after maintenance and configuration by back-end business personnel on the workbench, and batch generation at the end of the day. Regardless of the scenario, to ensure consistency in file resource package deployment, all new file resource packages are uniformly released and activated in the main data center. During production launch, the main data center dynamically generates CDN version information, file MD5, file version timestamp, resource directory address on the NAS file server, resource key data, production launch record, and CDN LOGO file address for each resource package, and stores this data uniformly in the main data center's database record table. Simultaneously, each file resource package is also saved to the corresponding dynamically generated NAS file directory.

[0137] Dual-center file storage is a necessary prerequisite for disaster recovery failover. Files in the primary data center need to be synchronized to the backup data center as a backup. File synchronization adopts a combination of real-time synchronization and scheduled synchronization. Files and business configuration files released on the production day are synchronized in real time to ensure that the timeliness meets the disaster recovery requirements. Files generated in batches at the end of the day and files that failed to be synchronized in real time are fully transmitted on a scheduled basis until successful or the number of retransmissions is exceeded.

[0138] While the primary data center is deploying or generating business configuration files, it transmits the newly generated files as real-time synchronization files to the secondary data center (backup data center) to achieve file disaster recovery synchronization.

[0139] Figure 4 A flowchart illustrating a cross-datacenter file disaster recovery synchronization method provided in an embodiment of this application is shown below. Figure 4 As shown, using a thread pool to transfer the real-time synchronized files to the second data center includes:

[0140] S401: Encapsulate the task of transmitting the real-time synchronized file into a thread pool task;

[0141] Specifically, when a new version of a file needs to be released in the first data center, the main business process is first triggered for production deployment. After preparing the new file version that needs to be synchronized, the first data center encapsulates the file synchronization operation into a thread pool task to ensure that the synchronization process does not affect the main business process.

[0142] S402: Submit the thread pool task to the thread pool to use the file transfer tool to transfer the real-time synchronized file to the second data center.

[0143] Specifically, the first data center submits the encapsulated file synchronization task to the thread pool, ready to execute the synchronization operation. Upon receiving the task, the thread pool uses a file transfer tool to transmit the file to the second data center in real time. The thread pool acts as a task manager, handling file synchronization tasks asynchronously and concurrently to prevent them from impacting the main business processes. The file transfer tool receives the file synchronization task assigned by the thread pool and transmits the file from the first data center to the second data center in real time over the network, ensuring that the file is not lost during transmission and successfully reaches the target data center.

[0144] In one embodiment, the file transfer tool is FCLIENT, and the deployment and file synchronization process is as follows: Figure 5 As shown, it includes:

[0145] The file resource package is put into production and released. Database records such as file version and file MD5 are generated. The files are stored in the official directory of the NAS server, and the file source address, file destination address, and file MD5 are recorded. The files are then packaged into a task for sending files in real time.

[0146] Submit the packaged task to the task sending thread pool, asynchronously check if there are still files to be published. If so, return to the previous step and perform loop operations. Otherwise, the production release of the file resource package ends.

[0147] After a task is submitted to the task sending thread pool, the address and port of the FSERVER server are obtained from the configuration file, the FCLIENT sets the API parameters, and the transmission is initiated.

[0148] Determine whether the transmission was successful. If it was, record the transmission result of the file in the file to be sent table and set the transmission status to successful. Otherwise, record the transmission result of the file in the file to be sent table and set the transmission status to failed.

[0149] This application provides a cross-datacenter file disaster recovery synchronization method. It involves scanning the network storage directory at first preset time intervals to obtain newly generated files. These newly generated files include end-of-day batch files and real-time synchronized files that failed to transmit. The method searches the file MD5 hash of the newly generated files in a list of files to be sent. If the search fails, it queries the main file table for the file object corresponding to the previous file version of the business ID of the newly generated file. If the query fails, the newly generated file is identified as a scheduled synchronization file. If the query succeeds, incremental file segmentation is performed based on the newly generated file and the file object to generate a scheduled synchronization file. The scheduled synchronization file is then transmitted to a temporary directory in a second data center. This method achieves real-time and accurate file resource synchronization, meets the requirements of disaster recovery deployment, and ensures high availability in the face of failures.

[0150] The strategy combines high-real-time file synchronization with end-of-day full file synchronization, effectively collecting and synchronizing changed files, ensuring the timeliness and integrity of file synchronization, and guaranteeing the business requirements for real-time files. Temporary directory transfers prevent the mistransmission and overwriting of large batches of files. Incremental file splitting significantly reduces intranet bandwidth consumption. The design of the file table structure further improves reliability and flexibility. Single-domain calls reduce implementation costs and simplify the calling logic.

[0151] This application also provides a cross-datacenter file disaster recovery synchronization device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to the cross-datacenter file disaster recovery synchronization method, the implementation of this device can refer to the implementation of the cross-datacenter file disaster recovery synchronization method; repeated details will not be elaborated further.

[0152] Figure 6 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application is shown below. Figure 6 As shown, the device includes:

[0153] The first file acquisition unit 601 is used to scan the network storage directory every first preset time interval to acquire newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that failed to be transmitted.

[0154] The first file lookup unit 602 is used to look up the file MD5 of the newly generated file in the file to be sent table;

[0155] The first file query unit 603 is used to query the file object of the previous file version corresponding to the business ID in the total file table if the search fails;

[0156] The first file determination unit 604 is used to determine the newly generated file as a timed synchronization file if the query result is a query failure; and to perform incremental file segmentation based on the newly generated file and the file object to generate a timed synchronization file if the query result is a query success.

[0157] The first file transfer unit 605 is used to transfer the timed synchronization file to the temporary directory of the second data center.

[0158] Figure 7 This is a schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application. Figure 6 Based on the embodiments, further, such as Figure 7 As shown, the cross-datacenter file disaster recovery synchronization device provided in this application also includes:

[0159] The file segmentation module 701 is used to segment the newly generated file and the file object to obtain multiple file blocks respectively;

[0160] The difference detection module 702 is used to perform sliding difference detection on the file blocks of the newly generated file using the file blocks of the file object, so as to generate a timed synchronization file.

[0161] Figure 8 This is a schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application. Figure 6 Based on the embodiments, further, such as Figure 8 As shown, the cross-datacenter file disaster recovery synchronization device provided in this application also includes:

[0162] The information writing module 801 is used to write the file information of the timed synchronized file into the file table to be sent and the total file table; the fields of the file table to be sent include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time, and original data center; the fields of the total file table include: file ID, service ID, file path, file soft link path, file MD6, file version, whether the file is deployed via CDN, creation time, creator, update time, updater, resource type, and file size;

[0163] The first file transfer module 802 is used to scan the file table to be sent every second preset time interval and transfer the timed synchronization files with the transmission status of not transmitted or transmission failure to the temporary directory of the second data center.

[0164] Figure 9 This is a schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application. Figure 6 Based on the embodiments, further, such as Figure 9 As shown, the cross-datacenter file disaster recovery synchronization device provided in this application also includes:

[0165] The second file transfer unit 901 is used to transfer the real-time synchronized file to the second data center using a thread pool when a real-time synchronized file is generated; the real-time synchronized file includes production release files and business configuration files.

[0166] Figure 10 This is a schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application. Figure 9 Based on the embodiments, further, such as Figure 10 As shown, the cross-datacenter file disaster recovery synchronization device provided in this application also includes:

[0167] Task encapsulation module 1001 is used to encapsulate the task of transmitting the real-time synchronized file into a thread pool task;

[0168] The second file transfer module 1002 is used to submit the thread pool task to the thread pool so as to use the file transfer tool to transfer the real-time synchronized file to the second data center.

[0169] This application provides a cross-datacenter file disaster recovery synchronization method and apparatus. The method involves scanning the network storage directory at first preset time intervals to obtain newly generated files. These newly generated files include end-of-day batch files and real-time synchronized files that failed to transmit. The method searches the file MD5 hash of the newly generated files in a list of files to be sent. If the search fails, the method queries the main file table for the file object corresponding to the previous file version of the business ID of the newly generated file. If the query result is also unsuccessful, the newly generated file is identified as a scheduled synchronization file. If the query result is successful, incremental file segmentation is performed based on the newly generated file and the file object to generate a scheduled synchronization file. The scheduled synchronization file is then transmitted to a temporary directory in a second data center. This method achieves real-time and accurate file resource synchronization, meets the requirements of disaster recovery deployment, and ensures high availability in the face of failures.

[0170] The strategy combines high-real-time file synchronization with end-of-day full file synchronization, effectively collecting and synchronizing changed files, ensuring the timeliness and integrity of file synchronization, and guaranteeing the business requirements for real-time files. Temporary directory transfers prevent the mistransmission and overwriting of large batches of files. Incremental file splitting significantly reduces intranet bandwidth consumption. The design of the file table structure further improves reliability and flexibility. Single-domain calls reduce implementation costs and simplify the calling logic.

[0171] Figure 11 A flowchart illustrating a cross-datacenter file disaster recovery synchronization method provided in an embodiment of this application is shown below. Figure 11 As shown, the cross-datacenter file disaster recovery synchronization method provided in this application includes:

[0172] S1101: Scan the temporary directory every third preset time interval to obtain the timed synchronization file transmitted from the first data center;

[0173] S1102: Based on the file symbolic link path of the timed synchronization file, search the master file table for the business ID corresponding to the timed synchronization file;

[0174] S1103: Based on the service ID of the timed synchronization file, query the file object of the previous file version corresponding to the service ID in the master file table;

[0175] S1104: If the query result is a query failure, the scheduled synchronization file is determined as a formal file; if the query result is a query success, incremental files are merged according to the scheduled synchronization file and the file object to generate a complete file, and the complete file is verified as a formal file.

[0176] S1105: Copy the official file to the official directory and write the file information of the official file into the received file table and the total file table.

[0177] from Figure 11 As shown in the process, the cross-data center file disaster recovery synchronization method provided in this application obtains the timed synchronization file transmitted from the first data center by scanning the temporary directory every third preset time interval; searching for the service ID corresponding to the timed synchronization file in the total file table according to the file soft link path of the timed synchronization file; querying the file object of the previous file version corresponding to the service ID in the total file table according to the service ID of the timed synchronization file; if the query result is a query failure, the timed synchronization file is identified as a formal file; if the query result is a query success, incremental file merging is performed according to the timed synchronization file and the file object to generate a complete file, and verifying whether the complete file is a formal file; the formal file is copied to the formal directory and the file information of the formal file is written into the received file table and the total file table, thereby realizing the real-time and accuracy of file resource synchronization, meeting the requirements of disaster recovery deployment, and ensuring high availability in the face of failure.

[0178] The following section uses the second data center as the execution entity to explain each step in detail.

[0179] S1101: Scan the temporary directory every third preset time interval to obtain the timed synchronization file transmitted from the first data center;

[0180] Specifically, the second data center starts a file update listener, which listens to the temporary directory at preset time intervals to check whether new timed synchronization files have been received. During the listening process, the find command is used to find new files to ensure the timeliness of file synchronization and updates.

[0181] S1102: Based on the file symbolic link path of the timed synchronization file, search the master file table for the business ID corresponding to the timed synchronization file;

[0182] Specifically, when the listener detects a new scheduled synchronization file, it uses the file's symbolic link path as a unique ID to search for the business ID (BUSINESS_ID) associated with that scheduled synchronization file in the main file table FT_BASE_FILE, in order to query the file object of the previous file version.

[0183] S1103: Based on the service ID of the timed synchronization file, query the file object of the previous file version corresponding to the service ID in the master file table;

[0184] Specifically, after the second data center finds the business ID (BUSINESS_ID) associated with the scheduled synchronization file in the master file table FT_BASE_FILE, it queries the master file table FT_BASE_FILE for the file object with the previous file version timestamp corresponding to the business ID, based on the business ID of the scheduled synchronization file. This ensures that the incremental file can be accurately identified and its corresponding business and version, avoiding incorrect file merging or updating.

[0185] S1104: If the query result is a query failure, the scheduled synchronization file is determined as a formal file; if the query result is a query success, incremental files are merged according to the scheduled synchronization file and the file object to generate a complete file, and the complete file is verified as a formal file.

[0186] Specifically, the second data center queries the FT_BASE_FILE table for the file object with the previous file version timestamp corresponding to the business ID of the scheduled synchronization file, and determines whether the scheduled synchronization file is a formal file put into production for the first time or an incremental file with the previous version based on the query results.

[0187] If this is the first time a business ID is being deployed, meaning a query in the master file table for the file object corresponding to the previous file version timestamp of the business ID fails, the second data center directly designates the scheduled synchronized file as the official file. If a previous version of the file exists, meaning a query in the master file table for the file object corresponding to the previous file version timestamp of the business ID succeeds, the second data center encapsulates the found file information into an incremental file assembly task and submits it to the assembly thread pool for processing. This allows for parallel processing of multiple incremental file assembly tasks, improving processing efficiency and ensuring that incremental files can be merged in a timely manner to generate the final business-usable file.

[0188] After receiving an incremental file assembly task, the assembly thread pool uses a file processing tool to incrementally merge the incremental file (the periodically synchronized file) in the temporary directory with the file object of the previous version, generating a complete file. Incremental merging processes only the changed parts of the file, effectively reducing resource consumption and improving the efficiency and accuracy of the merging process. After merging, the complete file is determined to be the final file based on the file object.

[0189] Figure 12 A flowchart illustrating a cross-datacenter file disaster recovery synchronization method provided in an embodiment of this application is shown below. Figure 12 As shown, verifying whether the complete file is an official file includes:

[0190] S1201: Calculate the MD5 of the complete file and look up the MD5 of the current version file in the total file table;

[0191] Specifically, after the merge is completed, the second data center calculates the MD5 value of the generated complete file and compares it with the file MD5 of the current version file stored in the total file table FT_BASE_FILE obtained according to the business ID.

[0192] S1202: If the MD5 hash of the complete file matches the MD5 hash of the current version file, the complete file is determined to be the official file.

[0193] Specifically, if the two MD5 values ​​are equal, it means that the merged file is completely identical to the same file in the original data center, verifying that the complete file is the official file. Verifying the integrity and accuracy of the file ensures that no errors or data corruption occurred during transmission and merging.

[0194] S1105: Copy the official file to the official directory and write the file information of the official file into the received file table and the total file table.

[0195] Specifically, if the MD5 value comparison is successful, the assembly thread copies the official file to the official directory. After the file is successfully copied to the official directory, the second data center writes the relevant data into the received file table and sets the status to "2" (written to the official directory) to ensure the traceability of file processing and the integrity of records, which facilitates subsequent monitoring and management.

[0196] In one embodiment, the application user "bye" can operate on the official directory, while the NFTSA tool user "nftsa" does not have the right to operate on it. This avoids accidental operation and prevents the tool from accidentally operating on files in the official directory, thereby enhancing security and the reliability of file management.

[0197] In one embodiment, the cross-datacenter file disaster recovery synchronization method further includes:

[0198] At a preset time, the received file table and the obtained file table to be sent from the first data center are compared to determine the file synchronization status; the fields of the received file table include: transmission task ID, file ID, transmission source, target data center, reception status, creation time, update time and original data center.

[0199] Specifically, within a preset time range, the system compares whether the files sent by the first data center and the files received by the second data center are consistent. The second data center retrieves the list of files to be sent from the first data center and compares it with its local list of received files. Based on the date, it retrieves files from the list of files to be sent that have been successfully transmitted, and retrieves files from the list of received files that have been written to the official directory. The system then compares whether the total number of files sent and received is consistent within a preset time range, and also compares whether the specific details of the files sent and received are completely consistent. The specific details of the files sent and received include the file ID, file path, and file MD5 hash.

[0200] In one embodiment, the table structure of the received file table is shown in Table 3. By designing the table structure, new backup computer rooms can be quickly expanded, further improving reliability and flexibility.

[0201] Table 3

[0202]

[0203]

[0204] In one embodiment, the preset time range can be days, weeks, or months, but this application is not limited to this.

[0205] In one embodiment, the cross-datacenter file disaster recovery synchronization method further includes:

[0206] The file receiving tool is used to receive real-time synchronized files transmitted by the first data center using a thread pool; the real-time synchronized files include production release files and business configuration files.

[0207] Specifically, dual-center file storage is a necessary prerequisite for disaster recovery switchover. Files in the primary data center need to be synchronized to the backup data center as a backup. File synchronization adopts a combination of real-time synchronization and scheduled synchronization. Files and business configuration files released on the production day are synchronized in real time to ensure that the timeliness meets the disaster recovery requirements. Files generated in batches at the end of the day and files that failed to be synchronized in real time are fully transmitted on a scheduled basis until successful or the number of retransmissions is exceeded.

[0208] The second data center uses a file receiving tool to receive the production release files and business configuration files transmitted by the first data center using a thread pool.

[0209] Figure 13 A flowchart illustrating a cross-datacenter file disaster recovery synchronization method provided in an embodiment of this application is shown below. Figure 13 As shown, the process of receiving real-time synchronized files transmitted from the first data center using a thread pool via a file receiving tool includes:

[0210] S1301: Use the file receiving tool to listen to the preset port to obtain file transfer requests from the first data center;

[0211] Specifically, the file receiving tool in the second data center is deployed in a cluster and listens on a preset port, waiting to receive file transfer requests from the first data center. When the file transfer tool in the first data center initiates a file transfer request, the file receiving tool receives the real-time synchronized files.

[0212] In one embodiment, the file receiving tool is an FSERVER server, and the preset port it listens on is port 9090.

[0213] In one embodiment, the FSERVER server is based on a network communication framework (such as the MINA framework) and relies on Java's NIO library (non-blocking I / O), which simplifies the complexity of network programming, encapsulates the low-level details of Socket communication, and supports efficient file reception and processing.

[0214] S1302: When a file transfer request is received from the first data center, the real-time synchronized file is received.

[0215] Specifically, the file receiving tool receives real-time synchronized files, stores them in a second data center, ensures that all files are successfully stored, completes data backup, and supports disaster recovery needs.

[0216] In one embodiment, the health check of the primary data center (first data center) origin server is performed via the HTTP layer. A small icon with a first preset name is placed in the NAS directory of the primary data center origin server. This icon is accessed at preset time intervals to determine the health of the primary data center origin server; if the icon can be accessed normally, the primary data center origin server is considered healthy. Similarly, the health check of the backup data center (second data center) origin server is performed via the HTTP layer. A small icon with a first preset name is placed in the official directory of the backup data center origin server. This icon is accessed at preset time intervals to determine the health of the backup data center origin server; if the icon can be accessed normally, the backup data center origin server is considered healthy.

[0217] When the primary data center fails or a disaster recovery drill is required, the disaster recovery management system issues a command to switch the origin server, indirectly calling the shell switching script on the primary data center origin server to rename the icon to a second and first preset name. At this time, origin detection requests will access the original first preset name icon. If a 404 error is returned and persists for a certain number of times, the CDN will automatically switch the detection request to the backup data center origin server. Since the first preset name icon on the backup data center origin server is now accessible, subsequent CDN node origin requests will successfully switch to the backup data center origin server.

[0218] This application provides a cross-datacenter file disaster recovery synchronization method. It involves scanning a temporary directory every third preset time interval to obtain the timed synchronization file transmitted from the first data center; searching the master file table for the corresponding business ID of the timed synchronization file based on its file symbolic link path; querying the master file table for the file object of the previous file version corresponding to the business ID; if the query fails, the timed synchronization file is identified as a formal file; if the query succeeds, incremental file merging is performed based on the timed synchronization file and the file object to generate a complete file, and the complete file is verified as a formal file; the formal file is copied to the formal directory, and its file information is written to the received file table and the master file table. This method achieves real-time and accurate file resource synchronization, meets the requirements of disaster recovery deployment, and ensures high availability in the face of failures.

[0219] The strategy combines high-real-time file synchronization with end-of-day full file synchronization, effectively collecting and synchronizing changed files, ensuring the timeliness and integrity of file synchronization, and guaranteeing the business requirements for real-time files. Temporary directory transfers prevent the mistransmission and overwriting of large batches of files. Incremental file splitting significantly reduces intranet bandwidth consumption. The design of the file table structure further improves reliability and flexibility. Single-domain calls reduce implementation costs and simplify the calling logic.

[0220] This application also provides a cross-datacenter file disaster recovery synchronization device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to the cross-datacenter file disaster recovery synchronization method, the implementation of this device can refer to the implementation of the cross-datacenter file disaster recovery synchronization method; repeated details will not be elaborated further.

[0221] Figure 14 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application is shown below. Figure 14 As shown, the device includes:

[0222] The second file acquisition unit 1401 is used to scan the temporary directory every third preset time interval to acquire the timed synchronization file transmitted from the first data center.

[0223] The second file lookup unit 1402 is used to look up the business ID corresponding to the timed synchronization file in the total file table according to the file soft link path of the timed synchronization file.

[0224] The second file query unit 1403 is used to query the file object of the previous file version corresponding to the service ID in the total file table according to the service ID of the timed synchronization file;

[0225] The second file determination unit 1404 is used to determine the timed synchronization file as a formal file if the query result is a query failure; if the query result is a query success, it performs incremental file merging based on the timed synchronization file and the file object to generate a complete file, and verifies whether the complete file is a formal file.

[0226] The file copying unit 1405 is used to copy the official file to the official directory and write the file information of the official file into the received file table and the total file table.

[0227] Figure 15 This is a schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application. Figure 14 Based on the embodiments, further, such as Figure 15 As shown, the cross-datacenter file disaster recovery synchronization device provided in this application also includes:

[0228] File MD5 calculation module 1501 is used to calculate the file MD5 of the complete file and look up the file MD5 of the current version file in the total file table;

[0229] The file MD5 comparison module 1502 is used to determine the complete file as the official file if the file MD5 of the complete file is consistent with the file MD5 of the current version file.

[0230] Figure 16 This is a schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application. Figure 14 Based on the embodiments, further, such as Figure 16 As shown, the cross-datacenter file disaster recovery synchronization device provided in this application also includes:

[0231] The file synchronization determination unit 1601 is used to compare the received file table with the acquired file table to be sent from the first data center at a preset time to determine the file synchronization status; the fields of the received file table include: transmission task ID, file ID, transmission source, target data center, receiving status, creation time, update time and original data center.

[0232] Figure 17 This is a schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application. Figure 14 Based on the embodiments, further, such as Figure 17As shown, the cross-datacenter file disaster recovery synchronization device provided in this application also includes:

[0233] The file receiving unit 1701 is used to receive real-time synchronized files transmitted by the first data center using a thread pool using a file receiving tool; the real-time synchronized files include production release files and business configuration files.

[0234] Figure 18 This is a schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided in an embodiment of this application. Figure 17 Based on the embodiments, further, such as Figure 18 As shown, the cross-datacenter file disaster recovery synchronization device provided in this application also includes:

[0235] The request listening module 1801 is used to listen to a preset port using the file receiving tool in order to obtain file transfer requests from the first data center.

[0236] The file receiving module 1802 is used to receive the real-time synchronized file when a file transfer request is received from the first data center.

[0237] This application provides a cross-datacenter file disaster recovery synchronization method and apparatus. It obtains timed synchronization files transmitted from a first datacenter by scanning a temporary directory every third preset time interval; it searches the master file table for the service ID corresponding to the timed synchronization file based on the file's symbolic link path; it then queries the master file table for the file object of the previous file version corresponding to the service ID; if the query fails, the timed synchronization file is identified as a formal file; if the query succeeds, incremental file merging is performed based on the timed synchronization file and the file object to generate a complete file, and the complete file is verified as a formal file; the formal file is copied to the formal directory, and its file information is written to the received file table and the master file table. This achieves real-time and accurate file resource synchronization, meets the requirements of disaster recovery deployment, and ensures high availability in the face of failures.

[0238] The strategy combines high-real-time file synchronization with end-of-day full file synchronization, effectively collecting and synchronizing changed files, ensuring the timeliness and integrity of file synchronization, and guaranteeing the business requirements for real-time files. Temporary directory transfers prevent the mistransmission and overwriting of large batches of files. Incremental file splitting significantly reduces intranet bandwidth consumption. The design of the file table structure further improves reliability and flexibility. Single-domain calls reduce implementation costs and simplify the calling logic.

[0239] It should be noted that the cross-data center file disaster recovery synchronization method provided in this application embodiment can be used in the financial field, or in any technical field other than the financial field. This application embodiment does not limit the application field of the cross-data center file disaster recovery synchronization method.

[0240] Figure 19 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 19 As shown, the electronic device includes: a processor 1901, a memory 1902, and a bus 1903.

[0241] The processor 1901 and the memory 1902 communicate with each other via the bus 1903.

[0242] The processor 1901 is used to call program instructions in the memory 1902 to execute the methods provided in the above-described method embodiments.

[0243] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described cross-data center file disaster recovery synchronization method.

[0244] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described cross-data center file disaster recovery synchronization method.

[0245] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0246] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0247] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0248] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0249] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A cross-data center file disaster recovery synchronization method, applied to a first data center, characterized in that, include: Scan the network storage directory every first preset time interval to obtain newly generated files; The newly generated files include end-of-day batch files and real-time synchronization files that failed to be transmitted. Find the file MD5 hash of the newly generated file in the list of files to be sent; If the search fails, query the file object of the previous file version corresponding to the business ID in the master file table based on the business ID of the newly generated file; If the query result is a failure, the newly generated file will be designated as a scheduled synchronization file; If the query result is successful, incremental file segmentation is performed based on the newly generated file and the file object to generate a timed synchronization file; The timed synchronization file is transferred to a temporary directory in the second data center; When a real-time synchronization file is generated, a thread pool is used to transfer the real-time synchronization file to the second data center; The real-time synchronized files include production release files and business configuration files; The step of transferring the timed synchronization file to the temporary directory of the second data center includes: The file information of the scheduled synchronized files is written into the file to be sent table and the total file table; the fields of the file to be sent table include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time, and original data center; the fields of the total file table include: file ID, service ID, file path, file symbolic link path, file MD5, file version, whether the file is deployed on CDN, creation time, creator, update time, updater, resource type, and file size; Every second preset time interval, the list of files to be sent is scanned, and timed synchronization files with transmission status of not transmitted or transmission failure are transmitted to the temporary directory of the second data center.

2. The method as described in claim 1, characterized in that, The step of incrementally splitting files based on the newly generated file and the file object to generate timed synchronization files includes: The newly generated file and the file object are divided into multiple file blocks. The file blocks of the newly generated file are used to perform sliding difference detection on the file blocks of the file object in order to generate a timed synchronization file.

3. The method as described in claim 1, characterized in that, The process of using a thread pool to transfer the real-time synchronized files to the second data center includes: The task of transmitting the real-time synchronized file is encapsulated as a thread pool task; The thread pool task is submitted to the thread pool to transfer the real-time synchronized file to the second data center using a file transfer tool.

4. A cross-data center file disaster recovery synchronization method, applied to a second data center, characterized in that, include: Every third preset time interval, the temporary directory is scanned to obtain the timed synchronization files transmitted from the first data center; Based on the file symbolic link path of the scheduled synchronization file, find the business ID corresponding to the scheduled synchronization file in the master file table; Based on the service ID of the scheduled synchronization file, query the file object of the previous file version corresponding to the service ID in the master file table; If the query result is a failure, the scheduled synchronization file will be designated as the official file. If the query result is successful, incremental file merging is performed based on the scheduled synchronization file and the file object to generate a complete file, and the complete file is verified to be an official file; Copy the official file to the official directory and write the file information of the official file into the received file table and the total file table; The received file table and the acquired file table to be sent from the first data center are compared at a preset time to determine the file synchronization status. The fields in the received file table include: transmission task ID, file ID, transmission source, target data center, reception status, creation time, update time, and original data center. The verification of whether the complete file is an official file includes: The MD5 hash of the complete file is calculated, and the MD5 hash of the current version file is found in the total file table. If the MD5 hash of the complete file matches the MD5 hash of the current version file, the complete file is identified as the official file.

5. The method as described in claim 4, characterized in that, Also includes: Utilize a file receiving tool to receive real-time synchronized files transmitted from the first data center using a thread pool; The real-time synchronized files include production release files and business configuration files.

6. The method as described in claim 5, characterized in that, The process of receiving real-time synchronized files transmitted from the first data center using a thread pool via a file receiving tool includes: The file receiving tool is used to listen to a preset port in order to obtain file transfer requests from the first data center; When a file transfer request is received from the first data center, the real-time synchronized file is received.

7. A cross-data center file disaster recovery synchronization device, characterized in that, include: The first file acquisition unit is used to scan the network storage directory every first preset time interval to acquire newly generated files; The newly generated files include end-of-day batch files and real-time synchronization files that failed to be transmitted. The first file search unit is used to search the file MD5 of the newly generated file in the file to be sent table; The first file query unit is used to query the file object of the previous file version corresponding to the business ID in the total file table if the search fails; The first file determination unit is used to determine the newly generated file as a timed synchronization file if the query result is a query failure. If the query result is successful, incremental file segmentation is performed based on the newly generated file and the file object to generate a timed synchronization file; The first file transfer unit is used to transfer the timed synchronization file to the temporary directory of the second data center; The second file transfer unit is used to transfer the real-time synchronized file to the second data center using a thread pool when a real-time synchronized file is generated. The real-time synchronized files include production release files and business configuration files; The first file transfer unit includes: The information writing module is used to write the file information of the timed synchronized files into the file table to be sent and the total file table. The fields of the file table to be sent include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time, and original data center. The fields of the total file table include: file ID, service ID, file path, file symbolic link path, file MD6, file version, whether the file is deployed via CDN, creation time, creator, update time, updater, resource type, and file size. The first file transfer module is used to scan the file table to be sent every second preset time interval and transfer the timed synchronization files with the transmission status of not transmitted or transmission failure to the temporary directory of the second data center.

8. A cross-data center file disaster recovery synchronization device, characterized in that, include: The second file acquisition unit is used to scan the temporary directory every third preset time interval to acquire the timed synchronization file transmitted from the first data center. The second file lookup unit is used to look up the business ID corresponding to the timed synchronization file in the master file table based on the file soft link path of the timed synchronization file. The second file query unit is used to query the file object of the previous file version corresponding to the service ID in the master file table according to the service ID of the timed synchronization file; The second file determination unit is used to determine the timed synchronization file as an official file if the query result is a query failure. If the query result is successful, incremental file merging is performed based on the scheduled synchronization file and the file object to generate a complete file, and the complete file is verified to be an official file; The file copying unit is used to copy the official file to the official directory and write the file information of the official file into the received file table and the total file table; The file synchronization determination unit is used to compare the received file table with the acquired file table to be sent from the first data center at a preset time to determine the file synchronization status. The fields in the received file table include: transmission task ID, file ID, transmission source, target data center, reception status, creation time, update time, and original data center. The second document determination unit includes: The file MD5 calculation module is used to calculate the file MD5 of the complete file and look up the file MD5 of the current version file in the total file table. The file MD5 comparison module is used to determine the complete file as the official file if the file MD5 of the complete file matches the file MD5 of the current version file.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images