Cross-data-center file disaster recovery synchronization method and device

Abstract

The invention discloses a cross-data-center file disaster recovery synchronization method and device, relates to the technical field of information security and can also be used in the financial field, and the method comprises the following steps: scanning a network storage directory every first preset time interval to obtain a newly generated file; the newly generated file comprises a day-end batch file and a real-time synchronization file which fails to be transmitted; searching a file MD5 of a newly generated file in a to-be-sent file table; if the search fails, querying a file object of a previous file version corresponding to a service ID in a total file table according to the service ID of the newly generated file; if the query result is query failure, determining the newly generated file as a timing synchronization file; if the query result is that the query is successful, performing incremental file segmentation according to the newly generated file and the file object to generate a timing synchronization file; and the timing synchronization file is transmitted to the temporary directory of the second data center, so that the real-time performance and accuracy of file resource synchronization are realized, the disaster recovery requirement is met, and the high availability for faults is ensured.

Description

Technical Field

[0001] The present application relates to the field of information security technology and can also be used in the financial field, and in particular to a method and device for synchronizing files in disaster recovery across data centers. Background Art

[0002] Mobile banking is a product in the field of Internet applications, providing customers with a wealth of business services. Static file resource packages are widely used in the technical implementation of business functions. The mobile client obtains the CDN download address of the file through the mobile server of three centers in different locations and sends a request to the CDN node to obtain the corresponding file. Once the CDN node receives the request, it determines whether it hits the cache. If it does not hit, it needs to return to the source station to obtain the file. Among them, CDN is a content delivery network (Content Delivery Network), which is used to provide edge acceleration services for user access.

[0003] In order to achieve high availability of the file resource package accessed by the client on CDN, the link between the client and the CDN node and the link between the CDN node and the source station are crucial. Whether the above two links are highly available determines the high availability of the entire link. The link between the client and the CDN node involves the operator's DNS domain name resolution, the number of CDN manufacturers' node deployment, and the CDN manufacturer's traffic scheduling strategy, and there is limited room for optimization; as for the link between the CDN node and the source station, there are currently two source stations logically, but only one physically, so there is room for optimization.

[0004] At present, considering the high availability of CDN back-to-source, the source station has applied for two CDN domain names at the same time when deploying resource files in a single computer room and a single data center, one of which is the primary domain name and the other is the backup domain name. The primary domain name and the backup domain name are resolved to two different web server clusters in the same computer room respectively. Web requests are forwarded to the same NAS file server through load balancing. Usually, the primary domain name mainly provides resource access services for clients. When the primary domain name accesses resources abnormally for many times, the client tries to obtain resources through the backup domain name to ensure the customer's business use.

[0005] The disadvantage of the above method is that it cannot meet the business requirements of high availability across computer rooms. When deploying static resource files, although two different web clusters are deployed in the same computer room, in the extreme case that the computer room is unavailable as a whole and the CDN cache fails, it is impossible to return to the source to access the file. It is impossible to provide transaction services for file downloads when CDN access is abnormal, and it is impossible to provide computer room-level horizontal expansion of download transactions. In addition, this method uses two domain names. When the primary domain name fails, the client obtains resources through the backup domain name. The call logic is complex, the call timing is uncertain, and the cost of applying for two domain names at the same time is too high.

[0006] This section is intended to provide a background or context to the embodiments of the present application as recited in the claims. No admission is made that the description herein is prior art by inclusion in this section. Summary of the invention

[0007] The present application provides a cross-data center file disaster recovery synchronization method, which is applied to a first data center to perform file disaster recovery synchronization. The method includes:

[0008] Scan the network storage directory at a first preset time interval to obtain newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that fail to be transmitted;

[0009] Search the file MD5 of the newly generated file in the file list to be sent;

[0010] If the search fails, query the file object of the previous file version corresponding to the business ID in the total file table according to the business ID of the newly generated file;

[0011] If the query result is a query failure, the newly generated file is determined as a scheduled synchronization file;

[0012] If the query result is successful, incremental file segmentation is performed according to the newly generated file and the file object to generate a timed synchronization file;

[0013] The timed synchronization file is transmitted to a temporary directory of the second data center.

[0014] Further, performing incremental file segmentation according to the newly generated file and the file object to generate a timed synchronization file includes:

[0015] Splitting the newly generated file and the file object to obtain a plurality of file blocks respectively;

[0016] The file blocks of the file object are used to perform sliding difference detection on the file blocks of the newly generated file to generate a timed synchronization file.

[0017] Further, the transmitting the timed synchronization file to a temporary directory of the second data center includes:

[0018] Write the file information of the scheduled synchronization file into the to-be-sent file table and the total file table; the fields of the to-be-sent file table include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time and original computer room center; the fields of the total file table include: file ID, service ID, file path, file soft link path, file MD5, file version, whether the file is CDN deployed, creation time, creator, update time, updater, resource type and file size;

[0019] The to-be-sent file list is scanned at every second preset time interval, and the scheduled synchronization files with a transmission status of not transmitted or failed to be transmitted are transmitted to a temporary directory of the second data center.

[0020] Furthermore, it also includes:

[0021] When a real-time synchronization file is generated, the real-time synchronization file is transmitted to the second data center by using a thread pool; the real-time synchronization file includes a production release file and a business configuration file.

[0022] Furthermore, the using a thread pool to transmit the real-time synchronization file to the second data center includes:

[0023] Encapsulating the task of transmitting the real-time synchronization file as a thread pool task;

[0024] The thread pool task is submitted to the thread pool, so as to utilize a file transfer tool to transfer the real-time synchronization file to the second data center.

[0025] The present application provides a cross-data center file disaster recovery synchronization method, which is applied to a second data center to perform file disaster recovery synchronization. The method includes:

[0026] Scan the temporary directory at every third preset time interval to obtain the received timed synchronization file transmitted by the first data center;

[0027] Searching the service ID corresponding to the scheduled synchronization file in the general file table according to the file soft link path of the scheduled synchronization file;

[0028] According to the service ID of the scheduled synchronization file, query the file object of the previous file version corresponding to the service ID in the general file table;

[0029] If the query result is a query failure, the scheduled synchronization file is determined as a formal file;

[0030] If the query result is successful, perform incremental file merging according to the scheduled synchronization file and the file object to generate a complete file, and verify whether the complete file is a formal file;

[0031] The formal file is copied to the formal directory and the file information of the formal file is written into the received file table and the total file table.

[0032] Furthermore, the verifying whether the complete document is a formal document includes:

[0033] The file MD5 of the complete file is calculated and the file MD5 of the current version file is searched in the total file table;

[0034] If the file MD5 of the complete file is consistent with the file MD5 of the current version file, the complete file is determined as a formal file.

[0035] Furthermore, it also includes:

[0036] At a preset time, the received file table is compared with the obtained list of files to be sent of the first data center to determine the file synchronization status; the fields of the received file table include: transmission task ID, file ID, transmission source, target data center, receiving status, creation time, update time and original computer room center.

[0037] Furthermore, it also includes:

[0038] The real-time synchronization file transmitted by the first data center using the thread pool is received by using a file receiving tool; the real-time synchronization file includes a production release file and a business configuration file.

[0039] Further, the using of the file receiving tool to receive the real-time synchronization file transmitted by the first data center using the thread pool includes:

[0040] Using the file receiving tool to monitor a preset port to obtain a file transfer request from the first data center;

[0041] When the file transfer request from the first data center is obtained, the real-time synchronization file is received.

[0042] The embodiment of the present application also provides a cross-data center file disaster recovery synchronization device for performing file disaster recovery synchronization, the device comprising:

[0043] A first file acquisition unit, configured to scan the network storage directory at a first preset time interval to acquire newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that fail to be transmitted;

[0044] A first file search unit, configured to search the file MD5 of the newly generated file in the to-be-sent file table;

[0045] A first file query unit, configured to query the file object of the previous file version corresponding to the service ID of the newly generated file in the general file table if the search fails;

[0046] A first file determination unit is used to determine the newly generated file as a scheduled synchronization file if the query result is a query failure; if the query result is a query success, perform incremental file segmentation according to the newly generated file and the file object to generate a scheduled synchronization file;

[0047] The first file transmission unit is used to transmit the scheduled synchronization file to a temporary directory of the second data center.

[0048] The embodiment of the present application also provides a cross-data center file disaster recovery synchronization device for performing file disaster recovery synchronization, the device comprising:

[0049] A second file acquisition unit, configured to scan the temporary directory at every third preset time interval to acquire the received timed synchronization file transmitted by the first data center;

[0050] A second file search unit, configured to search the total file table for a service ID corresponding to the timing synchronization file according to a file soft link path of the timing synchronization file;

[0051] A second file query unit, configured to query the total file table for a file object of a previous file version corresponding to the service ID of the timed synchronization file according to the service ID of the timed synchronization file;

[0052] A second file determination unit is used to determine the scheduled synchronization file as a formal file if the query result is a query failure; if the query result is a query success, perform incremental file merging according to the scheduled synchronization file and the file object to generate a complete file, and verify whether the complete file is a formal file;

[0053] The file copying unit is used to copy the formal file to the formal directory and write the file information of the formal file into the received file table and the total file table.

[0054] An embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned cross-data center file disaster recovery synchronization method when executing the computer program.

[0055] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned cross-data center file disaster recovery synchronization method is implemented.

[0056] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned cross-data center file disaster recovery synchronization method is implemented.

[0057] The cross-data center file disaster recovery synchronization method and device provided by the embodiment of the present application obtain newly generated files by scanning the network storage directory at a first preset time interval by the first data center; the newly generated files include end-of-day batch files and real-time synchronization files that fail to be transmitted; the file MD5 of the newly generated file is searched in the file table to be sent; if the search fails, the file object of the previous file version corresponding to the business ID of the newly generated file is searched in the total file table according to the business ID of the newly generated file; if the query result is a query failure, the newly generated file is determined as a scheduled synchronization file; if the query result is a query success, incremental file segmentation is performed according to the newly generated file and the file object to generate a scheduled synchronization file; the scheduled synchronization file is transmitted to a temporary directory of the second data center, and the second data center scans the temporary directory at a third preset time interval. record, obtain the timed synchronization file transmitted by the first data center; search the business ID corresponding to the timed synchronization file in the total file table according to the file soft link path of the timed synchronization file; search the file object of the previous file version corresponding to the business ID in the total file table according to the business ID of the timed synchronization file; if the query result is a query failure, determine the timed synchronization file as a formal file; if the query result is a query success, perform incremental file merging according to the timed synchronization file and the file object to generate a complete file, and verify whether the complete file is a formal file; copy the formal file to the formal directory and write the file information of the formal file into the received file table and the total file table, thereby realizing the real-time and accuracy of file resource synchronization, meeting the requirements of disaster recovery deployment, and ensuring high availability in the face of failures.

[0058] Among them, the strategy of combining high-real-time file synchronization with full-file synchronization at the end of the day is adopted to achieve effective collection and synchronization of changed files, ensure the timeliness and integrity of file synchronization, and ensure the business needs of real-time files; through temporary directory transfer, it is possible to prevent large quantities of files from being mistransmitted and overwritten; through incremental cutting of files, it is possible to significantly reduce the consumption of intranet bandwidth; through the design of the file table structure, reliability and flexibility are further improved; through single domain name calling, the implementation cost is reduced and the calling logic is simplified. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0060] Figure 1A flowchart of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application;

[0061] Figure 2 A flowchart of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application;

[0062] Figure 3 A flowchart of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application;

[0063] Figure 4 A flowchart of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application;

[0064] Figure 5 A schematic diagram of the process of production release and file synchronization provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0066] Figure 7 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0067] Figure 8 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0068] Fig. 9 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0069] Fig.10 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0070] Fig.11 A flowchart of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application;

[0071] Fig.12 A flowchart of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application;

[0072] Fig.13 A flowchart of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application;

[0073] Fig.14 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0074] Fig.15A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0075] Fig.16 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0076] Fig.17 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0077] Fig.18 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application;

[0078] Fig.19 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present application more clear, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. Here, the illustrative embodiments of the present application and their descriptions are used to explain the present application, but are not intended to limit the present application.

[0080] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0081] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0082] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0083] There is a risk of single point failure when storing files in a single data center. Therefore, this application provides a disaster recovery deployment for storing file resource packages in multiple data centers. In abnormal situations, the disaster recovery center issues a source station switching instruction, where the first data center serves as the main data center for production release and business configuration file resource release and CDN back to the source, and the second data center serves as a backup data center. In one embodiment, the third data center serves as another backup data center. This application is not limited to the number of data centers.

[0084] The primary data center and backup data center are configured in different computer rooms. The configuration for client access to CDN is adjusted from the original two primary and backup domain names to only one domain name, which reduces the implementation cost and simplifies the calling logic.

[0085] Figure 1 A schematic diagram of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application is shown in FIG. Figure 1 As shown, the cross-data center file disaster recovery synchronization method provided by the present application includes:

[0086] S101: Scan the network storage directory at a first preset time interval to obtain newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that fail to be transmitted;

[0087] S102: Searching for the file MD5 of the newly generated file in the file table to be sent;

[0088] S103: If the search fails, query the file object of the previous file version corresponding to the business ID in the general file table according to the business ID of the newly generated file;

[0089] S104: If the query result is a query failure, determine the newly generated file as a scheduled synchronization file; if the query result is a query success, perform incremental file segmentation according to the newly generated file and the file object to generate a scheduled synchronization file;

[0090] S105: Transmit the scheduled synchronization file to a temporary directory of the second data center.

[0091] from Figure 1 As can be seen from the illustrated process, the cross-data center file disaster recovery synchronization method provided by the present application obtains newly generated files by scanning the network storage directory at a first preset time interval; the newly generated files include end-of-day batch files and real-time synchronization files that fail to be transmitted; the file MD5 of the newly generated file is searched in the file table to be sent; if the search fails, the file object of the previous file version corresponding to the business ID of the newly generated file is queried in the total file table according to the business ID of the newly generated file; if the query result is a query failure, the newly generated file is determined as a scheduled synchronization file; if the query result is a query success, incremental file segmentation is performed according to the newly generated file and the file object to generate a scheduled synchronization file; the scheduled synchronization file is transmitted to the temporary directory of the second data center, thereby achieving real-time and accuracy of file resource synchronization, meeting the requirements of disaster recovery deployment, and ensuring high availability in the face of failures.

[0092] The following takes the first data center as the execution entity to explain each step in detail.

[0093] S101: Scan the network storage directory at a first preset time interval to obtain newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that fail to be transmitted;

[0094] Specifically, the first data center starts a file update listener to monitor and scan file changes in the network storage NAS directory. It scans whether there are new files generated or updated in the NAS directory at preset time intervals to ensure that newly generated or updated files can be discovered and recorded in time to avoid missing any important file updates and prepare for subsequent file synchronization and transmission.

[0095] In one embodiment, the file update listener of the first data center starts a process with the user "bye", uses the Linux find command to monitor the NAS directory, executes a scan command, and searches for files that are newly generated or updated in the specified directory within a recent period of time.

[0096] In one embodiment, the preset time interval may be 5 minutes, but the present application is not limited thereto.

[0097] In one embodiment, in emergency situations, the file scanning task can be manually triggered to ensure that file updates can be discovered and processed in a timely manner even under special circumstances, thereby increasing the flexibility of the system and its ability to cope with emergencies, and ensuring that the effectiveness and accuracy of file synchronization can be maintained under any circumstances.

[0098] S102: Searching for the file MD5 of the newly generated file in the file table to be sent;

[0099] Specifically, for the newly generated file scanned, the first data center searches the table of files to be sent (FT_IDC_SYN_FILE_SEND) to see if there is already a file record of the same file MD5. If it cannot be found and the trigger type of the file is "real-time synchronization" or "scheduled synchronization", the file object of the previous file version corresponding to the service ID of the newly generated file is searched in the general file table according to the service ID of the newly generated file.

[0100] S103: If the search fails, query the file object of the previous file version corresponding to the business ID in the general file table according to the business ID of the newly generated file;

[0101] Specifically, each resource file is generated after code compilation. For most businesses, the code for this production is compared with the previous code baseline, and the modified parts are very small, but a complete resource package is generated for each production. Modifying a line of content will also generate a complete resource package of several megabytes. If the complete package is transmitted when synchronizing files, it will cause a waste of intranet bandwidth. Therefore, no matter what type of file, in order to save bandwidth during scheduled unified transmission, the files will be merged after differential transmission.

[0102] If there is no file record of the same file MD5 in the table of files to be sent, before file synchronization, the first data center first searches the general file table (FT_BASE_FILE) for the file object of the last file version timestamp corresponding to the same business ID (BUSINESS_ID). By searching for the last version file, the first data center can determine whether to perform full file synchronization or incremental synchronization, thus avoiding unnecessary file transfers.

[0103] S104: If the query result is a query failure, determine the newly generated file as a scheduled synchronization file; if the query result is a query success, perform incremental file segmentation according to the newly generated file and the file object to generate a scheduled synchronization file;

[0104] Specifically, the first data center determines the scheduled synchronization file that needs to be transmitted to the second data center based on the query result of the file object of the last file version timestamp corresponding to the service ID in the general file table.

[0105] If the business ID is put into production for the first time, that is, the query result of the file object with the previous file version timestamp corresponding to the business ID in the total file table is a query failure, the first data center will directly synchronize the complete large file of this production as a scheduled synchronization file to the second data center.

[0106] If there is a previous version of the file, that is, the query result of the file object with the previous file version timestamp corresponding to the business ID in the total file table is successful, the first data center uses the file processing tool to compare the file put into production this time with the previous version of the file, calculates and generates an incremental file (Delta) containing only the different parts. The incremental file contains the changes between the new and old versions. By generating incremental files, the amount of data transmission is significantly reduced, the use of intranet bandwidth is optimized, and the bandwidth waste caused by transmitting complete file packages due to small changes is avoided.

[0107] In one embodiment, the file processing tool is an rsync4j tool, but the present application is not limited thereto.

[0108] Figure 2 A schematic diagram of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application is shown in FIG. Figure 2 As shown, S104 includes:

[0109] S201: splitting the newly generated file and the file object to obtain a plurality of file blocks respectively;

[0110] Specifically, the first data center divides the previous version of the file and the current production version of the file of the same business ID (BUSSINESS_ID) into blocks according to preset sizes for subsequent comparison.

[0111] In one embodiment, the first data center calculates a strong summary (such as hash values ​​such as MD5 and SHA) and a weak summary (such as a checksum) for each file block, and concatenates the strong and weak summaries of all blocks to form a signature string, which represents the block summary information of the previous version of the file.

[0112] S202: Perform sliding difference detection on the file blocks of the newly generated file using the file blocks of the file object to generate a timed synchronization file.

[0113] Specifically, the first data center performs a sliding difference check on the current version file to generate an incremental file as a scheduled synchronization file. The sliding window difference check is performed on the current production version file. Through the sliding window mechanism, the blocks of the current version file and the previous version file are compared block by block to detect which blocks are the same as the blocks of the previous version file and which blocks are different. For the same blocks, only their serial numbers in the current file are recorded; for different blocks, their contents are recorded.

[0114] The first data center concatenates the serial numbers of the same blocks and the contents of different blocks to form a Delta incremental file, which contains the changed parts and the corresponding serial numbers, representing the difference between the current version and the previous version.

[0115] S105: Transmit the scheduled synchronization file to a temporary directory of the second data center.

[0116] Specifically, the first data center starts a file sending timer, transmits the timed synchronization file to a temporary directory of the second data center at a preset time, and updates the file transmission status after the file is successfully transmitted.

[0117] In one embodiment, in an emergency situation, the first data center allows the background to manually trigger the file sending task to deal with the situation where the automatic scheduled transmission task cannot be executed normally, providing additional flexibility to ensure that in special or emergency situations, the file transfer task can still be executed in a timely manner without affecting the continuity of the overall business.

[0118] In one embodiment, scheduled synchronous file transfer can be performed by calling the NFTSA tool, which is a new generation of file transfer tool based on the security component (SA), similar to the FTP tool. The transmission direction includes the client sending files to the server, or the client pulling files from the server.

[0119] Figure 3A schematic diagram of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application is shown in FIG. Figure 3 As shown, S105 includes:

[0120] S301: Write the file information of the scheduled synchronization file into the to-be-sent file table and the total file table; the fields of the to-be-sent file table include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time and original computer room center; the fields of the total file table include: file ID, service ID, file path, file soft link path, file MD5, file version, whether the file is CDN deployed, creation time, creator, update time, updater, resource type and file size;

[0121] Specifically, the information of the scheduled synchronization files is written into the file table to be sent (FT_IDC_SYN_FILE_SEND) and the total file table (FT_BASE_FILE), and the recorded content includes important information such as the file ID, path, MD5 value, version timestamp, etc., to ensure that all files that need to be synchronized are accurately recorded, and provide necessary data support for subsequent file transfers to ensure the accuracy and consistency of synchronization. Among them, the fields of the file table to be sent include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time and original computer room center; the fields of the total file table include: file ID, business ID, file path, file soft link path, file MD5, file version, whether the file is CDN deployed, creation time, creator, update time, updater, resource type and file size.

[0122] In one embodiment, the table structure of the total file table is shown in Table 1, and the table structure of the file table to be sent is shown in Table 2. By designing the table structure, a new standby computer room can be quickly expanded, further improving reliability and flexibility.

[0123] Table 1

[0124]

[0125]

[0126] Table 2

[0127]

[0128]

[0129] S302: Scan the to-be-sent file list at every second preset time interval, and transmit the scheduled synchronization files whose transmission status is not transmitted or failed to be transmitted to a temporary directory of the second data center.

[0130] Specifically, the first data center starts the file sending timer, queries the task table of files to be sent every preset time interval, and selects the records with the status of "not sent" or "failed to send" and the number of times sent does not exceed the preset number. To prevent reading too many records at a time, a preset number of records are read in batches each time, which improves processing efficiency and ensures smooth execution of each transmission task, avoiding excessive load caused by reading too many records.

[0131] In one embodiment, the preset time interval is 10 minutes, the preset number of times is 100 times, and the preset number is 100 items, but the present application is not limited thereto.

[0132] In one embodiment, the user who starts the file sending timer in the first data center is "bye", and the process of the "bye" user calls the process service of the "nftsa" user, and passes in corresponding parameters (such as the IP address and port of the second data center NFTSA, the local directory address and the target directory address, etc.), and transfers the file to be sent to the temporary directory of the second data center. Delta files are transitional files that need to be temporarily stored in a temporary directory and then merged to generate available files, ensuring that even if problems occur during the transmission process, they will not affect the business files in the formal directory, thereby improving the security of file transmission.

[0133] After successfully transmitting the file, the first data center will update the status field STATUS in the file table to be sent to "1", indicating that the transmission is successful, and increase the number of sends (SEND_NUMBER) by 1 to ensure that the progress of the transmission task can be tracked in real time and provide a basis for subsequent exception handling and retransmission tasks.

[0134] In one embodiment, the cross-data center file disaster recovery synchronization method further includes:

[0135] When a real-time synchronization file is generated, the real-time synchronization file is transmitted to the second data center by using a thread pool; the real-time synchronization file includes a production release file and a business configuration file.

[0136] Specifically, typical scenarios for generating new file resource packages for mobile banking include release on the production day, generation after maintenance and configuration by back-end business personnel on the workbench, and batch generation at the end of the day. Regardless of the scenario, to ensure the consistency of the deployment of file resource packages, all new file resource packages will be uniformly released and take effect in the main data center. When put into production, the main data center will dynamically generate the CDN version information, file MD5, file version timestamp, directory address of the resource on the NAS file server, key data of the resource, production record, and CDN LOGO file address of each resource package, and store these data in the database record table of the main data center. At the same time, each file resource package will also be saved in the corresponding dynamically generated NAS file directory.

[0137] Dual-center file storage is a necessary prerequisite for disaster recovery switching. Files in the primary data center need to be synchronized to the backup data center as a backup. File synchronization uses a combination of real-time synchronization and scheduled synchronization. Files released on the production day and business configuration files are synchronized in real time to ensure that real-time meets disaster recovery requirements; files generated in batches at the end of the day and real-time synchronization files that fail to synchronize are fully transmitted on a scheduled basis until they succeed or the number of retransmissions is exceeded.

[0138] When the first data center (primary data center) is launching a production release or generating a business configuration file, it transmits the newly generated file as a real-time synchronization file to the second data center (backup data center) to achieve file disaster recovery synchronization.

[0139] Figure 4 A schematic diagram of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application is shown in FIG. Figure 4 As shown, using the thread pool to transmit the real-time synchronization file to the second data center includes:

[0140] S401: Encapsulating the task of transmitting the real-time synchronization file as a thread pool task;

[0141] Specifically, when a new version of a file needs to be released in the first data center, the main business process will be triggered to be put into production and released. The first data center triggers the main business process to execute the production release, and after the new file version to be synchronized is prepared, the file synchronization operation is encapsulated as a thread pool task to ensure that the synchronization process does not affect the main business process.

[0142] S402: Submit the thread pool task to the thread pool, so as to use a file transfer tool to transfer the real-time synchronization file to the second data center.

[0143] Specifically, the first data center submits the encapsulated file synchronization task to the thread pool and prepares to perform the synchronization operation; after receiving the task, the thread pool uses the file transfer tool to transfer the file to the second data center in real time. The thread pool is a task manager that processes file synchronization tasks asynchronously and concurrently to avoid the impact of synchronization tasks on the main business process. The file transfer tool receives the file synchronization task assigned by the thread pool and transfers the file from the first data center to the second data center in real time through the network, ensuring that the file is not lost during the transmission process and can successfully reach the target data center.

[0144] In one embodiment, the file transfer tool is the FCLIENT tool, and the process of production release and file synchronization is as follows: Figure 5 As shown, including:

[0145] The file resource package is launched and released, and database records such as file version and file MD5 are generated. The file is stored in the official directory of the NAS server, and the file source address, file target address, and file MD5 are recorded, and packaged into a task for sending files in real time;

[0146] Submit the encapsulated task to the sending task thread pool, and asynchronously determine whether there are any release files. If so, return to the previous step for a loop operation. Otherwise, the file resource package is put into production and released;

[0147] After the task is submitted to the sending task thread pool, the address and port of the FSERVER server are obtained from the configuration file, and the FCLIENT sets the API parameters to initiate the transmission;

[0148] Determine whether the transmission is successful. If so, record the transmission result of the file in the file to be sent table, and the transmission status is successful. If not, record the transmission result of the file in the file to be sent table, and the transmission status is failed.

[0149] The present application provides a cross-data center file disaster recovery synchronization method, which obtains newly generated files by scanning a network storage directory at a first preset time interval; the newly generated files include end-of-day batch files and real-time synchronization files that fail to be transmitted; the file MD5 of the newly generated file is searched in a table of files to be sent; if the search fails, the file object of the previous file version corresponding to the business ID of the newly generated file is searched in the total file table according to the business ID of the newly generated file; if the query result is a query failure, the newly generated file is determined as a scheduled synchronization file; if the query result is a query success, incremental file segmentation is performed according to the newly generated file and the file object to generate a scheduled synchronization file; the scheduled synchronization file is transmitted to a temporary directory of a second data center, thereby achieving real-time and accuracy of file resource synchronization, meeting the requirements of disaster recovery deployment, and ensuring high availability in the face of failures.

[0150] Among them, the strategy of combining high-real-time file synchronization with full-file synchronization at the end of the day is adopted to achieve effective collection and synchronization of changed files, ensure the timeliness and integrity of file synchronization, and ensure the business needs of real-time files; through temporary directory transfer, it is possible to prevent large quantities of files from being mistransmitted and overwritten; through incremental cutting of files, it is possible to significantly reduce the consumption of intranet bandwidth; through the design of the file table structure, reliability and flexibility are further improved; through single domain name calling, the implementation cost is reduced and the calling logic is simplified.

[0151] The present application also provides a cross-data center file disaster recovery synchronization device in the embodiment, as described in the following embodiment. Since the principle of the device to solve the problem is similar to the cross-data center file disaster recovery synchronization method, the implementation of the device can refer to the implementation of the cross-data center file disaster recovery synchronization method, and the repeated parts will not be repeated.

[0152] Figure 6 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application is shown in FIG. Figure 6 As shown, the device comprises:

[0153] The first file acquisition unit 601 is used to scan the network storage directory at a first preset time interval to acquire newly generated files; the newly generated files include end-of-day batch files and real-time synchronization files that fail to be transmitted;

[0154] A first file search unit 602, configured to search the file MD5 of the newly generated file in the to-be-sent file table;

[0155] The first file query unit 603 is used to query the file object of the previous file version corresponding to the business ID of the newly generated file in the general file table according to the business ID of the newly generated file if the search fails;

[0156] The first file determination unit 604 is used to determine the newly generated file as a scheduled synchronization file if the query result is a query failure; if the query result is a query success, perform incremental file segmentation according to the newly generated file and the file object to generate a scheduled synchronization file;

[0157] The first file transmission unit 605 is used to transmit the scheduled synchronization file to a temporary directory of the second data center.

[0158] Figure 7 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application, Figure 6 Based on the embodiment, further, Figure 7 As shown, the cross-data center file disaster recovery synchronization device provided by the present application also includes:

[0159] A file segmentation module 701, used to segment the newly generated file and the file object to obtain a plurality of file blocks;

[0160] The difference detection module 702 is used to perform sliding difference detection on the file blocks of the newly generated file using the file blocks of the file object to generate a timed synchronization file.

[0161] Figure 8 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application, Figure 6 Based on the embodiment, further, Figure 8 As shown, the cross-data center file disaster recovery synchronization device provided by the present application also includes:

[0162] The information writing module 801 is used to write the file information of the scheduled synchronization file into the to-be-sent file table and the total file table; the fields of the to-be-sent file table include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time and original computer room center; the fields of the total file table include: file ID, service ID, file path, file soft link path, file MD6, file version, whether the file is CDN deployed, creation time, creator, update time, updater, resource type and file size;

[0163] The first file transmission module 802 is used to scan the to-be-sent file table at every second preset time interval, and transmit the scheduled synchronization files with the transmission status of not transmitted and failed to the temporary directory of the second data center.

[0164] Fig. 9 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application, Figure 6 Based on the embodiment, further, Fig. 9 As shown, the cross-data center file disaster recovery synchronization device provided by the present application also includes:

[0165] The second file transmission unit 901 is used to transmit the real-time synchronization file to the second data center by using the thread pool when the real-time synchronization file is generated; the real-time synchronization file includes a production release file and a business configuration file.

[0166] Fig.10 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application, Fig. 9 Based on the embodiment, further, Fig.10 As shown, the cross-data center file disaster recovery synchronization device provided by the present application also includes:

[0167] The task encapsulation module 1001 is used to encapsulate the task of transmitting the real-time synchronization file into a thread pool task;

[0168] The second file transfer module 1002 is used to submit the thread pool task to the thread pool, so as to use a file transfer tool to transfer the real-time synchronization file to the second data center.

[0169] The present application provides a method and device for synchronizing files in disaster recovery across data centers, which obtain newly generated files by scanning a network storage directory at a first preset time interval; the newly generated files include end-of-day batch files and real-time synchronization files that fail to be transmitted; the file MD5 of the newly generated file is searched in a table of files to be sent; if the search fails, the file object of the previous file version corresponding to the business ID of the newly generated file is searched in a total file table according to the business ID of the newly generated file; if the query result is a query failure, the newly generated file is determined as a scheduled synchronization file; if the query result is a query success, incremental file segmentation is performed according to the newly generated file and the file object to generate a scheduled synchronization file; the scheduled synchronization file is transmitted to a temporary directory of a second data center, thereby achieving real-time and accuracy of file resource synchronization, meeting the requirements of disaster recovery deployment, and ensuring high availability in the face of failures.

[0170] Among them, the strategy of combining high-real-time file synchronization with full-file synchronization at the end of the day is adopted to achieve effective collection and synchronization of changed files, ensure the timeliness and integrity of file synchronization, and ensure the business needs of real-time files; through temporary directory transfer, it is possible to prevent large quantities of files from being mistransmitted and overwritten; through incremental cutting of files, it is possible to significantly reduce the consumption of intranet bandwidth; through the design of the file table structure, reliability and flexibility are further improved; through single domain name calling, the implementation cost is reduced and the calling logic is simplified.

[0171] Fig.11 A schematic diagram of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application is shown in FIG. Fig.11 As shown, the cross-data center file disaster recovery synchronization method provided by the present application includes:

[0172] S1101: Scan the temporary directory at every third preset time interval to obtain the received timed synchronization file transmitted by the first data center;

[0173] S1102: searching the total file table for a service ID corresponding to the scheduled synchronization file according to the file soft link path of the scheduled synchronization file;

[0174] S1103: Querying the file object of the previous file version corresponding to the service ID in the general file table according to the service ID of the scheduled synchronization file;

[0175] S1104: If the query result is a query failure, determine the scheduled synchronization file as a formal file; if the query result is a query success, merge incremental files according to the scheduled synchronization file and the file object to generate a complete file, and verify whether the complete file is a formal file;

[0176] S1105: Copy the official file to the official directory and write the file information of the official file into the received file table and the total file table.

[0177] from Fig.11 As can be seen from the illustrated process, the cross-data center file disaster recovery synchronization method provided by the present application obtains the timed synchronization file transmitted by the first data center by scanning the temporary directory at every third preset time interval; searches the total file table for the business ID corresponding to the timed synchronization file according to the file soft link path of the timed synchronization file; queries the total file table for the file object of the previous file version corresponding to the business ID according to the business ID of the timed synchronization file; if the query result is a query failure, determines the timed synchronization file as a formal file; if the query result is a query success, performs incremental file merging according to the timed synchronization file and the file object to generate a complete file, and verifies whether the complete file is a formal file; copies the formal file to the formal directory and writes the file information of the formal file into the received file table and the total file table, thereby realizing the real-time and accuracy of file resource synchronization, meeting the requirements of disaster recovery deployment, and ensuring high availability in the face of failures.

[0178] The following uses the second data center as the execution entity to explain each step in detail.

[0179] S1101: Scan the temporary directory at every third preset time interval to obtain the received timed synchronization file transmitted by the first data center;

[0180] Specifically, the second data center starts a file update listener, monitors the temporary directory once at a preset time interval, and checks whether a new scheduled synchronization file is received. The monitoring process uses the find command to find new files to ensure the timeliness of file synchronization and update.

[0181] S1102: searching the total file table for a service ID corresponding to the scheduled synchronization file according to the file soft link path of the scheduled synchronization file;

[0182] Specifically, when the listener finds a new timed synchronization file, it uses the file soft link path of the file as the unique ID to search the business ID (BUSINESS_ID) associated with the timed synchronization file in the general file table FT_BASE_FILE to query the file object of the previous file version.

[0183] S1103: Querying the file object of the previous file version corresponding to the service ID in the general file table according to the service ID of the scheduled synchronization file;

[0184] Specifically, after the second data center searches the general file table FT_BASE_FILE for the business ID (BUSINESS_ID) associated with the scheduled synchronization file, it queries the general file table FT_BASE_FILE for the file object of the previous file version timestamp corresponding to the business ID based on the business ID of the scheduled synchronization file, ensuring that the incremental file and its corresponding business and version can be accurately identified to avoid erroneous file merging or updating.

[0185] S1104: If the query result is a query failure, determine the scheduled synchronization file as a formal file; if the query result is a query success, merge incremental files according to the scheduled synchronization file and the file object to generate a complete file, and verify whether the complete file is a formal file;

[0186] Specifically, the second data center searches the general file table FT_BASE_FILE for the file object of the previous file version timestamp corresponding to the business ID of the scheduled synchronization file according to the business ID of the scheduled synchronization file, and determines whether the scheduled synchronization file is a formal file put into production for the first time or an incremental file with a previous version according to the query result.

[0187] If the business ID is put into production for the first time, that is, the query result of querying the file object with the last file version timestamp corresponding to the business ID in the total file table is a query failure, the second data center directly determines the scheduled synchronization file as the official file. If the previous version of the file exists, that is, the query result of querying the file object with the last file version timestamp corresponding to the business ID in the total file table is a query success, the second data center will encapsulate the found file information into an incremental file assembly task and submit it to the assembly thread pool for processing, which can process multiple incremental file assembly tasks in parallel, improve processing efficiency, and ensure that the incremental files can be merged in time to generate the final business-usable files.

[0188] After receiving the incremental file assembly task, the assembly thread pool uses the file processing tool to perform incremental merging based on the incremental file (timed synchronization file) in the temporary directory and the file object of the previous version to generate a complete file. Through incremental merging, only the changed part of the file is processed, which effectively reduces resource consumption and improves the efficiency and accuracy of merging. After the merge is completed, the complete file is judged based on the file object to see whether it is a formal file.

[0189] Fig.12 A schematic diagram of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application is shown in FIG. Fig.12 As shown, verifying whether the complete document is a formal document includes:

[0190] S1201: Calculate the file MD5 of the complete file and search the file MD5 of the current version file in the total file table;

[0191] Specifically, after the merge is completed, the second data center calculates the MD5 value of the generated complete file, and compares it with the file MD5 of the current version file stored in the total file table FT_BASE_FILE obtained by searching according to the business ID.

[0192] S1202: If the file MD5 of the complete file is consistent with the file MD5 of the current version file, the complete file is determined as a formal file.

[0193] Specifically, if the two MD5 values ​​are equal, it means that the merged file is exactly the same as the same file in the original data center, and the complete file is verified to be the official file. Verifying the integrity and accuracy of the file can ensure that no errors or data corruption occurred during the file transmission and merging process.

[0194] S1105: Copy the official file to the official directory and write the file information of the official file into the received file table and the total file table.

[0195] Specifically, if the MD5 value comparison is successful, the assembly thread will copy the official file to the official directory. After the file is successfully copied to the official directory, the second data center will write the relevant data into the received file table and set the status to "2" (written to the official directory) to ensure the traceability of file processing and the integrity of records, which is convenient for subsequent monitoring and management.

[0196] In one embodiment, the application user "bye" can operate the official directory, and the NFTSA tool user "nftsa" has no right to operate, which can avoid misoperation and prevent the tool from misoperating files under the official directory, thereby enhancing security and reliability of file management.

[0197] In one embodiment, the cross-data center file disaster recovery synchronization method further includes:

[0198] At a preset time, the received file table is compared with the obtained list of files to be sent of the first data center to determine the file synchronization status; the fields of the received file table include: transmission task ID, file ID, transmission source, target data center, receiving status, creation time, update time and original computer room center.

[0199] Specifically, at a preset time within a preset time range, the files sent by the first data center and the files received by the second data center are compared to see if they are consistent. The second data center obtains the to-be-sent file table of the first data center, compares it with the local received file table, retrieves the files in the to-be-sent file table with a transmission status of successful transmission according to the date, retrieves the files written into the formal directory in the received file table, compares the total number of files sent and received within the preset time range to see if they are consistent, and compares the specific receiving and sending details of the files to see if they are completely consistent, wherein the specific receiving and sending details of the files include the file ID, file path and file MD5.

[0200] In one embodiment, the table structure of the received file table is shown in Table 3. By designing the table structure, a new standby computer room can be quickly expanded, further improving reliability and flexibility.

[0201] Table 3

[0202]

[0203]

[0204] In one embodiment, the preset time range may be a day, a week, or a month, but the present application is not limited thereto.

[0205] In one embodiment, the cross-data center file disaster recovery synchronization method further includes:

[0206] The real-time synchronization file transmitted by the first data center using the thread pool is received by using a file receiving tool; the real-time synchronization file includes a production release file and a business configuration file.

[0207] Specifically, dual-center file storage is a necessary prerequisite for disaster recovery switching, and a copy of the files in the primary data center needs to be synchronized to the backup data center as a backup. File synchronization uses a combination of real-time synchronization and scheduled synchronization. Files released on the production day and business configuration files are synchronized in real time to ensure that real-time meets disaster recovery requirements; files generated in batches at the end of the day and real-time synchronization files that fail to synchronize are fully transmitted on a scheduled basis until they succeed or the number of retransmissions is exceeded.

[0208] The second data center uses a file receiving tool to receive the production release file and the business configuration file transmitted by the first data center using the thread pool.

[0209] Fig.13 A schematic diagram of a cross-data center file disaster recovery synchronization method provided by an embodiment of the present application is shown in FIG. Fig.13 As shown, using the file receiving tool to receive the real-time synchronization file transmitted by the first data center using the thread pool includes:

[0210] S1301: Using the file receiving tool to monitor a preset port to obtain a file transfer request from the first data center;

[0211] Specifically, the file receiving tool of the second data center monitors the preset port through cluster deployment, waiting to receive the file transfer request from the first data center. When the file transfer tool of the first data center initiates the file transfer request, the file receiving tool receives the real-time synchronization file.

[0212] In one embodiment, the file receiving tool is an FSERVER server, and the default listening port is port 9090.

[0213] In one embodiment, the FSERVER server is based on a network communication framework (such as the MINA framework), relies on Java's NIO library (non-blocking I / O), simplifies the complexity of network programming, encapsulates the underlying details of Socket communication, and supports efficient file reception and processing.

[0214] S1302: When a file transfer request is obtained from the first data center, the real-time synchronization file is received.

[0215] Specifically, the file receiving tool receives real-time synchronization files and stores them in the second data center to ensure that all files are successfully stored, complete data backup, and provide support for disaster recovery needs.

[0216] In one embodiment, the health check of the source station of the primary data center (first data center) is performed through the HTTP layer. Under the NAS directory of the source station of the primary data center, a small icon with a first preset name is arranged, and the icon is accessed once every preset time interval to determine whether the source station of the primary data center is healthy; if the icon can be accessed normally, the source station of the primary data center is considered to be healthy. Similarly, the health check of the source station of the backup data center (second data center) is performed through the HTTP layer. Under the official directory of the source station of the backup data center, a small icon with a first preset name is arranged, and the icon is accessed once every preset time interval to determine whether the source station of the backup data center is healthy; if the icon can be accessed normally, the source station of the backup data center is considered to be healthy.

[0217] When the primary data center fails or a disaster recovery drill is required, the disaster recovery control system will issue a command to switch the source station, indirectly calling the Shell switching script of the primary data center source station to rename the small icon to the second first preset name. At this time, the back-to-source detection request will access the original first preset name icon. If a 404 error is returned and continues for more than a certain number of times, the CDN will automatically switch the detection request to the backup data center source station. Since the first preset name icon of the backup data center source station is normally accessible, subsequent CDN node back-to-source requests will be smoothly switched to the backup data center source station.

[0218] The present application provides a cross-data center file disaster recovery synchronization method, which obtains the received scheduled synchronization file transmitted by the first data center by scanning the temporary directory at every third preset time interval; searches the business ID corresponding to the scheduled synchronization file in the total file table according to the file soft link path of the scheduled synchronization file; searches the total file table for the file object of the previous file version corresponding to the business ID according to the business ID of the scheduled synchronization file; if the query result is a query failure, determines the scheduled synchronization file as a formal file; if the query result is a query success, performs incremental file merging according to the scheduled synchronization file and the file object to generate a complete file, and verifies whether the complete file is a formal file; copies the formal file to the formal directory and writes the file information of the formal file into the received file table and the total file table, thereby realizing the real-time and accuracy of file resource synchronization, meeting the requirements of disaster recovery deployment, and ensuring high availability in the face of failures.

[0219] Among them, the strategy of combining high-real-time file synchronization with full-file synchronization at the end of the day is adopted to achieve effective collection and synchronization of changed files, ensure the timeliness and integrity of file synchronization, and ensure the business needs of real-time files; through temporary directory transfer, it is possible to prevent large quantities of files from being mistransmitted and overwritten; through incremental cutting of files, it is possible to significantly reduce the consumption of intranet bandwidth; through the design of the file table structure, reliability and flexibility are further improved; through single domain name calling, the implementation cost is reduced and the calling logic is simplified.

[0220] The present application also provides a cross-data center file disaster recovery synchronization device in the embodiment, as described in the following embodiment. Since the principle of the device to solve the problem is similar to the cross-data center file disaster recovery synchronization method, the implementation of the device can refer to the implementation of the cross-data center file disaster recovery synchronization method, and the repeated parts will not be repeated.

[0221] Fig.14 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application is shown in FIG. Fig.14 As shown, the device comprises:

[0222] The second file acquisition unit 1401 is used to scan the temporary directory at every third preset time interval to acquire the received timed synchronization file transmitted by the first data center;

[0223] The second file search unit 1402 is used to search the service ID corresponding to the timing synchronization file in the general file table according to the file soft link path of the timing synchronization file;

[0224] The second file query unit 1403 is used to query the file object of the previous file version corresponding to the service ID in the general file table according to the service ID of the scheduled synchronization file;

[0225] The second file determination unit 1404 is used to determine the scheduled synchronization file as a formal file if the query result is a query failure; if the query result is a query success, perform incremental file merging according to the scheduled synchronization file and the file object to generate a complete file, and verify whether the complete file is a formal file;

[0226] The file copying unit 1405 is used to copy the formal file to the formal directory and write the file information of the formal file into the received file table and the total file table.

[0227] Fig.15 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application, Fig.14 Based on the embodiment, further, Fig.15 As shown, the cross-data center file disaster recovery synchronization device provided by the present application also includes:

[0228] A file MD5 calculation module 1501, used to calculate the file MD5 of the complete file and search the file MD5 of the current version file in the total file table;

[0229] The file MD5 comparison module 1502 is used to determine the complete file as a formal file if the file MD5 of the complete file is consistent with the file MD5 of the current version file.

[0230] Fig.16 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application, Fig.14 Based on the embodiment, further, Fig.16 As shown, the cross-data center file disaster recovery synchronization device provided by the present application also includes:

[0231] The file synchronization determination unit 1601 is used to compare the received file table with the acquired list of files to be sent of the first data center at a preset time to determine the file synchronization status; the fields of the received file table include: transmission task ID, file ID, transmission source, target data center, receiving status, creation time, update time and original computer room center.

[0232] Fig.17 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application, Fig.14 Based on the embodiment, further, Fig.17As shown, the cross-data center file disaster recovery synchronization device provided by the present application also includes:

[0233] The file receiving unit 1701 is used to use a file receiving tool to receive a real-time synchronization file transmitted by the first data center using a thread pool; the real-time synchronization file includes a production release file and a business configuration file.

[0234] Fig.18 A schematic diagram of the structure of a cross-data center file disaster recovery synchronization device provided by an embodiment of the present application, Fig.17 Based on the embodiment, further, Fig.18 As shown, the cross-data center file disaster recovery synchronization device provided by the present application also includes:

[0235] A request monitoring module 1801 is used to monitor a preset port using the file receiving tool to obtain a file transfer request from the first data center;

[0236] The file receiving module 1802 is used to receive the real-time synchronization file when obtaining a file transmission request from the first data center.

[0237] The present application provides a cross-data center file disaster recovery synchronization method and device, which obtains the received scheduled synchronization file transmitted by the first data center by scanning the temporary directory at every third preset time interval; searches the business ID corresponding to the scheduled synchronization file in the total file table according to the file soft link path of the scheduled synchronization file; searches the total file table for the file object of the previous file version corresponding to the business ID according to the business ID of the scheduled synchronization file; if the query result is a query failure, determines the scheduled synchronization file as a formal file; if the query result is a query success, performs incremental file merging according to the scheduled synchronization file and the file object to generate a complete file, and verifies whether the complete file is a formal file; copies the formal file to the formal directory and writes the file information of the formal file into the received file table and the total file table, thereby realizing the real-time and accuracy of file resource synchronization, meeting the requirements of disaster recovery deployment, and ensuring high availability in the face of failures.

[0238] Among them, the strategy of combining high-real-time file synchronization with full-file synchronization at the end of the day is adopted to achieve effective collection and synchronization of changed files, ensure the timeliness and integrity of file synchronization, and ensure the business needs of real-time files; through temporary directory transfer, it is possible to prevent large quantities of files from being mistransmitted and overwritten; through incremental cutting of files, it is possible to significantly reduce the consumption of intranet bandwidth; through the design of the file table structure, reliability and flexibility are further improved; through single domain name calling, the implementation cost is reduced and the calling logic is simplified.

[0239] It should be noted that the cross-data center file disaster recovery synchronization method provided in the embodiment of the present application can be used in the financial field, and can also be used in any technical field other than the financial field. The embodiment of the present application does not limit the application field of the cross-data center file disaster recovery synchronization method.

[0240] Fig.19 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Fig.19 As shown, the electronic device includes: a processor (processor) 1901, a memory (memory) 1902 and a bus 1903.

[0241] The processor 1901 and the memory 1902 communicate with each other via a bus 1903 .

[0242] The processor 1901 is used to call the program instructions in the memory 1902 to execute the methods provided by the above-mentioned method embodiments.

[0243] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned cross-data center file disaster recovery synchronization method is implemented.

[0244] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned cross-data center file disaster recovery synchronization method is implemented.

[0245] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0246] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0247] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0248] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0249] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A cross-data center file disaster recovery synchronization method, applied to a first data center, characterized in that: include: Scanning the network storage directory at a first preset time interval to obtain newly generated files; The newly generated files include end-of-day batch files and real-time synchronization files that failed to be transmitted; Search the file MD5 of the newly generated file in the file list to be sent; If the search fails, query the file object of the previous file version corresponding to the business ID in the total file table according to the business ID of the newly generated file; If the query result is a query failure, the newly generated file is determined as a scheduled synchronization file; If the query result is successful, incremental file segmentation is performed according to the newly generated file and the file object to generate a timed synchronization file; The timed synchronization file is transmitted to a temporary directory of the second data center.

2. The method according to claim 1, characterized in that The step of performing incremental file segmentation according to the newly generated file and the file object to generate a timed synchronization file includes: Splitting the newly generated file and the file object to obtain a plurality of file blocks respectively; The file blocks of the file object are used to perform sliding difference detection on the file blocks of the newly generated file to generate a timed synchronization file.

3. The method according to claim 1, characterized in that The transmitting the timed synchronization file to a temporary directory of the second data center includes: Write the file information of the scheduled synchronization file into the to-be-sent file table and the total file table; the fields of the to-be-sent file table include: transmission task ID, file ID, transmission source, target data center, transmission status, number of transmissions, creation time, update time and original computer room center; the fields of the total file table include: file ID, service ID, file path, file soft link path, file MD5, file version, whether the file is CDN deployed, creation time, creator, update time, updater, resource type and file size; The to-be-sent file list is scanned at every second preset time interval, and the scheduled synchronization files with a transmission status of not transmitted or failed to be transmitted are transmitted to a temporary directory of the second data center.

4. The method according to claim 1, characterized in that Also includes: When a real-time synchronization file is generated, the real-time synchronization file is transmitted to the second data center by using a thread pool; The real-time synchronization files include production release files and business configuration files.

5. The method according to claim 4, characterized in that The using a thread pool to transmit the real-time synchronization file to the second data center includes: Encapsulating the task of transmitting the real-time synchronization file as a thread pool task; The thread pool task is submitted to the thread pool, so as to utilize a file transfer tool to transfer the real-time synchronization file to the second data center.

6. A cross-data center file disaster recovery synchronization method, applied to a second data center, characterized in that: include: Scan the temporary directory at every third preset time interval to obtain the received timed synchronization file transmitted by the first data center; Searching the service ID corresponding to the scheduled synchronization file in the general file table according to the file soft link path of the scheduled synchronization file; According to the service ID of the scheduled synchronization file, query the file object of the previous file version corresponding to the service ID in the general file table; If the query result is a query failure, the scheduled synchronization file is determined as a formal file; If the query result is successful, perform incremental file merging according to the scheduled synchronization file and the file object to generate a complete file, and verify whether the complete file is a formal file; The formal file is copied to the formal directory and the file information of the formal file is written into the received file table and the total file table.

7. The method according to claim 6, characterized in that The verifying whether the complete document is a formal document includes: The file MD5 of the complete file is calculated and the file MD5 of the current version file is searched in the total file table; If the file MD5 of the complete file is consistent with the file MD5 of the current version file, the complete file is determined as a formal file.

8. The method according to claim 6, characterized in that Also includes: Comparing the received file list with the acquired to-be-sent file list of the first data center at a preset time to determine the file synchronization status; The fields of the received file table include: transmission task ID, file ID, transmission source, target data center, receiving status, creation time, update time and original computer room center.

9. The method according to claim 6, characterized in that Also includes: Using a file receiving tool to receive the real-time synchronization file transmitted by the first data center using the thread pool; The real-time synchronization files include production release files and business configuration files.

10. The method according to claim 9, characterized in that The method of using a file receiving tool to receive a real-time synchronization file transmitted by the first data center using a thread pool includes: Using the file receiving tool to monitor a preset port to obtain a file transfer request from the first data center; When the file transfer request from the first data center is obtained, the real-time synchronization file is received.

11. A cross-data center file disaster recovery synchronization device, characterized in that: include: A first file acquisition unit, configured to scan the network storage directory at a first preset time interval to acquire a newly generated file; The newly generated files include end-of-day batch files and real-time synchronization files that failed to be transmitted; A first file search unit, configured to search the file MD5 of the newly generated file in the to-be-sent file table; A first file query unit, configured to query the file object of the previous file version corresponding to the service ID of the newly generated file in the general file table if the search fails; A first file determination unit, configured to determine the newly generated file as a scheduled synchronization file if the query result is a query failure; If the query result is successful, incremental file segmentation is performed according to the newly generated file and the file object to generate a timed synchronization file; The first file transmission unit is used to transmit the scheduled synchronization file to a temporary directory of the second data center.

12. A cross-data center file disaster recovery synchronization device, characterized in that: include: A second file acquisition unit, configured to scan the temporary directory at every third preset time interval to acquire the received timed synchronization file transmitted by the first data center; A second file search unit, configured to search the service ID corresponding to the timing synchronization file in the general file table according to the file soft link path of the timing synchronization file; A second file query unit, configured to query the total file table for a file object of a previous file version corresponding to the service ID of the timed synchronization file according to the service ID of the timed synchronization file; A second file determination unit, configured to determine the scheduled synchronization file as a formal file if the query result is a query failure; If the query result is successful, perform incremental file merging according to the scheduled synchronization file and the file object to generate a complete file, and verify whether the complete file is a formal file; The file copying unit is used to copy the formal file to the formal directory and write the file information of the formal file into the received file table and the total file table.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images