Batch file transmission method and device, storage medium and electronic equipment

Abstract

The invention discloses a batch file transmission method and device, a storage medium and electronic equipment, and relates to the field of distribution.The method comprises the steps that file service client platforms and batch scheduling platforms are deployed in multiple parks; when it is detected that the first park cannot provide services, a file transmission request related to a target application is transferred from the first park to a second park, a file service client platform and a batch scheduling platform of the second park are started, and batch file transmission and job scheduling are carried out with the second park as a main park; and during remote disaster recovery backup switching, transferring a file transmission request related to the target application from the first park and the second park to the third park, and switching the batch files to the third park. According to the method and the device, the technical problem of poor stability of a batch file transmission process caused by the fact that quick, automatic and effective disaster recovery park switching cannot be realized when batch file transmission faces a local park fault in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distribution, in particular, to a batch file transmission method and device, a storage medium and an electronic device. BACKGROUND

[0002] In the field of finance, e-commerce, Internet of Things and other distributed systems, in batch file transmission, the local and cross-site transmission of batch files is performed by the client IP node of the file server and the batch controller, but there are certain efficiency and manual operation switching problems.

[0003] The existing batch file transmission process sets the IP addresses of the sender and the receiver through the client of the file server, and the files are stored in the application file server in each subsystem. The file transmission is limited by the transmission mechanism, and the batch file cannot be quickly transmitted. The file synchronization between cross-sites consumes a long time. When a disaster occurs in the same city or different places, it cannot automatically and efficiently switch, affecting the rapid application emergency response of the system and the business continuity.

[0004] At present, there is no effective solution to the above problems. SUMMARY

[0005] The embodiments of the present application provide a batch file transmission method, device, storage medium and electronic device to at least solve the technical problem that the batch file transmission in the prior art cannot realize fast, automatic and effective disaster recovery park switching when facing local park failure, thereby causing poor stability of the batch file transmission process.

[0006] According to an aspect of an embodiment of the present application, a batch file transmission method is provided, comprising: deploying a file service client platform and a batch scheduling platform in multiple parks, wherein the multiple parks at least include a first park, a second park and a third park, the first park and the second park belong to the same region, the third park does not belong to the same region as the first park, and the file service client platform and the batch scheduling platform in each park realize local storage and cross-park synchronization of files through file sharing storage; when it is detected that the first park cannot provide services due to failure, modifying the domain name configuration information of the target application to divert the file transmission request related to the target application from the first park to the second park, and starting the file service client platform and the batch scheduling platform of the second park to perform batch file transmission and job scheduling in the second park as the main park; when it is necessary to switch to a different site for disaster recovery, modifying the domain name configuration information of the target application to divert the file transmission request related to the target application from the first park and the second park to the third park, and starting the file service client platform and the batch scheduling platform of the third park to switch the batch file to the third park by using the data synchronization mechanism of the different site storage.

[0007] Optionally, the batch file transfer method further includes: during the batch file transfer process with any one of the parks as the main park, when a file retransmission is detected, the target file uploaded by the file sender and its hash value are forwarded to the file receiver through the file service client platform of the main park. The file receiver determines whether to update the local file by comparing the hash value of the local file with the hash value uploaded by the file sender.

[0008] Optionally, the batch file transfer method further includes: during the batch file transfer process with any one of the parks as the main park, when the first retransmission of the target file is detected to have failed, the file service client platform of the main park moves the target file to the target directory and attempts a second retransmission according to the preset retransmission identifier and retransmission strategy. The retransmission strategy includes: a retransmission duration setting strategy, a retransmission number setting strategy, and a retransmission interval setting strategy.

[0009] Optionally, when the first campus is detected to be unable to provide service due to a fault, the file transfer requests related to the target application are redirected from the first campus to the second campus by modifying the domain name configuration information of the target application. Simultaneously, the file service client platform and batch scheduling platform of the second campus are activated, with the second campus acting as the primary campus for batch file transfer and job scheduling. This includes: modifying the domain name configuration information of the target application when the first campus is detected to be unable to provide service due to a fault, wherein the modification operation is used to point the domain name resolution of the target application to a service IP address within the second campus; activating the file service client platform of the second campus, wherein the activated file service client platform of the second campus takes over the file transfer tasks of the first campus; and activating the batch scheduling platform of the second campus, wherein the activated batch scheduling platform of the second campus takes over all ongoing and pending batch jobs.

[0010] Optionally, after starting the batch scheduling platform of the second park, the batch file transfer method further includes: remounting the same file-sharing storage system as the first park in the second park; implementing cluster master-slave switching between the first and second parks through the file-sharing storage system, and adjusting the file storage volume of the second park from the backup storage volume to the master storage volume. During the implementation of the cluster master-slave switching, file data writing operations are suspended until all data in transit is synchronized, and then the data read and write services of the file-sharing storage system in the second park are restarted.

[0011] Optionally, the batch file transfer method further includes: during the cluster master-slave switch between the first and second zones through the file-sharing storage system, maintaining data consistency between the file-sharing storage systems of the second and third zones through asynchronous or synchronous data transfer.

[0012] Optionally, when a disaster recovery switchover is required, the domain name configuration information of the target application is modified to redirect file transfer requests related to the target application from the first and second zones to the third zone. Simultaneously, the file service client platform and batch scheduling platform of the third zone are activated. Utilizing the data synchronization mechanism of the off-site storage, batch files are switched to the third zone. This includes: adjusting the domain name configuration information of the target application when a disaster recovery switchover is required, wherein the adjustment operation is used to direct access of the target application to the third zone; activating the file service client platform of the third zone, wherein the activated file service client platform of the third zone takes over file transfer tasks from the first and second zones; and activating the batch job scheduling platform of the third zone, wherein the activated batch job scheduling platform of the third zone takes over all ongoing and pending batch jobs.

[0013] Optionally, during the batch file transfer process using any one of the parks as the main park, when a file retransmission is detected, the file to be transferred uploaded by the file sender and its hash value are forwarded to the file receiver through the file service client platform of the main park. This includes: moving the target file uploaded by the file sender from the sending directory to the target directory through the file service client platform of the main park; creating a marker file with the same name as the target file in the target directory through the file service client platform of the main park; after detecting that the file sender reads the target file from the target directory, notifying the file sender to calculate the hash value for the target file; and sending the target file and the hash value calculated by the file sender for the target file to the file receiver through the file service client platform of the main park.

[0014] Optionally, after the target file and the hash value calculated by the file sender for the target file are sent to the file receiver through the file service client platform of the main campus, the file receiver performs the following steps: saves the received file to a temporary directory and calculates the hash value of the received file; checks whether the hash value calculated by the file receiver for the received file is consistent with the hash value provided by the file sender; if the hash value calculated by the file receiver for the received file is consistent with the hash value provided by the file sender, the received file is moved from the temporary directory to the official directory and a successful reception signal is sent to the file sender; if the hash value calculated by the file receiver for the received file is inconsistent with the hash value provided by the file sender, the file transfer is deemed to have failed.

[0015] Optionally, the batch file transfer method further includes: after the file sender receives a signal indicating that the target file failed to be received, moving the target file from the target directory to the retransmission failure directory; if a one-time retransmission strategy is configured, the file sender moves the files in the retransmission failure directory back to the sending directory and starts the one-time retransmission process; if the receiver still reports failure after the one-time retransmission process, and a two-time retransmission strategy is configured, the file sender moves the failed files back to the sending directory again for a second retransmission. During the second retransmission, the file sender repeats the file transfer and hash verification process according to the preset number of retransmissions and retransmission interval; during the second retransmission, the file receiver continues to perform file reception, hash value calculation, and hash value consistency comparison.

[0016] According to another aspect of the embodiments of this application, a batch file transfer device is also provided, comprising: a platform deployment unit, configured to deploy a file service client platform and a batch scheduling platform in multiple zones, wherein the multiple zones include at least a first zone, a second zone, and a third zone, the first zone and the second zone belong to the same region, and the third zone does not belong to the same region as the first zone, wherein the file service client platform and the batch scheduling platform in each zone achieve local storage and cross-zone synchronization of files through file sharing storage; a first processing unit, configured to, when detecting that the first zone cannot provide services due to a fault, redirect file transfer requests related to the target application from the first zone to the second zone by modifying the domain name configuration information of the target application, and simultaneously start the file service client platform and the batch scheduling platform in the second zone, using the second zone as the main zone for batch file transfer and job scheduling; and a second processing unit, configured to, when a remote disaster recovery switch is required, redirect file transfer requests related to the target application from the first zone and the second zone to the third zone by modifying the domain name configuration information of the target application, and simultaneously start the file service client platform and the batch scheduling platform in the third zone, using a remote storage data synchronization mechanism to switch the batch files to the third zone.

[0017] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described batch file transfer method.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described batch file transfer method.

[0019] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the above-described batch file transfer method.

[0020] In this application, the file transfer system first deploys a file service client platform and a batch scheduling platform across multiple zones. These zones include at least a first zone, a second zone, and a third zone. The first and second zones belong to the same region, while the third zone does not. Within each zone, the file service client platform and batch scheduling platform achieve local file storage and cross-zone synchronization through shared file storage. When the first zone is detected as unavailable due to a fault, the system modifies the domain name configuration information of the target application to redirect file transfer requests related to the target application from the first zone to the second zone. Simultaneously, the file service client platform and batch scheduling platform in the second zone are activated, making the second zone the primary zone for batch file transfer and job scheduling. When a disaster recovery switchover is required, the system modifies the domain name configuration information of the target application to redirect file transfer requests related to the target application from the first and second zones to the third zone. Simultaneously, the file service client platform and batch scheduling platform in the third zone are activated, and the batch files are switched to the third zone using a remote storage data synchronization mechanism.

[0021] As described above, this application deploys file service client platforms and batch scheduling platforms across multiple zones, including the first, second, and third zones. It utilizes shared file storage to achieve local storage and cross-zone synchronization. When the first zone fails to provide service, the file transfer system automatically detects this and quickly switches the file transfer request from the first zone to the second zone by modifying the domain name configuration information of the target application. Simultaneously, the file transfer system automatically starts the file service client platform and batch scheduling platform in the second zone, using the second zone as the primary zone to continue batch file transfer and job scheduling. This not only achieves automated failover but also ensures the continuity and stability of file transfer tasks, avoiding business interruptions caused by a single zone failure. Furthermore, when a disaster recovery switch is required, the file transfer system similarly switches the file transfer request from the first and second zones to the third zone by modifying the domain name configuration information and starts the relevant platforms in the third zone. By utilizing the off-site storage data synchronization mechanism, the file transfer system can quickly switch batch files to a third-party campus, ensuring that file transfer tasks can still be smoothly connected and continue to execute in cross-regional failure scenarios. The multi-layered switching mechanism not only improves the system's fault tolerance and reliability but also optimizes resource utilization efficiency, ensuring the high efficiency and stability of batch file transfer under various failure scenarios. This enhances the overall system's resilience and business continuity, thereby solving the technical problem of poor stability in batch file transfer processes caused by the inability to achieve fast, automatic, and effective disaster recovery campus switching when facing local campus failures in existing technologies. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a schematic diagram of an optional batch file transfer method according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of an optional batch file transfer architecture according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an optional same-city park-level handover according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of an optional remote campus switching according to an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of an optional retransmission mechanism according to an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of an optional retransmission file verification according to an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of an optional bulk file transfer device according to an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] According to an embodiment of this application, a method embodiment for batch file transfer is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0033] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0034] According to the embodiments of this application, a file transfer system (hereinafter referred to as the system) can be used as the execution subject of the batch file transfer method of this application embodiment. The system can be a software system or an embedded system combining software and hardware. Of course, the method execution subject in the embodiments of this application can also be other forms of execution subject, such as devices, equipment, etc. It should be known by those skilled in the art that this application does not particularly limit the specific form of the method execution subject.

[0035] First, the relevant technical terms in this application will be explained:

[0036] Local file storage is used to characterize devices that mount disks via direct connection or storage area networks, where stored files cannot be shared between devices, such as when using centralized storage disk drives.

[0037] File sharing storage is used to characterize devices that mount a file system based on a file network sharing protocol, allowing stored files to be shared between devices, such as distributed file storage and shared file storage.

[0038] A hot standby node is used to indicate that a node is running services, but there is no access traffic. The essential difference between a hot standby node and a dual-active / multi-active node is whether it carries business operations.

[0039] A cold standby node is used to indicate that the system is ready for operation, but the service is not running. The essential difference between a cold standby node and a hot standby node is whether the service is continuously online.

[0040] The off-site disaster recovery application / business-level drill is used to demonstrate that the production and disaster recovery environments are running simultaneously during the drill. Only a portion of the participating terminals and clients simulate transactions being transferred to the disaster recovery environment. The drill transaction data is recorded in the disaster recovery database. The drill process does not affect the operation of the production environment, and the data is cleared after the drill without being restored. The traffic of the participating applications during the drill is closed-loop across the off-site locations.

[0041] The off-site disaster recovery drill is used to characterize the off-site disaster recovery takeover of some applications within the implementation plan. It includes the switching, back-to-back and data recovery between the production and disaster recovery environments. It involves mutual access between the production and disaster recovery of upstream and downstream applications, which can verify the real business carrying capacity. The mutual access traffic of the participating applications during the drill is closed-loop in the off-site environment.

[0042] Figure 1 This is a batch file transfer method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0043] Step S101: Deploy file service client platforms and batch scheduling platforms in multiple parks. The multiple parks include at least a first park, a second park, and a third park. The first and second parks belong to the same region, while the third park does not belong to the same region as the first park. The file service client platform and batch scheduling platform in each park realize local storage and cross-park synchronization of files through file sharing storage.

[0044] Optionally, since the first and second campuses belong to the same region, they can be geographically closer and have better network connectivity, enabling efficient local storage and cross-campus synchronization of shared files. This deployment method can fully utilize the network advantages within the region, reduce data transmission latency, improve file access speed, and thus enhance overall work efficiency within the region.

[0045] Optionally, the third campus may not belong to the same region as the first campus, indicating that there can be significant differences in geographical and network environments between the third and first campuses, as well as between the third and second campuses. However, through file-sharing storage, the third campus can still achieve file synchronization with the first and second campuses. This cross-regional file synchronization function facilitates business collaboration among enterprises or organizations in different geographical areas, ensuring file consistency and real-time updates between different campuses, so as to quickly achieve file synchronization when switching between campuses.

[0046] Figure 2 A schematic diagram of a batch file transfer architecture is shown. The first zone can be represented by zone A, the second zone by zone B, and the third zone by zone C.

[0047] In Park A, upstream application A initiates file transfer tasks, communicating with target application B through a file service client (master node). The file service client (master node) handles the specific operations of file transfer, collaborating with the batch scheduling platform. The batch scheduling platform (master node) manages and schedules file transfer tasks, ensuring they execute as planned. Target application B receives files from upstream application A and processes them through the file service client (master node). Downstream application C receives files from target application B, completing the final stage of file transfer. The file service client platform provides the interface and services for file transfer, supporting task configuration and monitoring management. The batch scheduling platform is responsible for scheduling batch jobs and collaborates with the file service client platform.

[0048] In Campus B, a backup node for upstream application A can be deployed for disaster recovery. Target application B represents the same application as in Campus A, used for file reception and processing. Batch scheduling (master node) is responsible for managing and scheduling file transfer tasks in Campus B. File service client (cold backup) represents a backup node for the file service client deployed in Campus B.

[0049] In disaster recovery at Park C, upstream application A can be deployed at Park C for off-site disaster recovery. Target application B can also be deployed at Park C for off-site disaster recovery. Batch scheduling (cold backup) represents the backup node of the batch scheduling platform deployed at Park C. File service client (cold backup) represents the backup node of the file service client deployed at Park C. The file service client platform is the platform providing file transfer services at Park C. The batch scheduling platform is the platform responsible for batch job scheduling at Park C.

[0050] Optionally, the file storage system can employ file-sharing storage methods such as NAS (Network Attached Storage) to achieve data synchronization within the same city and across different locations, ensuring high availability and consistency of data. Same-city data synchronization uses a strong synchronization replication mechanism, achieving zero data loss and ensuring the real-time nature and integrity of data within the same city. Cross-regional data synchronization, on the other hand, uses an asynchronous replication mechanism, with synchronization time controlled within 10 to 20 minutes, optimizing the efficiency of cross-regional data synchronization while ensuring eventual data consistency. NAS refers to a device or system that provides data storage and file-sharing services through a network connection. When multiple NAS devices or nodes work together, they form a NAS cluster. The NAS clusters used for batch processing and online processing are completely independent. Batch-related NAS clusters are isolated according to business operation and maintenance modules, supporting individual switching based on partition fault domains, thereby improving system flexibility and fault tolerance.

[0051] Figure 2 In this configuration, the file-shared storage (NAS) includes NAS-1 (Application B) and NAS-2 (Application AC). NAS-1 provides file storage services for Application B, while NAS-2 provides file storage services for Applications A and C. The primary storage volume stores active data and supports strong synchronization. The backup storage volume is used for data backup and supports asynchronous replication.

[0052] Figure 2 In the diagram, solid arrows indicate the flow of received file data, representing the transmission path from sender to receiver. Dashed arrows indicate the flow of sent file data, representing the path from receiver back to sender. Dotted arrows indicate the flow of control and scheduling data, representing the flow of control and management information between the batch scheduling platform and the file service client platform.

[0053] For example, upstream application A initiates a file transfer task through a file service client (master node). The batch scheduling platform (master node) receives the task request and performs task scheduling. The file service client (master node) transfers the file from upstream application A to target application B. Target application B receives the file through the file service client (master node) and stores it on the NAS. Downstream application C receives the file from target application B, forwards the file, and completes the transfer. In the event of a failure in campus A, the file transfer request can be switched to campus B or C by modifying the domain name configuration information, thus achieving disaster recovery. This architecture ensures high availability and disaster recovery capabilities for file transfer. By deploying file service client platforms and batch scheduling platforms across multiple campuses and using a file-sharing storage system, it achieves rapid file synchronization and transfer, ensuring business continuity and stability.

[0054] In the deployment of file transfer tasks, the production environment and disaster recovery environment are explicitly isolated to ensure rapid switchover and maintain file transfer stability in the event of a failure. Taking the transfer from upstream application A to target application B as an example, the domain name configuration for intra-city transfer tasks is a.com.cn to b.com.cn, while the domain name configuration for inter-regional transfer tasks is a.dr.com.cn to b.dr.com.cn. These two tasks can run independently, with the initiation and closure of batch files controlled by the file service client platform. Under normal circumstances, file transfer tasks run in the intra-city environment, such as the production environment. However, when a complete switch to the disaster recovery zone is required, the inter-regional file service client platform can take over the batch file transfer tasks, completely independent of the production environment's A and B zones. Thus, by clearly distinguishing between production and disaster recovery tasks, efficiency and reliability during failover can be ensured.

[0055] File transfer nodes represent nodes where file transfer clients are deployed, and can be deployed as virtual machines. File transfer nodes can be deployed according to strategies of local cold backup and remote cold backup. Both upstream and downstream applications can access them via domain names, thus reducing the coupling of file transfer paths between applications. Two transfer tasks are created in the production and disaster recovery environments respectively. The disaster recovery task can be a shadow task of the production task, differing only in the domain names of the source and destination addresses. This allows for quick migration of tasks by modifying the domain name during a switchover, without requiring complex IP binding changes.

[0056] Executor nodes represent nodes that deploy batch jobs and are responsible for receiving job scheduling instructions from the batch controller to execute batch jobs. Executor nodes can be deployed according to strategies of active-active deployment within the same city and disaster recovery deployment in different locations. Regardless of whether the system switches to a local or remote environment, it can start independently after the switch and control the execution of batch jobs in the current region. This not only improves the system's flexibility and fault tolerance but also ensures efficient operation in different environments, thereby guaranteeing the continuity and stability of file transfer tasks.

[0057] In a failure scenario, the NAS cluster failover process includes: First, stopping services in one of the failure domains to simulate a failure scenario; then, the NAS performs a remote failover, waiting for data synchronization to complete before starting read / write services in the disaster recovery environment; next, the application layer starts containers in the disaster recovery environment, with virtual machines mounting disaster recovery NAS storage volumes to complete the application's remote failover; once the failure is resolved, the NAS performs reverse data synchronization, bringing data from the disaster recovery environment back to the production cluster; subsequently, the NAS starts the local cluster, waiting for data synchronization to complete before starting read / write services in the production environment; finally, the application layer starts containers in the production environment, with virtual machines resuming operation, completing the system's rollback. This process ensures that the file transfer system can quickly switch to the disaster recovery environment during a failure and smoothly switch back to the production environment after recovery, guaranteeing business continuity and data integrity.

[0058] Step S102: When it is detected that the first park cannot provide services due to a fault, the file transfer requests related to the target application are redirected from the first park to the second park by modifying the domain name configuration information of the target application. At the same time, the file service client platform and batch scheduling platform of the second park are started, and the second park is used as the main park for batch file transfer and job scheduling.

[0059] Optionally, during the operation of the file transfer system, if it detects that the first campus is unable to provide service due to a failure, the file transfer system can achieve failover by modifying the domain name configuration information of the target application. For example, modifying the domain name configuration information can redirect file transfer requests originally directed to the first campus to the second campus. This operation ensures that file transfer requests can be smoothly switched to the second campus, thereby avoiding service interruption caused by the failure of the first campus. Failover not only improves the availability and reliability of the system but also ensures business continuity, allowing users to continue file transfer operations normally even when the first campus fails.

[0060] Optionally, the file transfer system can activate the file service client platform and batch scheduling platform of the second campus, making the second campus the primary campus for batch file transfer and job scheduling. That is, the second campus will take over the file processing tasks originally handled by the first campus, including file reception, storage, synchronization, and batch job scheduling. In this way, the second campus can quickly respond to and process file transfer requests, ensuring the normal operation of the file service. This embodiment not only demonstrates the flexibility and fault tolerance of the file transfer system but also showcases the ability for collaboration between campuses, thereby ensuring the efficient operation of the entire system in case of failure.

[0061] Figure 3 This diagram illustrates a city-wide campus-level failover. City-wide campus-level failover allows for a complete switchover to another city-wide campus when one campus becomes unavailable, such as switching from Campus A to Campus B, ensuring business continuity. During this switchover, Campus B will act as the new primary campus, independently running the file service client platform and batch scheduling platform. The file shared storage system will quickly switch primary storage to support the operation of the primary node in Campus B. The primary storage in Campus B maintains off-site synchronization with the disaster recovery system in Campus C, ensuring data protection and rapid recovery in the event of a larger-scale disaster, guaranteeing data security and recoverability. When Campus A fails, file transfer requests and batch job scheduling tasks will switch from Campus A to Campus B through modifications to the domain name configuration information. The primary node in Campus B will take over file transfer requests and batch job scheduling tasks, continuing to receive, send, and process files. Meanwhile, NAS-1 and NAS-2, as shared file storage, will switch the primary storage volume from campus A to campus B and maintain remote synchronization with campus C to achieve high data availability and disaster recovery capabilities. This not only improves the system's fault tolerance but also optimizes resource utilization efficiency, ensuring that the file transfer system can quickly recover and maintain continuous business operation in the event of a failure in a campus within the same city.

[0062] Step S103: When a remote disaster recovery switch is required, the domain name configuration information of the target application is modified to redirect file transfer requests related to the target application from the first and second parks to the third park. At the same time, the file service client platform and batch scheduling platform of the third park are started, and the batch files are switched to the third park using the data synchronization mechanism of the remote storage.

[0063] Optionally, when a disaster recovery switchover is required, the file transfer system can modify the domain name configuration information of the target application to redirect file transfer requests originally directed to the first and second zones to the third zone. By adjusting the domain name configuration, file transfer requests can be quickly switched from the affected zones to the third zone, which is located in a different region, ensuring the continuity and stability of file services. Since the third zone is not located in the same region as the first and second zones and has its own independent infrastructure and network environment, it can effectively avoid service interruptions caused by regional failures such as natural disasters and network attacks in disaster recovery scenarios, thereby ensuring the high availability of the entire system.

[0064] Optionally, the file transfer system can activate the file service client platform and batch scheduling platform in the third zone. Utilizing the off-site storage data synchronization mechanism, batch files are switched to the third zone. The third zone then takes over the core functions of the file service, including file reception, storage, synchronization, and batch job scheduling. Through the off-site storage data synchronization mechanism, the third zone ensures the integrity and consistency of file data. Even in the event of failures in the first or second zones, it can smoothly take over file service tasks and continue to provide users with stable and reliable file transfer and processing services. This embodiment not only demonstrates the disaster recovery capability of the file transfer system but also showcases the efficiency and reliability of cross-regional collaborative work.

[0065] In an optional embodiment, the batch file transfer method further includes: when the file transfer system detects a file retransmission during the batch file transfer process with any one of the parks as the main park, it forwards the target file and hash value uploaded by the file sender to the file receiver through the file service client platform of the main park. The file receiver determines whether to update the local file by comparing the hash value of the local file with the hash value uploaded by the file sender.

[0066] Optionally, during batch file transfers in the file transfer system, if any one of the first, second, or third zones is designated as the primary zone, the file service client platform in the primary zone will initiate a corresponding processing mechanism when a file retransmission is detected. For example, the file service client platform can forward the target file uploaded by the file sender and its hash value to the file receiver. A hash value is a digital fingerprint used to verify file integrity and consistency. By calculating the file's hash value, it's possible to quickly determine whether the file has been tampered with or corrupted during transmission. This mechanism ensures the reliability and accuracy of file transfers, effectively preventing business interruptions due to file corruption even in complex network environments or high-concurrency transmission scenarios.

[0067] Optionally, after receiving the forwarded target file and its hash value, the file receiver can determine whether to update the local file by comparing its hash value with the hash value uploaded by the file sender. If the hash value of the local file matches the sender's hash value, it indicates that the local file is complete and has not been tampered with, and no update operation is required. Conversely, if the hash values ​​do not match, it indicates that the local file may have been corrupted or partially lost during transmission. In this case, the file receiver will update the file based on the complete file provided by the file sender to ensure the accuracy and integrity of the local file. The hash-based verification mechanism of this application not only improves the reliability of file transmission but also optimizes the efficiency of file updates, avoiding unnecessary data transmission and storage operations.

[0068] Figure 5 A schematic diagram of a retransmission mechanism is shown. During batch file transfer, sender A first attempts to transfer the file to receivers B and C. If the first transfer fails, the file will be retransmitted according to a preset retransmission strategy. In this process, sender A uses a file service client platform to send the file and its calculated hash value to the receivers. The receivers then compare the hash value of their local file with the hash value provided by the sender to determine whether the file needs to be updated, thereby ensuring the integrity and consistency of the file.

[0069] If the first retransmission fails, the file will be moved to the send.fail directory, and a second retransmission will be performed according to the retransmission policy. The retransmission policy includes settings for retransmission duration, number of attempts, and interval. These parameters ensure that the system can automatically attempt retransmissions in case of poor network conditions or temporary failures, thereby improving the success rate. During the second retransmission, sender A attempts to send the file to receiver C again through the file service client platform until the file is successfully transmitted or the preset retransmission limit is reached. This mechanism not only improves the reliability of file transmission but also ensures the accuracy of file transmission through hash comparison, thus achieving the effective transmission of the latest file.

[0070] For example, in a normal batch file transfer process, sender A moves the file test.bin from the send directory to the send.etransdtmp directory, preparing for transfer. Simultaneously, receiver B creates an empty test.bin file in the recv.etranrevtmp directory, ready for receiving. Sender A reads the contents of the test.bin file from the send.etransdtmp directory and writes it to the test.bin file in receiver B's recv.etranrevtmp directory. Once receiver B has finished writing the file, it moves the file from the recv.etranrevtmp directory to the actual recv directory and sends a successful reception response to sender A. After all receivers have successfully acknowledged the transfer, sender A moves the file from the send.etransdtmp directory to the send.succ directory, indicating that the transfer task is complete.

[0071] In an optional embodiment, the batch file transfer method further includes: when the file transfer system is transferring batch files with any one of the parks as the main park, if the first retransmission of the target file fails, the file service client platform of the main park moves the target file to the target directory and attempts a second retransmission according to the preset retransmission identifier and retransmission strategy. The retransmission strategy includes: a retransmission duration setting strategy, a retransmission number setting strategy, and a retransmission interval setting strategy.

[0072] This application's embodiments demonstrate the importance the file transfer system places on file transfer reliability. Through a preset retransmission mechanism, the system can effectively handle temporary failures or network problems that may occur during file transfer. The retransmission strategy ensures that file transfer is not interrupted by a single failure, but rather improves the success rate through multiple attempts, thereby guaranteeing the stability and continuity of file services.

[0073] For example, if a receiver (B) fails to receive the file due to timeout or other reasons during transmission, the sender (A) will initiate a Layer 1 retransmission mechanism according to preset parameters. The retransmission duration is controlled by the number of retries and the retry interval. The system will attempt to retransmit the file to the failed receiver until it successfully receives the file. If the retransmission still fails after a timeout, the sender (A) will move the test.bin file from the send.etransdtmp directory to the send.fail directory, marking it as a retransmission failure. At this point, if the system is not configured with task-level Layer 2 retransmission, the test.bin file will remain in the send.fail directory awaiting further processing. If task-level Layer 2 retransmission is configured, the sender (A) will move the test.bin file from the send.fail directory back to the send directory and restart a complete transmission process, attempting to transmit the file to all downstream nodes again to ensure that the file is eventually successfully transmitted. Through multi-layered retransmission mechanisms, the file transfer system can effectively cope with temporary failures that may occur during transmission, improving the reliability and stability of file transfer.

[0074] Optionally, the retransmission strategy includes three key aspects: the retransmission duration setting strategy, the retransmission count setting strategy, and the retransmission interval setting strategy. The retransmission duration setting strategy determines the duration of each retransmission attempt, ensuring file transfer is completed within a limited time. The retransmission count setting strategy specifies the maximum number of retransmission attempts the system can attempt after a file transfer failure, avoiding resource waste caused by infinite retransmission loops. The retransmission interval setting strategy controls the time interval between two retransmission attempts, preventing excessive pressure on network resources due to overly frequent retransmission attempts. By configuring the retransmission strategy appropriately, the file transfer system can optimize resource utilization efficiency while ensuring file transfer success rate, ensuring the efficiency and reliability of the entire file transfer process.

[0075] For example, during file transfer retransmission, the file transfer system can provide a flexible retransmission file handling strategy, which can be configured through the retransmission parameter configuration file `config.properties`. When a file with the same name already exists in the receiving directory, the file transfer system can handle it according to the duplicate file policy configured on the task. Duplicate file policy options include: 0 - Overwrite (directly overwrite the existing file), 1 - Notify of failure (do not overwrite the file and notify of transmission failure), and 2 - Backup (back up the existing file before overwriting). By default, the system can be set to option 1, which notifies of failure when a file with the same name is found. Furthermore, the first-level retransmission mechanism is responsible for controlling the retry logic after a file transfer failure. Its default retransmission duration is 60 minutes, the number of retries is 5, and the interval between each retry is 3 minutes. These default values ​​are suitable for most scenarios, but for special needs, users can adjust the retransmission parameters according to the actual situation, such as increasing or decreasing the number of retries or adjusting the retry interval.

[0076] Optionally, if the user chooses not to use the "0-overwrite" strategy but wants to ensure the file is up-to-date during retransmission, this can be achieved through a combination of overwrite and verification. This method allows for file verification while overwriting existing files, ensuring that the transmitted file is up-to-date and error-free. In this case, the user needs to simultaneously set the retransmission duration, number of retries, and retry interval. For example, setting the retransmission duration to 60 minutes, the number of retries to 5, and the retry interval to 3 minutes not only guarantees the reliability of file transmission but also ensures that data loss or corruption will not occur due to erroneous operations when overwriting files, thereby improving the overall stability and security of file transmission.

[0077] In one optional embodiment, when the first campus is detected to be unable to provide service due to a fault, file transfer requests related to the target application are redirected from the first campus to the second campus by modifying the domain name configuration information of the target application. Simultaneously, the file service client platform and batch scheduling platform of the second campus are activated, with the second campus acting as the primary campus for batch file transfer and job scheduling. This includes: when the file transfer system detects that the first campus is unable to provide service due to a fault, it modifies the domain name configuration information of the target application, wherein the modification operation points the domain name resolution of the target application to a service IP address within the second campus, and activates the file service client platform of the second campus. The activated file service client platform of the second campus then takes over the file transfer tasks of the first campus. Finally, the batch scheduling platform of the second campus is activated, taking over all ongoing and pending batch jobs.

[0078] Optionally, when the file transfer system detects that the first campus is unavailable due to a failure, it will trigger a series of failover operations to ensure service continuity. First, the file transfer system will modify the domain name configuration information of the target application, pointing the target application's domain name resolution to the service IP address in the second campus. By updating the domain name resolution, all file transfer requests originally directed to the first campus will be automatically redirected to the second campus, thus avoiding service interruption caused by a failure in the first campus. Furthermore, the file transfer system will activate the file service client platform in the second campus. Once activated, the platform will take over the file transfer tasks from the first campus; that is, the file service client platform in the second campus will begin handling all file transfer-related operations, including receiving, storing, and synchronizing files, ensuring a smooth switchover and continuous operation of the file service.

[0079] After activating the file service client platform and taking over file transfer tasks, the file transfer system will further activate the batch scheduling platform in the second campus. Once activated, the batch scheduling platform will take over all ongoing and pending batch jobs, ensuring the continuity of batch file processing tasks. Even in the event of a failure in the first campus, the file transfer system can still schedule and execute batch jobs according to the predetermined plan. This allows the batch scheduling platform in the second campus to effectively manage the priority and execution order of file transfer tasks, ensuring that the entire system's file processing flow is unaffected by failures in the first campus, thereby guaranteeing business continuity and high system availability.

[0080] Figure 4 This diagram illustrates a remote campus failover architecture. This architecture allows for switching to a remote campus to handle batch file transfers when an application in the local campus becomes unavailable. For example, if campuses A and B are unavailable, the system can switch to campus C. Campus C is configured as a disaster recovery campus, possessing its own file service client platform, batch scheduling platform, and independent file storage system. When a failover is required, campus C will operate as the primary campus, taking over batch file transfer tasks to ensure stable system operation and business continuity.

[0081] When an application in Zone A or Zone B fails, file transfer requests can be switched from the local zone to Zone C by modifying the domain name configuration information. The file service client master node and batch scheduling master node in Zone C will then start and take over the file transfer tasks, enabling file reception, sending, and processing through the file shared storage system. Figure 4Solid arrows indicate the flow of received file data, dashed arrows indicate the flow of sent file data, and dotted arrows indicate the flow of control and scheduling data. This switching mechanism not only improves the system's fault tolerance but also optimizes resource utilization efficiency, ensuring that the file transfer system can quickly recover and maintain continuous business operation in the face of local campus failures. Furthermore, the configuration of Campus C includes additional applications, such as upstream application D and downstream application E, to support more comprehensive business needs and disaster recovery capabilities.

[0082] For example, in a production environment, file transfer tasks may involve cross-site access. For instance, for a link from a.com.cn to b.com.cn, when the target application B performs a site switch, the domain name of b.com.cn needs to be resolved to the remote environment through domain name configuration information. However, for a link from b.com.cn to c.com.cn, c.com.cn needs to be resolved to the address in the production environment through the local hosts file. This configuration ensures that the file transfer task can correctly access the target application during site switches, while maintaining compatibility with the production environment.

[0083] In disaster recovery scenarios, file transfer tasks can be run in a closed loop across different locations. For example, when target application B and its upstream and downstream applications undergo a simultaneous off-site switchover, the access traffic of the links d.dr.com.cn→b.dr.com.cn and b.dr.com.cn→e.dr.com.cn can achieve closed-loop access in the disaster recovery environment. This ensures that all file transfer operations are completed in the off-site disaster recovery environment without relying on resources in the production environment, thereby improving the system's independence and reliability and ensuring rapid service recovery in the event of a failure in the main campus.

[0084] Because the file-shared storage system does not support remote off-site mounting, file storage failover must be performed in conjunction with off-site failover. When a failover occurs in the fault domain within the network-attached storage cluster where the participating application resides, the file transfer system can pause data writing and wait for the data in transit to complete its off-site transfer. After data synchronization is complete, the off-site disaster recovery storage volume will be promoted to the primary storage volume, and reverse data synchronization will be established. The entire failover process is expected to be completed within 10 to 15 minutes, ensuring file storage continuity and data consistency.

[0085] In a remote switchover scenario, the executor node switchover includes removing the executor node from the production park and starting the remote disaster recovery executor node. For batch tasks interrupted due to the switchover, the executor will automatically attempt to resubmit the tasks. If the resubmission fails, manual intervention is required to resume the download or redo the interrupted task. Before starting the remote disaster recovery executor node, it is necessary to wait for the file storage switchover to complete. The entire switchover process is expected to be completed within 10 to 20 minutes to ensure that batch tasks can be successfully resumed.

[0086] Switching file transfer nodes involves stopping the file receiving service in the production area, modifying the target application's domain name configuration information to point it to the IP address of the off-site backup node, and starting the file receiving service in the off-site disaster recovery environment. If access from the upstream application to the target application involves cross-regional connections, firewall rules need to be enabled in advance to ensure network connectivity. Before starting the off-site file receiving service, it is also necessary to wait for the file storage switchover to complete. The entire switchover process is expected to be completed within 10 to 20 minutes, ensuring that file transfer tasks can be successfully restored in the off-site environment.

[0087] Optionally, a new disaster recovery zone, zone D, can be added. Zone D can follow the deployment method of zone B, independently deploying the file service client platform and batch scheduling platform, and incorporating file storage into shared storage, such as NAS. This deployment method further enhances the disaster recovery capabilities of the file transfer system. For example, when a city-wide disaster causes zones A and B to become unavailable simultaneously, the file transfer system can quickly switch to zones C and D, achieving mutual backup between disaster recovery zones within the same city, thereby ensuring production stability and ensuring that, even in extreme circumstances, the file transfer system can still maintain the operation of critical businesses through the resources of the disaster recovery zone. Furthermore, disaster recovery file transfer tasks can run in a closed loop across different locations. For example, when it is necessary to transfer files from application D in zone D to target application B, upstream application D can achieve closed-loop access traffic within the disaster recovery environment through the link d.dr.com.cn→b.dr.com.cn. This closed-loop transmission mechanism ensures that the file transfer task is completed independently in the disaster recovery environment, without relying on the resources of the production zone, thereby improving the system's independence and reliability, and further guaranteeing business continuity.

[0088] In an optional embodiment, after starting the batch scheduling platform of the second park, the batch file transfer method further includes: the file transfer system remounts the same file shared storage system as the first park in the second park, and then implements the cluster master-slave switch between the first park and the second park through the file shared storage system, and adjusts the file storage volume of the second park from the backup storage volume to the master storage volume. During the implementation of the cluster master-slave switch, the file data writing operation is suspended until all data in transit is synchronized, and then the data read and write service of the file shared storage system of the second park is restarted.

[0089] Optionally, the file transfer system remounts the same shared file storage system as the first campus in the second campus, ensuring consistency in storage architecture between the two campuses and facilitating high availability and data consistency between them. Through the shared file storage system, data can be smoothly synchronized and shared between the first and second campuses. During cluster failover, the file transfer system will upgrade the file storage volume in the second campus from a standby volume to a primary volume. This means the storage volume in the second campus will take over the functions of the storage volume in the first campus, becoming the primary data storage and access point. This failover mechanism effectively addresses potential failures in the first campus, ensuring the continuity and reliability of data storage.

[0090] During the cluster master-slave failover process, the file transfer system suspends file data write operations to prevent potential data conflicts or inconsistencies during the failover, ensuring that all data in transit is safely and completely synchronized to guarantee data consistency and integrity. Once all data synchronization is complete, the file transfer system restarts the data read and write services of the shared file storage system in the second campus, restoring normal file storage and access functionality. In this way, the second campus can smoothly take over the storage tasks of the first campus, continuing to provide users with stable and reliable data storage and access services, thereby ensuring high availability and data consistency for the entire system.

[0091] In an optional embodiment, the batch file transfer method further includes: during the process of cluster master-slave switching between the first and second zones through the file shared storage system, the file transfer system maintains the data consistency between the second and third zones through asynchronous data transfer or synchronous data transfer.

[0092] Optionally, during the cluster master-slave switchover process between the first and second zones in the file transfer system, the system employs either asynchronous or synchronous data transmission to ensure data consistency between the shared file storage systems of the second and third zones. Asynchronous data transmission allows data between the second and third zones to be updated and synchronized at different times, reducing real-time requirements and improving the flexibility and efficiency of the file transfer system, especially suitable for situations with high network latency or limited resources. Synchronous data transmission, on the other hand, ensures data consistency between the second and third zones at all times. Through real-time update and synchronization mechanisms, it guarantees data real-time performance and accuracy, making it suitable for scenarios with high data consistency requirements. By flexibly utilizing both asynchronous and synchronous data transmission methods, the file transfer system can effectively maintain data consistency between the shared file storage systems of the second and third zones under different network environments and business needs, thereby ensuring the stable operation of the entire file transfer system and data reliability.

[0093] For example, the overall switchover is incorporated into the campus-level emergency response process and can be completed within half an hour. Regarding file storage switchover, the file-sharing storage system implements a primary / backup switch, promoting the local backup storage volume to the primary volume to ensure data storage continuity and reliability. Simultaneously, switching file transfer nodes involves modifying the target application's domain name configuration information, pointing the domain name to the local backup node's IP address, then restarting the file service and batch scheduling nodes taking over the campus, remounting the file storage of the file-sharing storage system, and starting the local file receiving service to restore file transfer functionality. Switching executor nodes involves restarting the executor nodes taking over the campus and remounting the file storage of the file-sharing storage system. For batch tasks interrupted due to the switchover, the executor will automatically attempt to resubmit the tasks; if the resubmission fails, manual intervention is required to perform breakpoint resume or breakpoint redo to ensure successful task completion. This integrated switchover process ensures that the file transfer system can quickly respond and restore critical functions in the event of a campus-level failure, minimizing the impact of the failure on business operations and guaranteeing the continuity and stability of file transfer and batch processing tasks.

[0094] In one optional embodiment, when a disaster recovery switchover is required, file transfer requests related to the target application are redirected from the first and second zones to the third zone by modifying the domain name configuration information of the target application. Simultaneously, the file service client platform and batch scheduling platform of the third zone are activated. Utilizing the data synchronization mechanism of the off-site storage, batch files are switched to the third zone. This includes: when a disaster recovery switchover is required, the file transfer system adjusts the domain name configuration information of the target application, redirecting access from the target application to the third zone, and activates the file service client platform of the third zone. The activated file service client platform of the third zone takes over file transfer tasks from the first and second zones. Then, the batch job scheduling platform of the third zone is activated, taking over all ongoing and pending batch jobs.

[0095] Optionally, when the file transfer system needs to perform a disaster recovery switchover, the domain name configuration information of the target application can be adjusted first, thereby redirecting access requests to the third zone. By modifying the domain name resolution, all access requests originally directed to the first or second zone will be automatically forwarded to the third zone, ensuring that users can still access application services smoothly even if the first or second zone fails. Furthermore, the file transfer system can activate the file service client platform in the third zone. Once activated, the platform will take over the file transfer tasks from the first and second zones. That is, the file service client platform in the third zone will begin handling all file transfer-related operations, including receiving, storing, and synchronizing files, ensuring the continuity and stability of the file service and maintaining normal file processing even if other zones fail.

[0096] After activating the file service client platform and taking over file transfer tasks, the file transfer system will further activate the batch job scheduling platform in the third campus. Once activated, the batch job scheduling platform will take over all ongoing and pending batch jobs to ensure the continuity of batch file processing tasks. Even in the event of a failure in the first or second campus, the file transfer system can still schedule and execute batch jobs according to the predetermined plan. As scheduled, the batch scheduling platform in the third campus can effectively manage the priority and execution order of file transfer tasks, ensuring that the entire system's file processing flow is not affected by failures in other campuses, thereby guaranteeing business continuity and high system availability.

[0097] In one optional embodiment, during batch file transfer using any one of the main campuses as the primary campus, when a file retransmission is detected, the file to be transferred uploaded by the file sender and its hash value are forwarded to the file receiver via the file service client platform of the primary campus. This includes: the file transfer system moving the target file uploaded by the file sender from the sending directory to the target directory via the file service client platform of the primary campus, and creating a marker file with the same name as the target file in the target directory via the file service client platform of the primary campus. After detecting that the file sender has read the target file from the target directory, the system notifies the file sender to calculate the hash value for the target file, and then sends the target file and the hash value calculated by the file sender for the target file to the file receiver via the file service client platform of the primary campus.

[0098] Optionally, during file transfer within the file transfer system, the main campus file service client platform can move the target file uploaded by the sender from the sending directory to the target directory, ensuring the file's correct storage location within the system. Furthermore, the file service client platform can create a marker file with the same name as the target file in the target directory. This marker file records the file's transfer status or serves as an identifier for completed file transfer, facilitating system tracking and management of the file transfer process. When the system detects that the sender has read the target file from the target directory, it notifies the sender to calculate the hash value of the target file. The hash value, as a digital fingerprint of the file content, is used to verify whether the file maintains its integrity and consistency during transfer, ensuring that the file has not been tampered with or corrupted.

[0099] Optionally, the file service client platform in the main campus can send the target file and its hash value calculated by the sender to the file receiver. This not only transmits the file itself but also provides information to verify file integrity. Upon receiving the file and hash value, the receiver can compare its locally calculated hash value with the one provided by the sender to determine if the file has been altered during transmission. If the hash values ​​match, the file is intact and usable; if they do not match, the file may have been corrupted or transmitted incorrectly, requiring appropriate processing. This ensures the reliability and accuracy of file transmission and improves the stability and trustworthiness of the entire file processing system.

[0100] In one optional embodiment, after the target file and the hash value calculated by the file sender for the target file are sent to the file receiver via the file service client platform of the main campus, the file receiver performs the following steps: The file transfer system saves the received file to a temporary directory, calculates the hash value of the received file, and then checks whether the hash value calculated by the file receiver for the received file is consistent with the hash value provided by the file sender. If the hash value calculated by the file receiver for the received file is consistent with the hash value provided by the file sender, the received file is moved from the temporary directory to the official directory, and a successful reception signal is sent to the file sender; if the hash value calculated by the file receiver for the received file is inconsistent with the hash value provided by the file sender, the file transfer is deemed to have failed.

[0101] Optionally, during file reception, the file transfer system can save the received file to a temporary directory and calculate its hash value. This allows for integrity checks at the initial stage of file reception, ensuring the file hasn't been tampered with or corrupted during transmission. The system then checks if the hash value calculated by the recipient matches the hash value provided by the sender. This hash value comparison verifies the integrity of the received file, quickly determining if it has maintained its original state during transmission. If the hash value calculated by the recipient matches the hash value provided by the sender, the file transfer is successful without errors. At this point, the file transfer system can move the received file from the temporary directory to the final directory and send a success signal to the sender, completing the entire file reception process.

[0102] Optionally, if the hash value calculated by the file receiver does not match the hash value provided by the file sender, it indicates that the file may have been corrupted or erroneous during transmission, resulting in inconsistencies between the file content and the original file. In this case, the file transfer system will determine that the file transfer has failed and take corresponding measures, such as logging errors and notifying the file sender to retransmit the file. The hash value comparison-based verification mechanism of this application not only improves the reliability of file transfer but also provides the file transfer system with the ability to quickly detect and handle transmission errors, thereby ensuring the accuracy and stability of the entire file transfer process.

[0103] Figure 6This diagram illustrates a process for verifying retransmitted files. The file service client platform in the main campus, acting as sender A, first performs a SHA-256 hash calculation on the file test.bin, generating a hash value H1, and sends it along with the file to receiver B. Upon receiving the file, receiver B also performs the same hash calculation on the test.bin file. By comparing the hash value calculated by receiver B with the hash value provided by sender A, receiver B can verify whether the file has been tampered with or corrupted during transmission. If the two hash values ​​are equal, it means the file has not been altered, and receiver B moves the file from the temporary directory to the official directory and sends a successful reception signal to sender A. If the two hash values ​​do not match, receiver B overwrites the old file, records the new hash value H1, and sends a successful transmission signal. This process ensures that even if file transmission fails and retransmission is required, receiver B receives the latest and complete file, thus guaranteeing the stable operation of the system and business continuity.

[0104] For example, during file transfer, when sender A prepares to send the file test.bin, it first reads the file located in the send.etransdtmp directory and hashes it using an encryption algorithm, calculating a hash value H1. Then, sender A transmits the file test.bin and its hash value H1 to receiver B through the file service client platform. This process ensures that the file has a unique hash identifier before transmission, used for subsequent verification. Upon receiving a retransmission request, receiver B calculates the hash value H0 of the previously received test.bin file in the recv.etranrevtmp directory. By comparing hash values ​​H0 and H1, receiver B can determine if the file has been modified. If the hash values ​​are the same, it means the file has not been modified, and receiver B returns a "file unchanged, skip transmission" message and notifies sender A of the retransmission result, avoiding unnecessary file transfers, saving resources, and improving efficiency. If hash values ​​H0 and H1 are different, it indicates that the latest uploaded batch of files is a new file. In this scenario, receiver B overwrites the old file and records the new hash value H1. It then returns a "transmission successful" message to sender A. This not only ensures the accuracy of file transmission but also prevents duplicate file transmissions through hash verification. Furthermore, it ensures that the old version is overwritten in a timely manner when the file is updated, guaranteeing that the receiver always has the latest file content.

[0105] In an optional embodiment, the batch file transfer method further includes: after the file sender receives a signal indicating that the target file has failed to be received, the file transfer system moves the target file from the target directory to the retransmission failure directory; if a one-time retransmission strategy is configured, the file sender moves the files in the retransmission failure directory back to the sending directory and initiates a one-time retransmission process; if the receiver still reports failure after the one-time retransmission process, and a two-time retransmission strategy is configured, the file sender moves the failed files back to the sending directory again for a two-time retransmission, wherein, during the two-time retransmission, the file sender repeatedly executes the file transfer and hash verification process according to a preset number of retransmissions and retransmission interval; during the two-time retransmission process, the file receiver continues to execute file reception, hash value calculation, and hash value consistency comparison.

[0106] Optionally, when the file transfer system detects that the file recipient reports a failure to receive the target file, the file sender will move the target file from the current target directory to the retransmission failure directory to categorize and manage failed files, facilitating subsequent retransmission. If the file transfer system is configured with a single retransmission policy, the file sender will move the files in the retransmission failure directory back to the sending directory and initiate a single retransmission process. By retransmitting the files, the file transfer system has the opportunity to correct transmission failures caused by network problems or other temporary faults. If the recipient still reports failure after the first retransmission, and the file transfer system is configured with a second retransmission policy, the file sender will move the failed files back to the sending directory again for a second retransmission. During the second retransmission process, the file sender will repeatedly execute the file transfer and hash verification process according to the preset number of retransmissions and retransmission intervals to ensure that the file is accurately transmitted to the recipient.

[0107] Optionally, during the second retransmission, the file receiver will continue to perform file reception, hash value calculation, and hash value consistency comparison. That is, the receiver will re-verify the integrity of the received file, comparing its hash value to determine whether the file has maintained its integrity during retransmission. This not only improves the reliability of file transmission but also ensures that file integrity and consistency are rigorously verified even with multiple retransmissions. In this way, the file transfer system can effectively handle file transmission failures in complex network environments, improving the success rate of file transmission while ensuring the accuracy and security of file data.

[0108] During batch file transfers, the file transfer system utilizes the domain name switching mechanism of the file service client to quickly switch between the file source and destination locations. This ensures efficient and fast file transfer while maintaining file integrity and consistency. The shared file storage is routinely connected to application file servers and uses a regional synchronization mechanism to quickly synchronize files locally and across regions. This effectively guarantees the connectivity of the shared file storage and ensures that batch files can be quickly connected and synchronized when switching between local and cross-regional locations, thereby enabling rapid system recovery and business continuity. Furthermore, during batch file uploads, if retransmissions occur, the file transfer system uses a retransmission mechanism, combining first and second retransmission strategies with hash value comparison, to ensure that the latest files are successfully transmitted. This avoids data inconsistencies caused by transmission failures, further improving the reliability and stability of file transfers.

[0109] See Figure 7 According to another aspect of the embodiments of this application, a batch file transfer device is also provided, including: a platform deployment unit 701, a first processing unit 702, and a second processing unit 703.

[0110] The platform deployment unit 701 is used to deploy file service client platforms and batch scheduling platforms in multiple zones. These zones include at least a first zone, a second zone, and a third zone. The first and second zones belong to the same region, while the third zone does not belong to the same region as the first zone. The file service client platforms and batch scheduling platforms within each zone achieve local file storage and cross-zone synchronization through file-sharing storage. The first processing unit 702, when detecting that the first zone is unable to provide service due to a fault, modifies the domain name configuration information of the target application to redirect file transfer requests related to the target application from the first zone to the second zone. Simultaneously, it starts the file service client platform and batch scheduling platform in the second zone, using the second zone as the primary zone for batch file transfer and job scheduling. The second processing unit 703, when a remote disaster recovery switch is required, modifies the domain name configuration information of the target application to redirect file transfer requests related to the target application from the first and second zones to the third zone. Simultaneously, it starts the file service client platform and batch scheduling platform in the third zone, utilizing the remote storage data synchronization mechanism to switch batch files to the third zone.

[0111] Optionally, the batch file transfer device further includes: a third processing unit, used to forward the target file uploaded by the file sender and its hash value to the file receiver through the file service client platform of the main campus when a file retransmission is detected during the batch file transfer process with any campus as the main campus. The file receiver determines whether to update the local file by comparing the hash value of the local file with the hash value uploaded by the file sender.

[0112] Optionally, the batch file transfer device further includes: a fourth processing unit, used to, when a first retransmission failure of a target file is detected during the batch file transfer process with any one of the parks as the main park, move the target file to the target directory and attempt a second retransmission through the file service client platform of the main park according to the preset retransmission identifier and retransmission strategy, wherein the retransmission strategy includes: a retransmission duration setting strategy, a retransmission number setting strategy, and a retransmission interval setting strategy.

[0113] Optionally, the first processing unit 702 includes: a modification operation subunit, used to modify the domain name configuration information of the target application when the first campus is found to be unable to provide services due to a fault, wherein the modification operation is used to point the domain name resolution of the target application to the service IP address in the second campus; a first client platform activation subunit, used to activate the file service client platform of the second campus, wherein the activated file service client platform of the second campus takes over the file transfer tasks of the first campus; and a first batch scheduling platform startup subunit, used to start the batch scheduling platform of the second campus, wherein the started batch scheduling platform of the second campus takes over all ongoing and pending batch jobs.

[0114] Optionally, the batch file transfer device further includes: a remount unit for remounting the same file-sharing storage system as the first park in the second park; and a switching processing unit for implementing cluster primary / standby switching between the first and second parks through the file-sharing storage system, and adjusting the file storage volume in the second park from the standby storage volume to the primary storage volume. During the implementation of the cluster primary / standby switching, file data writing operations are suspended until all data in transit is synchronized, and then the data read / write service of the file-sharing storage system in the second park is restarted.

[0115] Optionally, the batch file transfer device further includes a data maintenance unit, used to maintain the data consistency between the second and third campuses of the file-sharing storage system during the cluster master-slave switch between the first and second campuses through the file-sharing storage system, by means of asynchronous data transmission or synchronous data transmission.

[0116] Optionally, the second processing unit 703 includes: an adjustment operation subunit, used to adjust the domain name configuration information of the target application when a remote disaster recovery switch is required, wherein the adjustment operation is used to direct the access of the target application to the third park; a second client platform activation subunit, used to activate the file service client platform of the third park, wherein the activated file service client platform of the third park takes over the file transfer tasks of the first and second parks; and a second batch scheduling platform startup subunit, used to start the batch job scheduling platform of the third park, wherein the started batch scheduling platform of the third park takes over all ongoing and pending batch jobs.

[0117] Optionally, the third processing unit includes: a file transfer subunit, used to move the target file uploaded by the file sender from the sending directory to the target directory through the file service client platform of the main campus; a file creation subunit, used to create a tag file with the same name as the target file in the target directory through the file service client platform of the main campus; a hash value calculation subunit, used to notify the file sender to perform hash value calculation for the target file after detecting that the file sender has read the target file from the target directory; and a hash value sending subunit, used to send the target file and the hash value calculated by the file sender for the target file to the file receiver through the file service client platform of the main campus.

[0118] Optionally, the batch file transfer device further includes: a file receiving processing unit, used to save the received file to a temporary directory and calculate the hash value of the received file; a hash value detection unit, used to detect whether the hash value calculated by the file receiver for the received file is consistent with the hash value provided by the file sender; a hash value consistency processing unit, used to move the received file from the temporary directory to the official directory and send a successful reception signal to the file sender if the hash value calculated by the file receiver for the received file is consistent with the hash value provided by the file sender; and a hash value inconsistency processing unit, used to determine that the file transfer has failed if the hash value calculated by the file receiver for the received file is inconsistent with the hash value provided by the file sender.

[0119] Optionally, the batch file transfer device further includes: a file transfer unit, used to move the target file from the target directory to the retransmission failure directory after the file sender receives a signal indicating that the target file has failed to be received; a retransmission process processing unit, used to move the files in the retransmission failure directory back to the sending directory and start the first retransmission process if a first retransmission policy is configured; and a second retransmission processing unit, used to move the files that failed to be received back to the sending directory for a second retransmission if the receiver still reports failure after the first retransmission process and a second retransmission policy is configured. During the second retransmission, the file sender repeatedly executes the file transfer and hash verification process according to a preset number of retransmissions and retransmission interval. During the second retransmission, the file receiver continues to perform file reception, hash value calculation, and hash value consistency comparison.

[0120] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described batch file transfer method.

[0121] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described batch file transfer method.

[0122] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the above-described batch file transfer method.

[0123] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0124] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0126] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0129] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A batch file transfer method, characterized in that, include: File service client platforms and batch scheduling platforms are deployed in multiple parks, including at least a first park, a second park, and a third park. The first park and the second park belong to the same region, while the third park does not belong to the same region as the first park. The file service client platform and batch scheduling platform in each park achieve local storage and cross-park synchronization of files through file sharing storage. When it is detected that the first park is unable to provide services due to a fault, the file transfer requests related to the target application are redirected from the first park to the second park by modifying the domain name configuration information of the target application. At the same time, the file service client platform and batch scheduling platform of the second park are started, and the second park is used as the main park for batch file transfer and job scheduling. When a disaster recovery switchover is required, the domain name configuration information of the target application is modified to redirect file transfer requests related to the target application from the first and second parks to the third park. At the same time, the file service client platform and batch scheduling platform of the third park are started, and the batch files are switched to the third park using the data synchronization mechanism of the off-site storage.

2. The batch file transfer method according to claim 1, characterized in that, The batch file transfer method further includes: During the batch file transfer process using any one of the parks as the main park, when a file retransmission is detected, the file service client platform of the main park forwards the target file and hash value uploaded by the file sender to the file receiver. The file receiver determines whether to update the local file by comparing the hash value of the local file with the hash value uploaded by the file sender.

3. The batch file transfer method according to claim 2, characterized in that, The batch file transfer method further includes: During the batch file transfer process using any one of the parks as the main park, when the first retransmission of the target file fails, the file service client platform of the main park moves the target file to the target directory and attempts a second retransmission according to the preset retransmission identifier and retransmission strategy. The retransmission strategy includes: a retransmission duration setting strategy, a retransmission number setting strategy, and a retransmission interval setting strategy.

4. The batch file transfer method according to claim 1, characterized in that, When the first campus is detected to be unable to provide service due to a fault, the file transfer requests related to the target application are redirected from the first campus to the second campus by modifying the domain name configuration information of the target application. Simultaneously, the file service client platform and batch scheduling platform of the second campus are launched, using the second campus as the primary campus for batch file transfer and job scheduling, including: When it is detected that the first park is unable to provide services due to a fault, the domain name configuration information of the target application is modified, wherein the modification operation is used to point the domain name resolution of the target application to the service IP address in the second park; Activate the file service client platform of the second park, wherein the activated file service client platform of the second park takes over the file transfer tasks of the first park; The batch scheduling platform of the second park is started, and the started batch scheduling platform of the second park takes over all batch jobs that are in progress and pending execution.

5. The batch file transfer method according to claim 4, characterized in that, After activating the batch scheduling platform in the second park, the batch file transfer method further includes: Remount the same file-sharing storage system as the first zone in the second zone; The file-sharing storage system is used to implement cluster master-slave switching between the first and second parks, and the file storage volume in the second park is changed from a backup storage volume to a master storage volume. During the implementation of the cluster master-slave switching, file data writing operations are suspended until all data in transit is synchronized, and then the data read and write services of the file-sharing storage system in the second park are restarted.

6. The batch file transfer method according to claim 5, characterized in that, The batch file transfer method further includes: During the cluster master-slave switchover between the first and second parks through the file-sharing storage system, data consistency between the file-sharing storage systems of the second and third parks is maintained through asynchronous or synchronous data transmission.

7. The batch file transfer method according to claim 1, characterized in that, When a disaster recovery switchover is required, by modifying the domain name configuration information of the target application, file transfer requests related to the target application are redirected from the first and second zones to the third zone. Simultaneously, the file service client platform and batch scheduling platform of the third zone are activated. Utilizing the off-site storage data synchronization mechanism, batch files are switched to the third zone, including: When a disaster recovery switch is required, the domain name configuration information of the target application is adjusted, wherein the adjustment operation is used to direct the access of the target application to the third park. Activate the file service client platform of the third park, wherein the activated file service client platform of the third park takes over the file transfer tasks of the first park and the second park; The batch job scheduling platform of the third park is started, wherein the started batch scheduling platform of the third park takes over all batch jobs that are in progress and pending execution.

8. The batch file transfer method according to claim 2, characterized in that, During batch file transfer using any one of the main campuses as the primary campus, when a file retransmission is detected, the file service client platform of the primary campus forwards the file to be transferred uploaded by the file sender and its hash value to the file receiver, including: The target file uploaded by the file sender is moved from the sending directory to the target directory through the file service client platform of the main park. Using the file service client platform of the main park, create a tag file with the same name as the target file in the target directory; After detecting that the file sender reads the target file from the target directory, the file sender is notified to calculate the hash value for the target file; The target file and the hash value calculated by the file sender for the target file are sent to the file receiver through the file service client platform of the main park.

9. The batch file transfer method according to claim 8, characterized in that, After the target file and the hash value calculated by the file sender for the target file are sent to the file receiver through the file service client platform of the main park, the file receiver performs the following steps: Save the received file to a temporary directory and calculate the hash value of the received file; The system checks whether the hash value calculated by the file recipient for the received file is consistent with the hash value provided by the file sender. If the hash value calculated by the file recipient for the received file is found to be consistent with the hash value provided by the file sender, the received file will be moved from the temporary directory to the official directory, and a successful reception signal will be sent to the file sender. If the hash value calculated by the file recipient for the received file is inconsistent with the hash value provided by the file sender, the file transfer is determined to have failed.

10. The batch file transfer method according to claim 9, characterized in that, The batch file transfer method further includes: After the file sender receives a signal indicating that the target file failed to be received, the target file is moved from the target directory to the retransmission failure directory; If a retransmission policy is configured, the file sender will move the files in the failed retransmission directory back to the sending directory and start the retransmission process. If the receiver still reports failure after one retransmission process, and a second retransmission strategy is configured, the file sender will move the failed file back to the sending directory for a second retransmission. During the second retransmission, the file sender will repeatedly execute the file transfer and hash verification process according to the preset number of retransmissions and retransmission interval. During the second retransmission, the file receiver will continue to execute file reception, hash value calculation, and hash value consistency comparison.

11. A batch file transfer device, characterized in that, include: The platform deployment unit is used to deploy a file service client platform and a batch scheduling platform in multiple parks. The multiple parks include at least a first park, a second park, and a third park. The first park and the second park belong to the same region, while the third park does not belong to the same region as the first park. The file service client platform and the batch scheduling platform in each park achieve local storage and cross-park synchronization of files through file sharing storage. The first processing unit is used to, when it is detected that the first park cannot provide services due to a fault, modify the domain name configuration information of the target application to redirect the file transfer request related to the target application from the first park to the second park, and at the same time start the file service client platform and batch scheduling platform of the second park, and use the second park as the main park to perform batch file transfer and job scheduling. The second processing unit is used to, when a disaster recovery switch is required, modify the domain name configuration information of the target application to redirect file transfer requests related to the target application from the first and second parks to the third park, and simultaneously start the file service client platform and batch scheduling platform of the third park, and use the data synchronization mechanism of the off-site storage to switch batch files to the third park.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device on which the computer-readable storage medium is located performs the batch file transfer method according to any one of claims 1 to 10.

13. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the bulk file transfer method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the batch file transfer method of any one of claims 1 to 10.

Images