File processing method, device and system during GFS fault

Abstract

The invention discloses a file processing method, device and system during GFS failure, a GFS comprises a plurality of data centers, each data center comprises at least two GFS clusters, one cluster is a main GFS cluster, the other clusters are auxiliary GFS clusters, and the method comprises the steps that when the main GFS cluster of the current data center is abnormal, the main GFS cluster of the current data center is abnormal, the main GFS cluster of the current data center is abnormal, and the auxiliary GFS cluster of the current data center is abnormal; when the to-be-stored file cannot be written into the main GFS cluster of the current data center, writing the to-be-stored file into a normal main cluster of any other data center; and synchronizing the copy of the file to be stored to the auxiliary GFS cluster of each data center by using the GFS synchronization function of the normal main cluster of the data center written into the original. According to the method and the device, the file can be ensured to be successfully uploaded to the greatest extent, and the file can be obtained from the current downloading data center when any data center receives the file downloading request subsequently, so that the time consumption, the network flow and the possible fault probability caused by cross-center downloading are reduced.

Description

Technical Field

[0001] The present invention relates to the field of distributed technologies, and in particular, to a method, apparatus, and system for file processing when GFS fails. Background Art

[0002] This section aims to provide background or context for the embodiments of the present invention described in the claims. The description herein is not admitted to be prior art merely because it is included in this section.

[0003] In the business of the network payment and settlement platform, some services involve the mutual transmission of files among payment institutions, the network joint clearinghouse, and banks. As the central node of the service link, the network joint clearinghouse needs to receive file upload requests from participants, store them on the storage medium, and when receiving a request from a participant to download a file, read the file from the storage medium and return it to the requester. The current common storage medium of the network joint clearinghouse is GFS. The network joint clearinghouse supports receiving requests from participants from 6 processing centers. After the file is uploaded, the file downloader does not need to care about the processing center where the file was originally uploaded. The original file can be obtained by applying for download at any center.

[0004] Content of the Prior Art Solution

[0005] As Figure 1 shown, a participant uploads file a to Center A, and the system of Center A stores the file in the GFS-A-1 cluster; the GFS background synchronization function synchronizes the file on this cluster to other GFS clusters: GFS-B-1, that is, the replicated file a'; the file download participant requests Center B to download the file, and the system of Center B reads the replicated file a' of the original file from the GFS-B-1 cluster and returns it to the participant.

[0006] Disadvantages of the Prior Art Solution

[0007] 1) The prior art solution only supports uploading files to the GFS cluster of the local center. When the GFS cluster of the local center fails and files cannot be written, all participants' file uploads to this center will fail. The schematic diagram is shown in Figure 2 .

[0008] 2) The prior art solution can run well when the GFS synchronization function is normal. Once the GFS synchronization function fails, it is impossible to obtain from other centers - non-the original upload center of the file - and participants will not be able to download the file, and must download the file from the original upload center of the file. The schematic diagram is shown in Figure 3 .

[0009] 3) When there is only one GFS cluster in each center in the existing technical solution, one center will not support participants to upload files: There is only one GFS-A cluster in Center A and only GFS-B cluster in Center B, and the GFS synchronization direction is from the GFS-A cluster to the GFS-B cluster. If a participant initiates a file upload request (file b) to Center B, Center B will have nowhere to store file b, resulting in all file uploads by participants to Center B failing. The schematic diagram is shown in Figure 4 。

[0010] In summary, in the existing technology, when GFS fails, it will cause file uploads to be unsuccessful. File downloads require cross-center operations, resulting in time-consuming, network traffic, and possible failure probability issues. Summary of the Invention

[0011] An embodiment of the present invention provides a method for file processing when GFS fails, to ensure the success of file uploads to the greatest extent possible, and subsequently, when any data center receives a file download request, the file can be obtained from the current download data center. The distributed file system GFS includes multiple data centers, and each data center includes at least two GFS clusters. One of the GFS clusters is the main GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. The method includes:

[0012] When any data center receives a file upload request and the main GFS cluster of the current data center is abnormal and unable to write the file to be stored into the main GFS cluster of the current data center, write the original file to be stored into the normal main GFS cluster of any other data center;

[0013] Use the GFS synchronization function of the normal main GFS cluster of the data center where the original file is written to synchronize the file copy to be stored to the secondary GFS clusters of each data center, so that subsequently, when any data center receives a file download request for the file to be stored, the downloaded file can be obtained from the current download data center.

[0014] An embodiment of the present invention provides a method for file processing when GFS fails, to ensure the success of file uploads to the greatest extent possible, and subsequently, when any data center receives a file download request, the file can be obtained from the current download data center. The distributed file system GFS includes multiple data centers, and each data center includes at least two GFS clusters. One of the GFS clusters is the main GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. The method includes:

[0015] When a request to download a file is received at any data center, and it is detected that the current download data center is the data center where the original file is written, the downloaded file is obtained from the primary GFS cluster or the secondary GFS cluster of the data center where the original file is written; wherein, the data center where the original file is written is the data center where the original file upload center receives a file upload request, and the primary GFS cluster of the original file upload center is abnormal and cannot write the file to be stored into the primary GFS cluster of the original file upload center, then the data center where the original file to be stored is written; the GFS synchronization function of the normal primary GFS cluster of the data center where the original file is written synchronizes the copy of the file to be stored to the secondary GFS cluster of each data center;

[0016] When a request to download a file is received at any data center, and it is detected that the current download data center is other than the data center where the original file is written, the downloaded file is obtained from the secondary GFS cluster of the data center other than the data center where the original file is written.

[0017] An embodiment of the present invention further provides an upload device for file processing when GFS fails, which is used to ensure the successful upload of files to the greatest extent, and subsequent when a request to download a file is received at any data center, the file can be obtained from the current download data center. GFS includes multiple data centers, and each data center includes at least two GFS clusters, where one GFS cluster is the primary GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. The upload device includes:

[0018] A writing unit, configured to write the original file to be stored into the normal primary GFS cluster of any other data center when a file upload request is received at any data center, and the primary GFS cluster of the current data center is abnormal and cannot write the file to be stored into the primary GFS cluster of the current data center;

[0019] A synchronization unit, configured to use the GFS synchronization function of the normal primary GFS cluster of the data center where the original file is written to synchronize the copy of the file to be stored to the secondary GFS cluster of each data center, so that subsequent when a request to download a file for the file to be stored is received at any data center, the downloaded file can be obtained from the current download data center.

[0020] An embodiment of the present invention further provides a download device for file processing when GFS fails, which is used to ensure the successful upload of files to the greatest extent, and subsequent when a request to download a file is received at any data center, the file can be obtained from the current download data center. GFS includes multiple data centers, and each data center includes at least two GFS clusters, where one GFS cluster is the primary GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. The download device includes:

[0021] The first download unit is configured to, when receiving a request to download a file at any data center and detecting that the current download data center is the data center where the original copy is written, obtain the download file from the primary GFS cluster or the secondary GFS cluster of the data center where the original copy is written; wherein, the data center where the original copy is written is the data center where the original file upload center receives a file upload request, and the primary GFS cluster of the original file upload center is abnormal and cannot write the file to be stored into the primary GFS cluster of the original file upload center, and the original copy of the file to be stored is written into this data center; the GFS synchronization function of the normal primary GFS cluster of the data center where the original copy is written synchronizes the copy of the file to be stored to the secondary GFS cluster of each data center.

[0022] The second download unit is configured to, when receiving a request to download a file at any data center and detecting that the current download data center is other than the data center where the original copy is written, obtain the download file from the secondary GFS cluster of the data center other than the data center where the original copy is written.

[0023] An embodiment of the present invention further provides a system for file processing during GFS failure, which is used to ensure the success of file upload to the greatest extent, and subsequent when receiving a request to download a file at any data center, the file can be obtained from the current download data center. The system includes: the upload device and the download device as described above.

[0024] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for file processing during GFS failure as described above is implemented.

[0025] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for file processing during GFS failure as described above is implemented.

[0026] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for file processing during GFS failure as described above is implemented.

[0027] In the file processing solution when GFS fails provided by the embodiments of the present invention, the distributed file system GFS includes multiple data centers, and each data center includes at least two GFS clusters. One of the GFS clusters is the primary GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. When working: when a file upload request is received in any data center and the primary GFS cluster in the current data center is abnormal and cannot write the file to be stored into the primary GFS cluster in the current data center, write the original file to be stored into the normal primary GFS cluster in any other data center; use the GFS synchronization function of the normal primary GFS cluster in the data center where the original is written to synchronize the file copy to be stored to the secondary GFS cluster in each data center. The embodiments of the present invention can ensure the successful upload of files to the greatest extent, and subsequently, when a file download request is received in any data center, the file can be obtained from the current download data center. Specifically, the downloaded file can be obtained from the primary GFS cluster or the secondary GFS cluster in the data center where the original is written, or from the secondary GFS cluster in the data center other than the data center where the original is written. Therefore, it reduces the time consumption, network traffic and possible failure probability involved in cross-center downloads. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. In the drawings:

[0029] Figure 1 is a schematic diagram of a method for file processing when GFS fails in the prior art;

[0030] Figure 2 is a schematic diagram of the defects of the method for file processing when GFS fails in the prior art Figure 1 ;

[0031] Figure 3 is a schematic diagram of the defects of the method for file processing when GFS fails in the prior art Figure 2 ;

[0032] Figure 4 is a schematic diagram of the defects of the method for file processing when GFS fails in the prior art Figure 3 ;

[0033] Figure 5 is a schematic diagram of the principle of file processing when GFS fails in the embodiments of the present invention Figure 1 ;

[0034] Figure 6Schematic diagram of file processing in case of GFS failure in the embodiments of the present invention Figure 2 ;

[0035] Figure 7 Schematic diagram of file processing in case of GFS failure in the embodiments of the present invention Figure 3 ;

[0036] Figure 8 Schematic diagram of file processing in case of GFS failure in the embodiments of the present invention Figure 4 ;

[0037] Figure 9 Schematic flow chart of the upload method for file processing in case of GFS failure in the embodiments of the present invention;

[0038] Figure 10 Schematic flow chart of the download method for file processing in case of GFS failure in the embodiments of the present invention;

[0039] Figure 11 Schematic structural diagram of the upload device for file processing in case of GFS failure in the embodiments of the present invention;

[0040] Figure 12 Schematic structural diagram of the download device for file processing in case of GFS failure in the embodiments of the present invention;

[0041] Figure 13 Schematic structural diagram of the system for file processing in case of GFS failure in the embodiments of the present invention. Detailed implementation manners

[0042] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer and more understandable, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

[0043] In the technical solutions of the present application, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of laws and regulations.

[0044] Before introducing the embodiments of the present invention, the nouns related to the present invention will be introduced first.

[0045] Network payment clearing platform: electronics payment clearing of China, a platform connecting banking financial institutions (referred to as "banks" for short) and non-bank payment institutions (referred to as "institutions" for short), mainly processing transaction transfer and fund clearing services related to payment institutions and banks, and its English abbreviation is EPCC.

[0046] NetsUnion: NetsUnion Clearing Corporation (NUCC) is the operating agency of the non-bank payment institution network payment clearing platform approved to be established, providing a unified clearing system for the non-bank payment institution network payment clearing platform.

[0047] GFS: Google File System, an extensible distributed file system designed by Google for storing massive amounts of search data, used for large-scale, distributed applications that access large amounts of data.

[0048] Primary and standby systems: Systems are deployed in multiple centers. At a certain moment, only one center's system provides services, and the systems in other centers are disaster recovery systems.

[0049] Participants: Member institutions accessing the platform to handle business, including banks, payment institutions, and other institutions approved to access the platform.

[0050] Since the inventor found technical problems: The probability of all failures in a single-center GFS cluster is not high, but it is still a potential threat; GFS synchronization function failures are not uncommon, and recently, the frequency of failures has shown an upward trend; moreover, the early warning of synchronization function failures is not timely, and troubleshooting sometimes cannot be completed in a short time. The above-mentioned various abnormal situations have affected both the service level provided by the EPCC platform to participants and the participants' experience. Therefore, a file processing solution when GFS fails is proposed, and the file processing solution when GFS fails will be introduced in detail below.

[0051] Figure 9 It is a schematic flowchart of the upload method for file processing when GFS fails in an embodiment of the present invention. The distributed file system GFS includes multiple data centers, and each data center includes at least two GFS clusters. One GFS cluster is the primary GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. As Figure 9 shown, the method includes the following steps:

[0052] Step 101: When a file upload request is received in any data center and the primary GFS cluster in the current data center is abnormal and cannot write the file to be stored into the primary GFS cluster in the current data center, write the original file to be stored into the normal primary GFS cluster in any other data center;

[0053] Step 102: Use the GFS synchronization function of the normal primary GFS cluster of the data center where the original is written to synchronize the copy of the file to be stored to the secondary GFS cluster of each data center, so that when a download file request for the file to be stored is received in any data center subsequently, the download file is obtained from the current download data center.

[0054] In the method for processing files when a GFS fails provided by an embodiment of the present invention, Figure 5 As shown, the distributed file system GFS includes multiple data centers, each of which includes at least two GFS clusters, one of which is used to store file copies (e.g. Figure 5 The primary GFS cluster for file a) (e.g. Figure 5 GFS-A-1 in , and other clusters are used to store file replicas (such as Figure 5 The secondary GFS cluster of file a′ in Figure 5 GFS-B-1 in the work: in any data center (such as Figure 5 The A center in the current data center receives the file upload request, and the main GFS cluster in the current data center (for example Figure 5 GFS-A-1 in the data center) is abnormal and the file to be stored cannot be written to the main GFS cluster of the current data center (for example Figure 5 When the original file to be stored is written to any other data center (such as GFS-A-1 in Figure 5 The normal primary GFS cluster (e.g. Figure 5 GFS-B-2 in ); using the data center where the original is written (e.g. Figure 5 The normal primary GFS cluster (e.g. Figure 5 The GFS synchronization function of GFS-B-2) in the data center synchronizes the copy of the file to be stored to the secondary GFS cluster of each (all) data center. The embodiment of the present invention can ensure the success of file upload to the greatest extent, and when a request to download a file is received in any data center subsequently, the file can be obtained from the current download data center. Specifically, the download file can be obtained from the primary GFS cluster or the secondary GFS cluster of the data center where the original is written, or from the secondary GFS cluster other than the data center where the original is written. Therefore, the time consumption, network traffic and possible failure probability caused by cross-center downloading are reduced.

[0055] The following is a detailed introduction to the file processing method when GFS fails.

[0056] like Figure 5 As shown, when the GFS clusters in each center are normal and the synchronization function of GFS is running normally, participants can upload or download files and they can be processed normally in this center.

[0057] When the GFS cluster of this center (the current data center) is abnormal and files cannot be written to this cluster, the files written by this center will be retried and written to the GFS clusters of other centers, thus ensuring the successful upload of files to the greatest extent. When the synchronization function of GFS is running abnormally, when participants request to download files to other original upload centers that do not write files, they cannot read files from the GFS of this center and will instead obtain files from the GFS clusters of other centers, so as to download the specified files for the participants.

[0058] In the embodiments of the present invention, although only the situation of two centers and 2 GFS clusters in each center is discussed, in fact, there are 3 centers and 3 GFS clusters in each center in the actual implementation of this technical solution. In addition, this technical solution can support more centers and the number of GFS clusters in each center.

[0059] The following is a detailed introduction.

[0060] As Figure 5 shown, the overall solution of the embodiments of the present invention is that when participants upload files to this center, when the GFS clusters of each center are normal and the synchronization function of GFS is running well, participants can successfully upload files to any one center, and any other participants who need to download the file can request any center and can download the file.

[0061] 1) When participants upload files to a certain center, this center preferentially stores the files on the GFS cluster of this center. When participants download files, this center preferentially reads the files or file copies from the GFS cluster of this center. Preferentially storing in this center and preferentially reading from this center reduces the failure probability increased due to the increase in network devices involved in cross-center on the one hand, and reduces the additional time consumption and cross-center network traffic brought by cross-center on the other hand. That is, in one embodiment, when writing a file to be stored, the file to be stored is preferentially stored on the GFS cluster of the current data center.

[0062] 2) When the GFS cluster (GFS-A-1) of this center malfunctions, this center preferentially tries to store the file on the GFS cluster (GFS-A-1) of this center. Due to the failure of this GFS cluster, the storage of the file will also fail, and the application of this center will try to upload the file to the GFS cluster (GFS-B-2) of other centers. This file can also be synchronized to the GFS clusters (such as GFS-B-1) of all centers based on the synchronization function of GFS. Subsequently, other participants can directly download this file or the file copy from the GFS cluster of this center. The schematic diagram is shown in Figure 6 。

[0063] 3) Participants upload the files to the local GFS (GFS-A-1) cluster of a certain center (Center A). Other participants download the original files from other centers (Center B). This center first attempts to download the files from the local GFS (GFS-B-1) cluster of that center (Center B). When the synchronization function of GFS fails, reading files from the local GFS cluster (GFS-B-1) will fail. This center will then attempt to read the files from other centers (Center A). After the GFS synchronization function is restored, the file will be synchronized to the GFS clusters of other centers. Subsequent participants can download the file from this center. For the schematic diagram, see Figure 7 .

[0064] In one embodiment, the above upload method for file processing when GFS fails may further include: when the main GFS cluster of the current data center is normal, after writing the original file to be stored into the main GFS cluster of the current data center, if it is detected that the synchronization function of the main GFS cluster of the current data center is abnormal and the copy of the file to be stored cannot be written to the secondary GFS cluster of any other data center, this center will then attempt to read the file from other centers (Center A), or the file to be stored can also be written to the normal main GFS cluster of any other data center. After writing the file to be stored into the normal main GFS cluster of any other data center, step 102 above is executed. Subsequently, when any data center receives a download file request for the file to be stored, the download file can be obtained from the current download data center.

[0065] In one embodiment, the above upload method for file processing when GFS fails may further include: when it is detected that the synchronization function of the main GFS cluster of the current data center is abnormal and the copy of the file to be stored cannot be written to the secondary GFS cluster of any other data center, a warning alarm is issued, which can enable the staff to handle the GFS failure in a timely manner and ensure the stable operation of the platform.

[0066] 4) When only one GFS cluster in any center is working properly, for example, the GFS-A-1 cluster in Center A is working properly and the GFS-A-2 cluster is abnormal, the GFS-B-1 cluster in Center B is working properly and the GFS-B-2 cluster is abnormal, and the GFS synchronization direction is from the GFS-A-1 cluster to the GFS-B-1 cluster. If the existing technical solution is adopted, Center B will not be able to provide the function of uploading files for participants. However, in the embodiments of the present invention, the platform can still provide basic file upload and download services: 1) Participants upload file a to Center A, and Center A stores file a in the GFS-A-1 cluster. Or when participants send a request to upload file b to Center B, and Center B fails to store file b in the GFS-B-2 cluster, Center B will store file b in the GFS-A-1 cluster; 2) The file synchronization function of GFS synchronizes the files on the GFS-A-1 cluster to the GFS-B-1 cluster; 3) When participants initiate a file download application to Center A, Center A can read the GFS-A-1 cluster of this center. If the reading fails, it can read the file from the GFS-B-1 cluster of Center B. When participants initiate a file download application to Center B, Center B can read the GFS-B-1 cluster of this center. If the reading fails, it can read the file from the GFS-A-1 cluster of Center A, see Figure 8 。

[0067] In addition, in one embodiment, the above upload method for file processing when GFS fails may further include: when it is detected that the main GFS cluster in the current data center is abnormal and the file to be stored cannot be written into the main GFS cluster of the current data center, write the copy of the file to be stored into the normal secondary GFS clusters of each data center, so that the file can be downloaded in any data center subsequently.

[0068] The technical solution provided by the embodiments of the present invention realizes:

[0069] 1) Regarding the GFS clusters in multiple centers as an overall cluster with a priority order;

[0070] 2) When storing and reading files, preferentially store them in or read them from the GFS cluster of this center;

[0071] 3) When the storage of the file to this center fails, retry storing it in the GFS clusters of other centers until a certain cluster succeeds or all fail;

[0072] 4) When the reading of the file from this center fails, retry reading the GFS clusters of other centers based on the synchronization logic of GFS.

[0073] An embodiment of the present invention also provides a download method for file processing when GFS fails, as described in the following embodiments. Since the principle of solving problems by this download method is similar to that of the upload method for file processing when GFS fails, the implementation of this download method can refer to the implementation of the upload method for file processing when GFS fails, and the repeated parts will not be elaborated.

[0074] Figure 10 It is a schematic flowchart of the download method for file processing when GFS fails in an embodiment of the present invention. As Figure 10 shown, GFS includes multiple data centers, and each data center includes at least two GFS clusters. One of the GFS clusters is the main GFS cluster for storing the original file, and the other clusters are the secondary GFS clusters for storing file copies. The method includes the following steps:

[0075] Step 201: When a request to download a file is received in any data center and it is detected that the current download data center is the data center where the original is written, obtain the downloaded file from the main GFS cluster or the secondary GFS cluster of the data center where the original is written; wherein, the data center where the original is written is the data center that receives a file upload request at the original file upload center, and the main GFS cluster of the original file upload center is abnormal and cannot write the file to be stored into the main GFS cluster of the original file upload center (as Figure 6 shown), the data center where the original of the file to be stored is written; the GFS synchronization function of the normal main GFS cluster of the data center where the original is written synchronizes the file copy to be stored to the secondary GFS clusters of each data center.

[0076] Step 202: When a request to download a file is received in any data center and it is detected that the current download data center is other than the data center where the original is written, obtain the downloaded file from the secondary GFS cluster of the data center other than the data center where the original is written.

[0077] In one embodiment, the above upload method for file processing when GFS fails may further include: as Figure 7 shown, after the original of the file to be stored is written into the main GFS cluster of the current data center due to the normal main GFS cluster of the original file upload center, and it is detected that the synchronization function of the main GFS cluster of the current data center is abnormal and cannot write the file copy to be stored into the secondary GFS clusters of any other data center, resulting in the inability to obtain the downloaded file from the secondary GFS cluster of the current download data center, obtain the downloaded file from the main GFS cluster of the original file upload center.

[0078] In one embodiment, as Figure 8As shown, the above-mentioned file processing upload method in case of GFS failure may further include: when the main GFS cluster of the original file upload center is abnormal and the file to be stored cannot be written into the main GFS cluster of the original file upload center, resulting in the inability to obtain the file to be downloaded from the main GFS cluster of the current download data center, obtaining the file to be downloaded from the secondary GFS cluster of the current download data center.

[0079] In one embodiment, when reading a file, it is preferred to read from the GFS cluster of the current download data center.

[0080] In an embodiment of the present invention, there is also provided an upload device for file processing in case of GFS failure, as described in the following embodiments. Since the principle of the upload device for solving problems is similar to the file processing upload method in case of GFS failure, the implementation of the upload device can refer to the implementation of the file processing upload method in case of GFS failure, and the repeated parts will not be elaborated.

[0081] Figure 11 It is a schematic structural diagram of the upload device for file processing in case of GFS failure in an embodiment of the present invention. GFS includes multiple data centers, and each data center includes at least two GFS clusters. One of the GFS clusters is the main GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies, as Figure 11 shown, the upload device includes:

[0082] A writing unit 011, configured to, when any data center receives a file upload request and the main GFS cluster of the current data center is abnormal and the file to be stored cannot be written into the main GFS cluster of the current data center, write the original file to be stored into the normal main GFS cluster of any other data center;

[0083] A synchronization unit 012, configured to use the GFS synchronization function of the normal main GFS cluster of the data center where the original is written to synchronize the file copy to be stored to the secondary GFS cluster of each data center, so that when any data center receives a download file request for the file to be stored later, the download file can be obtained from the current download data center.

[0084] In one embodiment, the writing unit is further configured to: after the original file to be stored is written into the main GFS cluster of the current data center when the main GFS cluster of the current data center is normal, and it is detected that the synchronization function of the main GFS cluster of the current data center is abnormal and the file copy to be stored cannot be written into the secondary GFS cluster of any other data center, write the file to be stored into the normal main GFS cluster of any other data center.

[0085] In one embodiment, the uploading device for file processing when the GFS fails may further include: a warning unit, configured to issue a warning alarm when it detects that the synchronization function of the primary GFS cluster in the current data center is abnormal and the copy of the file to be stored cannot be written into any secondary GFS cluster in other data centers.

[0086] In one embodiment, the writing unit is further configured to: when it detects that the primary GFS cluster in the current data center is abnormal and the file to be stored cannot be written into the primary GFS cluster in the current data center, write the copy of the file to be stored into the normal secondary GFS clusters of each data center.

[0087] In one embodiment, when writing the file to be stored, the file to be stored is preferentially stored in the GFS cluster of the current data center.

[0088] An embodiment of the present invention also provides a downloading device for file processing when the GFS fails, as described in the following embodiments. Since the principle of the downloading device for solving problems is similar to the uploading method for file processing when the GFS fails, the implementation of the downloading device can refer to the implementation of the uploading method for file processing when the GFS fails, and the repeated parts will not be described again.

[0089] Figure 12 FIG. is a schematic structural diagram of the downloading device for file processing when the GFS fails in an embodiment of the present invention. The GFS includes multiple data centers, and each data center includes at least two GFS clusters. One of the GFS clusters is the primary GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies, as Figure 12 shown, the downloading device includes:

[0090] A first downloading unit 021, configured to, when a request for downloading a file is received in any data center and it is detected that the current downloading data center is the data center where the original file is written, obtain the downloading file from the primary GFS cluster or the secondary GFS cluster of the data center where the original file is written; wherein, the data center where the original file is written is the data center where the copy of the file to be stored is written when a file uploading request is received in the original file uploading center and the primary GFS cluster of the original file uploading center is abnormal and the file to be stored cannot be written into the primary GFS cluster of the original file uploading center; the GFS synchronization function of the normal primary GFS cluster of the data center where the original file is written synchronizes the copy of the file to be stored to the secondary GFS clusters of each data center;

[0091] A second downloading unit 022, configured to, when a request for downloading a file is received in any data center and it is detected that the current downloading data center is other than the data center where the original file is written, obtain the downloading file from the secondary GFS clusters of the data centers other than the data center where the original file is written.

[0092] In one embodiment, the download device for file processing when GFS fails may further include: a third download unit, configured to, after writing the original file to be stored to the main GFS cluster of the current data center because the main GFS cluster of the original file upload center is normal, detect that the synchronization function of the main GFS cluster of the current data center is abnormal and it is impossible to write the copy of the file to be stored to the secondary GFS cluster of any other data center, resulting in the inability to obtain the downloaded file from the secondary GFS cluster of the current download data center, obtain the downloaded file from the main GFS cluster of the original file upload center.

[0093] In one embodiment, the download device for file processing when GFS fails may further include: a fourth download unit, configured to, when the main GFS cluster of the original file upload center is abnormal and it is impossible to write the file to be stored to the main GFS cluster of the original file upload center, resulting in the inability to obtain the file to be downloaded from the main GFS cluster of the current download data center, obtain the file to be downloaded from the secondary GFS cluster of the current download data center.

[0094] In one embodiment, when reading a file, it is preferred to read from the GFS cluster of the current download data center.

[0095] In an embodiment of the present invention, a system for file processing when GFS fails is also provided as described in the following embodiments. Since the principle of the system for solving problems is similar to the upload method for file processing when GFS fails, the implementation of the system can refer to the implementation of the upload method for file processing when GFS fails, and the repeated parts will not be described again.

[0096] Figure 13 It is a schematic structural diagram of the system for file processing when GFS fails in an embodiment of the present invention, as Figure 13 shown, the system includes: the upload device 01 as described above and the download device 02 as described above.

[0097] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for file processing when GFS fails as described above is implemented.

[0098] An embodiment of the present invention also provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for file processing when GFS fails as described above is implemented.

[0099] An embodiment of the present invention also provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, the method for file processing when GFS fails as described above is implemented.

[0100] In the file processing solution when GFS fails provided by the embodiments of the present invention, the distributed file system GFS includes multiple data centers, and each data center includes at least two GFS clusters. One of the GFS clusters is the primary GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. When working: when a file upload request is received in any data center and the primary GFS cluster in the current data center is abnormal and cannot write the file to be stored into the primary GFS cluster in the current data center, write the original file to be stored into the normal primary GFS cluster in any other data center; use the GFS synchronization function of the normal primary GFS cluster in the data center where the original is written to synchronize the file copy to be stored to the secondary GFS clusters in each data center. The embodiments of the present invention can ensure the success of file upload to the greatest extent, and subsequently when a file download request is received in any data center, the file can be obtained from the current download data center. Specifically, the downloaded file can be obtained from the primary GFS cluster or the secondary GFS cluster in the data center where the original is written, or from the secondary GFS clusters in the data centers other than the data center where the original is written. Therefore, the time consumption, network traffic, and possible failure probability caused by cross-center download are reduced.

[0101] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, system, or computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the function specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0103] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the process inFigure 1 one process or multiple processes and / or blocks Figure 1 the functions specified in one block or multiple blocks.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing the steps of the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0105] In the specific embodiments described above, the purpose, technical solutions and beneficial effects of the present invention have been further described in detail. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for file processing when GFS fails, characterized in that, the distributed file system GFS includes multiple data centers, each data center includes at least two GFS clusters, one of the GFS clusters is the primary GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. The method for file processing when GFS fails includes: When a file upload request is received in any data center and the primary GFS cluster in the current data center is abnormal and cannot write the file to be stored into the primary GFS cluster in the current data center, write the original file to be stored into the normal primary GFS cluster in any other data center; Use the GFS synchronization function of the normal primary GFS cluster in the data center where the original is written to synchronize the file copy to be stored to the secondary GFS clusters in each data center, so that when a download file request for the file to be stored is received in any data center later, the downloaded file can be obtained from the current download data center.

2. The method according to claim 1, characterized in that, it further includes: After the primary GFS cluster in the current data center is normal and the original file to be stored is written into the primary GFS cluster in the current data center, when it is detected that the synchronization function of the primary GFS cluster in the current data center is abnormal and cannot write the file copy to be stored into the secondary GFS clusters in any other data center, write the file to be stored into the normal primary GFS cluster in any other data center.

3. The method according to claim 2, characterized in that, it further includes: When it is detected that the synchronization function of the primary GFS cluster in the current data center is abnormal and cannot write the file copy to be stored into the secondary GFS clusters in any other data center, issue a warning alarm.

4. The method according to claim 1, characterized in that, it further includes: When it is detected that the primary GFS cluster in the current data center is abnormal and cannot write the file to be stored into the primary GFS cluster in the current data center, write the file copy to be stored into the normal secondary GFS clusters in each data center.

5. The method according to claim 1, characterized in that, When writing the file to be stored, preferentially store the file to be stored in the GFS cluster in the current data center.

6. A method for file processing when GFS fails, characterized in that, GFS includes multiple data centers, each data center includes at least two GFS clusters, one of the GFS clusters is the primary GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. The method for file processing when GFS fails includes: When a request to download a file is received at any data center and it is detected that the current download data center is the data center where the original is written, the downloaded file is obtained from the primary GFS cluster or the secondary GFS cluster of the data center where the original is written; wherein, the data center where the original is written is the data center where the original of the file to be stored is written when a file upload request is received at the original file upload center and the primary GFS cluster of the original file upload center is abnormal and unable to write the file to be stored to the primary GFS cluster of the original file upload center; the GFS synchronization function of the normal primary GFS cluster of the data center where the original is written synchronizes the copy of the file to be stored to the secondary GFS cluster of each data center. When a request to download a file is received at any data center and it is detected that the current download data center is other than the data center where the original is written, the downloaded file is obtained from the secondary GFS cluster of the data center other than the data center where the original is written.

7. The method according to claim 6, characterized in that, further comprising: After the original of the file to be stored is written to the primary GFS cluster of the current data center due to the normal primary GFS cluster of the original file upload center, if it is detected that the synchronization function of the primary GFS cluster of the current data center is abnormal and unable to write the copy of the file to be stored to the secondary GFS cluster of any other data center, resulting in the inability to obtain the downloaded file from the secondary GFS cluster of the current download data center, the downloaded file is obtained from the primary GFS cluster of the original file upload center.

8. The method according to claim 6, characterized in that, further comprising: When the primary GFS cluster of the original file upload center is abnormal and unable to write the file to be stored to the primary GFS cluster of the original file upload center, resulting in the inability to obtain the file to be downloaded from the primary GFS cluster of the current download data center, the file to be downloaded is obtained from the secondary GFS cluster of the current download data center.

9. The method according to claim 6, characterized in that, When reading a file, it is preferentially read from the GFS cluster of the current download data center.

10. An upload device for file processing when GFS fails, characterized in that, GFS includes multiple data centers, each data center includes at least two GFS clusters, one of which is the primary GFS cluster for storing the original of the file, and the other clusters are secondary GFS clusters for storing copies of the file, and the upload device includes: A writing unit, configured to write the original of the file to be stored to the normal primary GFS cluster of any other data center when a file upload request is received at any data center and the primary GFS cluster of the current data center is abnormal and unable to write the file to be stored to the primary GFS cluster of the current data center; A synchronization unit, configured to use the GFS synchronization function of the normal primary GFS cluster of the data center where the original is written to synchronize the copy of the file to be stored to the secondary GFS cluster of each data center, so that when a request to download a file for the file to be stored is received at any data center later, the downloaded file is obtained from the current download data center.

11. A download device for file processing when GFS fails, characterized in that, GFS includes multiple data centers, each data center includes at least two GFS clusters, one of the GFS clusters is the primary GFS cluster for storing the original file, and the other clusters are secondary GFS clusters for storing file copies. The download device includes: A first download unit, configured to, when receiving a request to download a file in any data center and detecting that the current download data center is the data center where the original is written, obtain the downloaded file from the primary GFS cluster or the secondary GFS cluster of the data center where the original is written; wherein, the data center where the original is written is the data center where the original file of the to-be-stored file is written when the primary GFS cluster of the original file upload center is abnormal and cannot write the to-be-stored file into the primary GFS cluster of the original file upload center when receiving a file upload request; the GFS synchronization function of the normal primary GFS cluster of the data center where the original is written synchronizes the to-be-stored file copy to the secondary GFS clusters of each data center; A second download unit, configured to, when receiving a request to download a file in any data center and detecting that the current download data center is other than the data center where the original is written, obtain the downloaded file from the secondary GFS clusters of the data centers other than the data center where the original is written.

12. A system for file processing when GFS fails, characterized in that, it includes: The upload device according to claim 10 and the download device according to claim 11.

13. A computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the method according to any one of claims 1 to 9.

14. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 9.

15. A computer program product, characterized in that, the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 9.

Images