File processing method, server, and storage medium

Abstract

The application provides a file processing method, a server and a storage medium, and relates to the distributed field. The method comprises the following steps: reading a queue element in a transmission queue, the queue element being identification information of a to-be-processed file; establishing a communication connection of a second server corresponding to the transmission queue, and sending a first verification request to the second server, so as to instruct the second server to verify whether a transmission thread for the to-be-processed file exists in the second server and to feed back a verification result; receiving the verification result fed back by the second server, and if it is determined that the verification result represents that the transmission thread for the to-be-processed file does not exist in the second server, obtaining the to-be-processed file in a file copy directory, and transmitting the to-be-processed file to the second server. The method can be universally arranged in a distributed system, is not limited by a computer room and a region, can widely realize sharing and synchronization of files in the distributed system, and widens an application range of batch job services.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distribution, and in particular to a file processing method, a server and a storage medium. BACKGROUND

[0002] In a distributed service system, all services must be deployed in a multi-active manner to ensure basic high availability of application services. For batch job services, a distributed lock can be used to ensure that only one thread executes the same job at the same time, thereby achieving multi-active deployment of batch job services. However, many batch jobs are executed based on files, which requires the sharing and synchronization of files in a distributed system.

[0003] In the prior art, the multiple servers on which the batch job services are deployed are configured as shared storage, and the files commonly used are operated based on the shared storage disks.

[0004] However, in the prior art, the file sharing storage method only supports servers in the same machine room, does not support high-availability deployment across machine rooms and regions, and cannot widely achieve the sharing and synchronization of files, resulting in a great limitation on the application of batch job services. SUMMARY

[0005] The present application provides a file processing method, a server and a storage medium to solve the problem of limited application of batch job services.

[0006] In a first aspect, the present application provides a file processing method, which is applied to a first server, and the method comprises:

[0007] reading a queue element in a transmission queue, wherein the transmission queue corresponds to a second server that performs file transmission with the first server, and the queue element is identification information of a to-be-processed file;

[0008] establishing a communication connection with the second server corresponding to the transmission queue, and sending a first verification request to the second server, wherein the first verification request is used to instruct the second server to verify whether there is a transmission thread for the to-be-processed file in the second server and to feed back a verification result;

[0009] receiving the verification result fed back by the second server, and if it is determined that the verification result represents that there is no transmission thread for the to-be-processed file in the second server, obtaining the to-be-processed file in a file replication directory, and transmitting the to-be-processed file to the second server.

[0010] In an optional implementation, obtaining the to-be-processed file in the file replication directory and transmitting the to-be-processed file to the second server comprises:

[0011]

[0011] sequentially obtaining file data of each batch of the to-be-processed files in the file copy directory, and sequentially transmitting the file data of each batch to the second server until the file data of N batches of the to-be-processed files are all transmitted to the second server; wherein N is a positive integer greater than 1.

[0012] In an optional implementation, after the file data of N batches of the to-be-processed files are all transmitted to the second server, the method further includes:

[0013] sending prompt information to the second server, removing the queue element in the transmission queue, and disconnecting the connection with the second server; wherein the prompt information is used to prompt the second server that the transmission of the to-be-processed files is completed.

[0014] In an optional implementation, the method further includes:

[0015] In response to an interface request sent by a batch job service process, obtaining the identification information of the to-be-processed files under the file copy directory, and adding the identification information as a queue element in the transmission queue; wherein the interface request is used to instruct to process the to-be-processed files.

[0016] In an optional implementation, the method further includes:

[0017] based on a timing thread, timing sending a second verification request to the second server, and receiving return information fed back by the second server; wherein the second verification request includes identification information corresponding to each file in the file copy directory, and the second verification request is used to instruct the second server to verify whether the file corresponding to each identification information exists in the second server;

[0018] adding the identification information corresponding to the to-be-processed file in the return information as a queue element in the transmission queue; wherein the return information is used to indicate the identification information corresponding to the to-be-processed file, and the to-be-processed file is a file existing in the file copy directory and not existing in the second server.

[0019] In a second aspect, the present application provides a file processing method, the method is applied to a second server, and the method includes:

[0020] receiving a first verification request sent by a first server, wherein the first server is a server performing file transmission with the second server, and the first verification request is used to instruct the second server to verify whether a transmission thread for a to-be-processed file exists in the second server and feed back a verification result;

[0021] reading the queue element carried by the first verification request, and determining that the queue element is a verification keyword, wherein the queue element is identification information of the to-be-processed file;

[0022] According to the verification keyword, performing verification processing on each transmission thread running in the second server, determining the verification result, and feeding back the verification result to the first server; wherein each transmission thread has identification information of a file being transmitted, and the verification result is used to represent whether the second server has a transmission thread for the to-be-processed file or not.

[0023] If the verification result represents that the second server does not have a transmission thread for the to-be-processed file, receiving the to-be-processed file transmitted by the first server.

[0024] In an optional implementation, receiving the to-be-processed file transmitted by the first server comprises:

[0025] starting a target transmission thread for the to-be-processed file;

[0026] receiving each batch of file data of the to-be-processed file transmitted by the first server based on the target transmission thread, and sequentially writing each batch of file data into a file directory corresponding to the to-be-processed file until N batches of file data of the to-be-processed file are all transmitted to the second server; wherein N is a positive integer greater than 1.

[0027] In an optional implementation, the method further comprises:

[0028] receiving prompt information sent by the first server, and removing the target transmission thread; wherein the prompt information is used to prompt that the second server has completed transmission for the to-be-processed file.

[0029] In a third aspect, the present application provides a server comprising a memory and a processor;

[0030] The memory is configured to store a computer program.

[0031] The processor is configured to read the computer program stored in the memory, and execute the file processing method according to the computer program in the memory.

[0032] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the file processing method according to the first aspect or the file processing method according to the second aspect is implemented.

[0033] The file processing method, server and storage medium provided by the present application, through the following steps: reading the queue element in the transmission queue; wherein the transmission queue corresponds to the second server for file transmission with the first server, and the queue element is the identification information of the to-be-processed file; establishing a communication connection with the second server corresponding to the transmission queue, and sending a first verification request to the second server, instructing the second server to verify whether there is a transmission thread for the to-be-processed file in the second server and feedback the verification result; receiving the verification result fed back by the second server, if it is determined that the verification result represents that there is no transmission thread for the to-be-processed file in the second server, obtaining the to-be-processed file in the file copy directory, and transmitting the to-be-processed file to the second server. The method can be universally set in a distributed system, is not limited by the computer room and the region, can widely realize the sharing and synchronization of the file in the distributed system, and widens the application range of the batch job service. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0035] Figure 1 A flowchart of a file processing method provided for an embodiment of the present application;

[0036] Figure 2 A flowchart of another file processing method provided for an embodiment of the present application;

[0037] Figure 3 A flowchart of still another file processing method provided for an embodiment of the present application;

[0038] Figure 4 A structural schematic diagram of a file processing device provided for an embodiment of the present application;

[0039] Figure 5 A structural schematic diagram of another file processing device provided for an embodiment of the present application;

[0040] Figure 6 A structural schematic diagram of a server provided for an embodiment of the present application;

[0041] Figure 7 A block diagram of a server provided for an embodiment of the present application.

[0042] Through the above-mentioned drawings, the specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0043] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description of exemplary embodiments is not intended to represent all embodiments in accordance with the present application. Rather, they are merely examples of apparatus and methods in accordance with aspects of the present application as detailed in the appended claims.

[0044] It should be noted that the file processing method, device, equipment and storage medium provided by the present application can be used in the distributed field, and can also be used in any field other than the distributed field. The application field of the file processing method, server and storage medium provided by the present application is not limited.

[0045] In a distributed service system, all services must be deployed in a multi-active manner to ensure basic high availability of application services. Batch job services also need to meet this basic requirement. For batch job services, a distributed lock can be used to ensure that only one thread executes the same job at the same time, thereby achieving multi-active deployment of batch jobs. However, many batch jobs are based on files, and the file system cannot be synchronized in a multi-active manner. For example, a batch job includes two activities: file download and file parsing and storage. In a multi-server deployment, server A may execute the file download activity, and server B may execute the file parsing and storage activity. However, because servers A and B are independent file systems, when server B executes file parsing, it cannot find the file that has been downloaded by server A.

[0046] In an example, two servers deployed for batch jobs can be configured to share storage, and the files commonly used by the two servers can be operated based on shared storage disks.

[0047] However, shared storage can only be used in the same machine room and does not support high-availability deployment across machine rooms or regions. When a machine room fails, it cannot normally provide batch job services, and the file sharing storage method also has a high limit on the number of services, etc. This results in that file sharing and synchronization in a distributed system cannot be widely implemented, and the application of batch job services is also greatly limited.

[0048] Therefore, the present application provides a file processing method to solve the above technical problems.

[0049] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0050] Figure 1 A flowchart of a file processing method provided by an embodiment of the present application is shown in FIG. 1. The method is applied to a first server, and the method includes the following steps. Figure 1

[0051] 101. Reading a queue element in a transmission queue, wherein the transmission queue corresponds to a second server that can perform file transmission with the first server, and the queue element is identification information of a to-be-processed file.

[0052] For example, the first server has one or more transmission queues, each of which corresponds to a server in a distributed system that can perform file transmission with the first server. The first server checks whether there is an unprocessed queue element in all transmission queues, and reads a queue element in a transmission queue, wherein the transmission queue corresponds to a second server that can perform file transmission with the first server, and the queue element in the transmission queue is identification information of a to-be-processed file. The identification information can be a file name or other unique file identification.

[0053] 102. Establishing a communication connection with the second server corresponding to the transmission queue, and sending a first verification request to the second server, wherein the first verification request is used to instruct the second server to verify whether there is a transmission thread for the to-be-processed file in the second server and to feed back a verification result.

[0054] For example, after reading the queue element in the transmission queue, the first server establishes a communication connection with the second server corresponding to the transmission queue, and sends a first verification request to the second server after establishing the connection, to instruct the second server to verify whether there is a transmission thread for the to-be-processed file in the second server and to feed back a verification result, so as to ensure that no other server is also copying and transmitting the same file to the second server.

[0055] In one example, after receiving the first verification request, the second server checks whether there is a transmission thread for the to-be-processed file in the second server, for example, checks whether a file with the same identification information is being transmitted in all file copying threads. If there is no to-be-processed file being transmitted, the second server adds one to the number of file copying threads and returns a successful verification result to the first server, indicating that there is no transmission thread for the to-be-processed file in the second server. If there is a file being transmitted, the second server returns a failed verification result to the first server, indicating that there is a transmission thread for the to-be-processed file in the second server.

[0056] 103. Receiving the verification result fed back by the second server, and if it is determined that the verification result indicates that there is no transmission thread for the to-be-processed file in the second server, obtaining the to-be-processed file in a file copying directory and transmitting the to-be-processed file to the second server.​

[0057] Exemplarily, the first server receives the verification result fed back by the second server, and if it is determined that the verification result represents that there is no transmission thread for the to-be-processed file in the second server, the transmission of the to-be-processed file can be performed, the to-be-processed file in the file replication directory is acquired, and the to-be-processed file is transmitted to the second server, so as to realize file sharing between the first server and the second server about the to-be-processed file.

[0058] To sum up, the file processing method provided by the embodiment comprises the following steps: reading a queue element in a transmission queue; wherein the transmission queue corresponds to a second server for file transmission with a first server, and the queue element is identification information of a to-be-processed file; establishing a communication connection with the second server corresponding to the transmission queue, and sending a first verification request to the second server, wherein the first verification request is used to instruct the second server to verify whether there is a transmission thread for the to-be-processed file in the second server and feed back a verification result; receiving the verification result fed back by the second server, and if it is determined that the verification result represents that there is no transmission thread for the to-be-processed file in the second server, acquiring the to-be-processed file in a file replication directory, and transmitting the to-be-processed file to the second server. The method can be universally set in a distributed system and is not limited by a computer room or a region, and can widely realize file sharing and synchronization in the distributed system, thereby widening the application range of batch job services.

[0059] Figure 2 Another flowchart of a file processing method provided by the embodiment of the application is shown in FIG. 2, which is applied to a second server. As shown in FIG. 2, the method comprises the following steps. Figure 2

[0060] 201. Receiving a first verification request sent by a first server, wherein the first server is a server for file transmission with the second server, and the first verification request is used to instruct the second server to verify whether there is a transmission thread for a to-be-processed file in the second server and feed back a verification result.

[0061] Exemplarily, the second server receives the first verification request sent by the first server, wherein the first server is a server for file transmission with the second server, and the first verification request is used to instruct the second server to verify whether there is a transmission thread for a to-be-processed file in the second server and feed back a verification result.

[0062] 202. Reading a queue element carried by the first verification request, and determining that the queue element is a verification keyword, wherein the queue element is identification information of a to-be-processed file.

[0063] Exemplarily, the first verification request carries the queue element, which is identification information of a to-be-processed file, and the second server reads and determines that the queue element is a verification keyword. ​

[0064] 203、according to the verification keyword, performing verification processing on each transmission thread running in the second server, determining a verification result, and feeding back the verification result to the first server; wherein each transmission thread has identification information of a file being transmitted, and the verification result is used to represent whether the second server has a transmission thread for the to-be-processed file.

[0065] For example, each transmission thread has identification information of a file being transmitted, and the second server performs verification processing on each transmission thread running in the second server according to the verification keyword, for example, checks whether there is a transmission thread for the to-be-processed file corresponding to the identification information in all file copying threads, determines whether the second server has a transmission thread for the to-be-processed file, that is, the verification result, and feeds back the verification result to the first server.

[0066] 204、if the verification result represents that the second server does not have a transmission thread for the to-be-processed file, receiving the to-be-processed file transmitted by the first server.

[0067] For example, if the verification result represents that the second server does not have a transmission thread for the to-be-processed file, the first server performs transmission of the to-be-processed file after receiving the verification result fed back by the second server, and the second server receives the to-be-processed file transmitted by the first server.

[0068] In summary, the file processing method provided by the embodiment includes the following steps: receiving a first verification request sent by a first server, reading a queue element carried by the first verification request, and determining that the queue element is a verification keyword, wherein the queue element is identification information of a to-be-processed file; according to the verification keyword, performing verification processing on each transmission thread running in the second server, determining a verification result, and feeding back the verification result to the first server; wherein each transmission thread has identification information of a file being transmitted, and the verification result is used to represent whether the second server has a transmission thread for the to-be-processed file; if the verification result represents that the second server does not have a transmission thread for the to-be-processed file, receiving the to-be-processed file transmitted by the first server. This method can be used in a distributed system and is not limited by a computer room or a region, can be widely used for sharing and synchronization of files in a distributed system, and widens the application range of batch job services.

[0069] Figure 3 A flowchart of another file processing method provided by the embodiment of the application is shown in FIG. 4, and the method includes the following steps. Figure 3

[0070] ​301、the first server reads a queue element in a transmission queue; wherein the transmission queue corresponds to a second server which performs file transmission with the first server, and the queue element is identification information of a file to be processed.

[0071] Exemplarily, one or more transmission queues are configured in the first server, each transmission queue corresponds to a server which can perform file transmission with the first server, and a file replication agent process in the first server checks whether there is a queue element to be processed in all transmission queues. The file replication agent process reads a queue element in a transmission queue, the transmission queue corresponds to a second server which performs file transmission with the first server, and then the file can be transmitted to the second server according to the read queue element in the transmission queue. The queue element is identification information of a file to be processed, which can be a file name.

[0072] In one example, a file replication agent process is started on each server of a batch job in a distributed system, and the file replication agent process is used for automatic file replication and sharing. The file replication agent process needs to be started on each server of the batch job independently of the batch job service process. The file replication agent process includes the following basic configuration information: (1) server list: an IP list of all servers on which the file replication agent process is deployed, indicating the number of servers which need to perform file synchronization and replication; (2) file synchronization listening port: used to realize file transmission between servers through a socket mode; (3) client interaction listening port: used for communication between a client and the file replication agent, realizing an interface function, wherein the client refers to a batch job service of the server; (4) timing check listening port: used for a timing check module to check files on other servers on which file replication agents are located; (5) file replication directory: a directory in which files need to be replicated, not all files in a directory on a server need to be replicated, only files in a specified directory configured to be replicated; (6) file replication temporary directory: different from the file replication directory, but needs to be in the same disk symbol. In order to ensure the integrity of the file, when the file is generated, the file is written in the temporary directory, and when the file writing is completed, the file is transferred to the file replication directory through renaming; (7) number of file replication threads: records the number of threads currently performing file transmission on the server, used to check whether there is a file in the process of transmission. The number is set to 0 at initialization.

[0073] In one example, when the file copy process is started, multiple transmission queues are generated according to the configured server list. For example, the configured server list is: a first server, a second server, and a third server, and the local machine is the first server, then two transmission queues of the first server-the second server and the first server-the third server are generated, wherein the only attribute of the queue element is "file name", and the queue element is removed from the queue only after the file copy is completed, so that when the queue element is put into the queue, it can be ensured that the same file will not be duplicated. The file copy agent process checks whether there is an element (file to be transmitted) to be processed in all file copy queues (multiple), and if there is, the queue element is read at this time, and the queue element is only read and not removed. Then the next step of file copy transmission is performed, and the next step of file transmission can be implemented in an asynchronous multithreading manner to improve the processing performance when multiple files need to be copied and transmitted. For example, when there are 3 elements to be copied and transmitted, they can be processed in parallel at the same time, without waiting for the previous file copy to be completed before processing the next one.

[0074] 302. The first server establishes a communication connection with the second server corresponding to the transmission queue, and sends a first verification request to the second server, wherein the first verification request is used to instruct the second server to verify whether there is a transmission thread for the to-be-processed file in the second server and feed back the verification result.

[0075] Illustratively, after reading the queue element in the transmission queue, the file copy agent process establishes a connection with the second server based on the "file synchronization listening port", and after establishing the connection with the second server, sends a first verification request to the second server, instructing the second server to verify whether there is a transmission thread for the to-be-processed file in the second server and feed back the verification result, to ensure that no other server is also copying and transmitting the same file to the second server.

[0076] 303. The second server receives the first verification request sent by the first server, wherein the first server is a server for file transmission with the second server, and the first verification request is used to instruct the second server to verify whether there is a transmission thread for the to-be-processed file in the second server and feed back the verification result.

[0077] Illustratively, this step is described in step 201 and will not be repeated.

[0078] 304. The second server reads the queue element carried by the first verification request, and determines that the queue element is a verification keyword, wherein the queue element is identification information of the to-be-processed file.

[0079] Illustratively, this step is described in step 202 and will not be repeated.

[0080] 305、the second server performs verification processing on each transmission thread running in the second server according to the verification keyword to determine a verification result; each transmission thread has identification information of a file being transmitted, and the verification result is used to represent whether the second server has a transmission thread for the to-be-processed file or not.

[0081] By way of example, reference can be made to step 203, and no further elaboration is provided.

[0082] 306、the second server feeds back the verification result to the first server.

[0083] By way of example, reference can be made to step 203, and no further elaboration is provided.

[0084] 307、the first server receives the verification result fed back by the second server.

[0085] By way of example, the file replication agent process of the first server reads the verification result fed back by the second server, and if it is determined that the verification result represents that there is no transmission thread for the to-be-processed file in the second server, it is indicated that the transmission of the to-be-processed file can be performed.

[0086] 308、if it is determined that the verification result represents that there is no transmission thread for the to-be-processed file in the second server, the first server sequentially acquires file data of each batch of the to-be-processed file in the file replication directory and sequentially transmits the file data of each batch to the second server until the file data of N batches of the to-be-processed file are all transmitted to the second server; where N is a positive integer greater than 1.

[0087] By way of example, after receiving the verification result fed back by the second server, if it is determined that the verification result represents that there is no transmission thread for the to-be-processed file in the second server, the file replication agent process of the first server can perform batch transmission of the to-be-processed file based on the file replication module in the file replication agent process: sequentially acquire file data of each batch of the to-be-processed file in the file replication directory and sequentially transmit the file data of each batch to the second server until the file data of N batches of the to-be-processed file are all transmitted to the second server; where N is a positive integer greater than 1.

[0088] In one example, the first server receives the verification result fed back by the second server, and if it is determined that the verification result represents that there is already a transmission thread for the to-be-processed file in the second server, the file replication agent process removes the queue element corresponding to the to-be-processed file, i.e., the identification information of the to-be-processed file, in the transmission queue and closes the connection with the second server.

[0089] 309、the second server starts a target transmission thread for the to-be-processed file.

[0090] Exemplarily, after determining that there is no transmission thread for the to-be-processed file in the second server, the second server starts a target transmission thread for the to-be-processed file, so as to receive the to-be-processed file based on the target transmission thread.

[0091] 310. The second server receives each batch of file data of the to-be-processed file transmitted by the first server based on the target transmission thread, and writes each batch of file data into the file directory corresponding to the to-be-processed file in sequence, until N batches of file data of the to-be-processed file are all transmitted to the second server; wherein N is a positive integer greater than 1.

[0092] Exemplarily, the second server receives each batch of file data of the to-be-processed file transmitted by the first server based on the target transmission thread, and writes each batch of file data into the file directory corresponding to the to-be-processed file in sequence, until N batches of file data of the to-be-processed file are all transmitted to the second server.

[0093] In one example, when the second server receives the file data of the to-be-processed file for the first time, it creates a file with the same name as the to-be-processed file under the temporary directory, and writes the received file data into the file with the same name corresponding to the to-be-processed file under the temporary directory in an appended manner. When all the file data has been written into the file under the temporary directory, the second server transfers the file with the same name corresponding to the to-be-processed file under the temporary directory to the formal file copy directory in a renamed manner.

[0094] 311. The first server sends prompt information to the second server; wherein the prompt information is used to prompt the second server that the transmission of the to-be-processed file is completed.

[0095] Exemplarily, after the transmission of the to-be-processed file is completed, the file copy agent process in the first server sends prompt information to the second server, prompting the second server that the transmission of the to-be-processed file is completed.

[0096] 312. The first server removes the queue element in the transmission queue and disconnects the connection with the second server.

[0097] Exemplarily, after the transmission of the to-be-processed file is completed and the prompt information is sent to the second server, the file copy agent process in the first server removes the queue element in the transmission queue and actively disconnects the connection with the second server.

[0098] 313. The second server receives the prompt information sent by the first server and removes the target transmission thread; wherein the prompt information is used to prompt the second server that the transmission of the to-be-processed file is completed.

[0099] Exemplarily, the second server receives the prompt information sent by the first server, removes the target transmission thread after determining that the transmission of the to-be-processed file is completed.

[0100] After the data of the to-be-processed file is completely transmitted, the first server initiatively closes the connection; after the second server receives the connection closing request and disconnects, the temporary directory file reading and writing operation is closed, at this time, all file data has been written into the file under the temporary directory, the second server transfers the same-named file corresponding to the to-be-processed file under the temporary directory to the formal file copy directory in the form of renaming. The target transmission thread for the to-be-processed file is removed, and the recorded transmission thread number is reduced by one.

[0101] To sum up, the file processing method provided by the embodiment realizes automatic file copying between multiple servers through the pre-configured file copy agent process, supports deployment of multiple batch job services, each batch service is to operate the file system of the server, and there is no any influence between them; the transmission of the to-be-processed file is realized based on the socket communication mode, the transmission mode supports cross-machine room and cross-territory file copying, and does not depend on the file transfer protocol (FTP) or the SSH file transfer protocol (sftp) file transfer authentication; in the transmission process of the to-be-processed file, the file data is read and transmitted multiple times, instead of reading all contents of the file, so that memory overflow caused by too large file is avoided. The method can be universally set in a distributed system and is not limited by machine room and region, can widely realize file sharing and synchronization in a distributed system, and widens the application range of the batch job service.

[0102] One or more embodiments of the present application can also include:

[0103] The first server acquires the identification information of the to-be-processed file under the file copy directory in response to the interface request sent by the batch job service process, and adds the identification information as a queue element in the transmission queue; wherein the interface request is used to indicate processing of the to-be-processed file.

[0104] Exemplarily, the first server acquires the identification information of the to-be-processed file under the file copy directory in response to the interface request sent by the batch job service process, and adds the identification information as a queue element in the transmission queue; wherein the interface request is used to indicate processing of the to-be-processed file.

[0105] In one example, the batch job service in the first server, when performing file class operations, interacts with the file replication agent of the server, and the application programming interface (API) in the software development kit (SDK) client is called, and the "client interaction listening port" is used internally to interact with the file replication agent process. When the batch job service writes a file, it needs to write the file to a temporary file directory, rename the file after all write operations are completed, and then call the API interface to notify the file replication agent. After the file replication agent receives the API request, it places the file name in all transmission queues and waits for file replication transmission. For example, there are two transmission queues: the first server-second server and the first server-third server. The identification information of the file is placed in these two queues as queue elements, indicating that the file on the first server needs to be copied and transmitted to the second server and the third server. This method can maximize the timeliness of file replication transmission.

[0106] In one example, when the batch job service reads a file, it needs to first send a check request containing the identification information of the file, such as the file name, to the file replication agent to query whether the same-named file is being transmitted. After receiving the request, the file replication agent can retrieve whether the file replication transmission is being performed. Since the file replication agent process records all running transmission threads and the file names being transmitted, it can check whether the specified file name is being replicated and transmitted, and return the check result. When checking, the file replication agent can first check the number of current file replication transmission threads. If it is equal to 0, it means that no file replication transmission is running. If it is greater than 0, it needs to retrieve whether all running file replication transmission threads contain the current file name. After receiving the return, if the file being read is being replicated and transmitted, the batch job service needs to wait a few seconds and then initiate the check request again until it confirms that the current file name is not being replicated and transmitted, and then reads the file, which can ensure the validity of reading the file.

[0107] In one example, the file replication and timing check module are implemented in the file replication agent process, which needs to be started separately on each batch job server and is independent of the batch job service process. The client interaction module is provided for the batch job service using the SDK client, which is embedded in the batch job service to implement the API interaction between the batch job service and the file replication agent.

[0108] In summary, in the embodiment, when the batch service completes the file writing operation, a message can be sent to the file replication agent process of the server, at this time, the file replication agent process pushes the file to all servers deployed with the file replication agent process in the form of queue processing, so as to achieve the purpose of timely file replication.

[0109] One or more embodiments of the application can also include:

[0110] The first server sends a second verification request to the second server based on a timing thread, and receives return information fed back by the second server; wherein the second verification request includes identification information corresponding to each file included in the file replication directory, and the second verification request is used to instruct the second server to verify whether the file corresponding to each identification information exists in the second server.

[0111] The first server adds the identification information corresponding to the to-be-processed file in the return information as a queue element in the transmission queue; wherein the return information is used to indicate the identification information corresponding to the to-be-processed file, and the to-be-processed file is a file existing in the file replication directory and not existing in the second server.

[0112] In summary, the file replication agent process also scans all files under the file replication directory at a timing, and verifies with all servers deployed with the file replication agent process, and performs directional pushing for the server lacking the file. Based on a timing thread, a second verification request is sent to the second server, wherein the second verification request includes identification information corresponding to each file included in the file replication directory of the first server, and the second verification request is used to instruct the second server to verify whether the file corresponding to each identification information exists in the second server. After receiving the second verification request, the second server checks the file in the file replication directory of the second server, determines the file existing in the file replication directory of the first server but not existing in the file replication directory of the second server as a to-be-processed file, and returns the identification information of the to-be-processed file to the first server. The file replication agent process of the first server receives the return information fed back by the second server, and adds the identification information corresponding to the to-be-processed file in the return information as a queue element in the transmission queue.

[0113] In one example, the timing check module in the file replication agent process is to prevent the problem that the batch job service cannot timely notify the file replication agent in abnormal conditions, resulting in failure to timely perform the replication transmission of the corresponding file. For example, the batch job service is stopped before sending the notification, or the batch job service server of the multi-active deployment is not started for a long time, etc. In such a case, there will be files that cannot be correctly replicated and transmitted to each server. The timing thread in the timing check module executes the check compensation function once every period of time, and the specific execution function is as follows:

[0114] Step 1: Obtain all files in the file replication directory of the server. The "timed check listening port" is used to communicate with the file replication agent of the other server, and the file list of the server is sent to the other server for checking. If the first server scans the file list of the server, it needs to be sent to the second server and the third server for checking respectively.

[0115] Step 2: After the file replication agent of the other server receives the check, the file name of all files in the file replication directory of the server is checked, and the file name that does not exist in the server is returned to the checking party. For example, after the second server receives the request sent by the first server, the file list sent by the first server is checked with all file names in the file replication directory of the second server, and the file name list that does not exist in the second server is returned to the first server.

[0116] Step 3: After the first server sending the check receives the return information, the file name in the return information is put into the corresponding transmission queue, and the file is waiting for transmission. For example, the first server receives the return information of the second server, and gets the file name that exists in the first server but does not exist in the second server, and then puts the file name into the transmission queue of the first server-second server. The same processing is also applied to the request sent by the first server to the third server and other servers.

[0117] In an example, for the availability of the file replication agent process, a daemon process for the file replication agent process is also provided in the electronic device.

[0118] In summary, in the embodiment, the timed check module can avoid the problem that the batch service cannot notify the agent in time due to an exception, so that the file cannot be copied, and the timeliness of file synchronization is improved.

[0119] Figure 4 A structural schematic diagram of a file processing device provided in the embodiment of the application is shown in FIG. 1. The device is deployed in a first server, as shown in FIG. 1, the device comprises: Figure 4

[0120] The reading unit 41 is configured to read the queue element in the transmission queue, wherein the transmission queue corresponds to a second server for file transmission of the first server, and the queue element is identification information of a to-be-processed file.

[0121] The first processing unit 42 is configured to establish a communication connection with the second server corresponding to the transmission queue, and send a first verification request to the second server, wherein the first verification request is used to instruct the second server to verify whether a transmission thread for the to-be-processed file exists in the second server and feed back a verification result. ​

[0122] The second processing unit 43 is configured to receive a verification result fed back by the second server, and if it is determined that the verification result indicates that there is no transmission thread for the to-be-processed file in the second server, acquire the to-be-processed file in the file copy directory, and transmit the to-be-processed file to the second server.

[0123] In one example, the second processing unit 43 comprises:

[0124] The transmission subunit is configured to sequentially acquire file data of each batch of to-be-processed files in the file copy directory, and sequentially transmit the file data of each batch to the second server until the file data of N batches of to-be-processed files are all transmitted to the second server; wherein N is a positive integer greater than 1.

[0125] In one example, the apparatus further comprises, after the transmission subunit:

[0126] The third processing unit is configured to send prompt information to the second server, remove the queue element in the transmission queue, and disconnect the connection with the second server; wherein the prompt information is used to prompt the second server that the transmission of the to-be-processed file is completed.

[0127] In one example, the apparatus further comprises:

[0128] The first execution unit is configured to, in response to an interface request sent by the batch job service process, acquire identification information of the to-be-processed file under the file copy directory, and add the identification information as a queue element in the transmission queue; wherein the interface request is used to instruct to process the to-be-processed file.

[0129] In one example, the apparatus further comprises:

[0130] The second execution unit is configured to, based on the timing thread, send a second verification request to the second server at a timing, and receive return information fed back by the second server; wherein the second verification request comprises identification information corresponding to each file in the file copy directory, and the second verification request is used to instruct the second server to verify whether the file corresponding to each identification information exists in the second server.

[0131] The third execution unit is configured to add the identification information corresponding to the to-be-processed file in the return information as a queue element in the transmission queue; wherein the return information is used to indicate the identification information corresponding to the to-be-processed file, and the to-be-processed file is a file existing in the file copy directory and not existing in the second server.

[0132] Figure 5 A structural schematic diagram of a file processing apparatus provided by an embodiment of the present application is shown in FIG. 1. The apparatus is deployed in a second server, as shown in FIG. 1, the apparatus comprises: Figure 5

[0133] ​The first receiving unit 51 is configured to receive a first verification request sent by a first server, wherein the first server is a server for file transmission with the second server, and the first verification request is used to instruct the second server to verify whether there is a transmission thread for a to-be-processed file in the second server and to feed back a verification result.

[0134] The obtaining unit 52 is configured to read a queue element carried by the first verification request and determine that the queue element is a verification keyword, wherein the queue element is identification information of the to-be-processed file.

[0135] The executing unit 53 is configured to perform verification processing on each transmission thread running in the second server according to the verification keyword, determine a verification result, and feed back the verification result to the first server, wherein each transmission thread has identification information of a file being transmitted, and the verification result is used to represent whether there is a transmission thread for the to-be-processed file in the second server.

[0136] The second receiving unit 54 is configured to receive the to-be-processed file transmitted by the first server if the verification result represents that there is no transmission thread for the to-be-processed file in the second server.

[0137] In one example, the second receiving unit 54 includes:

[0138] The starting sub-unit is configured to start a target transmission thread for the to-be-processed file.

[0139] The receiving sub-unit is configured to receive file data of each batch of the to-be-processed file transmitted by the first server based on the target transmission thread, and sequentially write the file data of each batch into a file directory corresponding to the to-be-processed file until N batches of file data of the to-be-processed file are all transmitted to the second server, wherein N is a positive integer greater than 1.

[0140] In one example, the apparatus further includes:

[0141] The third receiving unit is configured to receive prompt information sent by the first server and remove the target transmission thread, wherein the prompt information is used to prompt that the second server has completed transmission for the to-be-processed file.

[0142] Figure 6 A structural schematic diagram of a server provided by an embodiment of the present application is shown in FIG. 1. Figure 6 As shown in FIG. 1, the server includes a memory 61 and a processor 62.

[0143] The memory 61 is configured to store a computer program.

[0144] The processor 62 is configured to read the computer program stored in the memory and perform the file processing method of any one of the above embodiments according to the computer program in the memory.

[0145] Figure 7 A block diagram of a server is provided for embodiments of the present application, wherein the apparatus 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0146] The processing component 802 usually controls the overall operations of the apparatus 800, such as operations associated with displaying, making phone calls, conducting data communications, operating cameras and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the methods described above. In addition, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0147] The memory 804 is configured to store various types of data to support operations of the apparatus 800. Examples of these data include instructions for any applications or methods operating on the apparatus 800, contact data, phonebook data, messages, pictures, videos, and so on. The memory 804 can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic or optical disk.

[0148] The power supply component 806 supplies electrical power for the various components of the apparatus 800. The power supply component 806 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing electrical power for the apparatus 800.

[0149] The multimedia component 808 includes a screen providing an output interface between the device 800 and a user. In some embodiments, the screen includes a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensor can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0150] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) configured to receive an external audio signal when the device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0151] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0152] The sensor component 814 includes one or more sensors to provide various state assessments for the device 800. For example, the sensor component 814 can detect an open / closed state of the device 800, relative positioning of components, such as a display and a keypad of the device 800, a change in position of the device 800 or a component of the device 800, presence or absence of user contact with the device 800, a change in orientation of the device 800 or acceleration / deceleration of the device 800, and a temperature change of the device 800. The sensor component 814 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 814 can further include a light sensor, such as a CMOS or CCD image sensor, for use in an imaging application. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0153] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate close proximity communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technology.

[0154] In an exemplary embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components, for performing the above-described methods.

[0155] In an exemplary embodiment, a non-transitory computer readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the device 800 to complete the above-described methods. For example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0156] The embodiments of the present application further provide a computer program product. The computer program product includes a computer program stored in a readable storage medium. At least one processor of a server can read the computer program from the readable storage medium, and execute the computer program to enable the server to perform the solutions provided in any of the above-described embodiments.

[0157] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present application cover any and all variations of the application that come within the scope of the claims and their equivalents. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application indicated by the following claims.

[0158] It is to be understood that the application is not limited to the precise construction herein disclosed and shown in the drawings, and that various changes in form and detail can be made therein without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

Claims

1. A file processing method, characterized in that, The method is applied to a first server, and the method includes: Read queue elements from the transmission queue; wherein the transmission queue corresponds to a second server that performs file transmission with the first server, and the queue element is the identification information of the file to be processed; Establish a communication connection with the second server corresponding to the transmission queue, and send a first verification request to the second server, wherein the first verification request is used to instruct the second server to verify whether there is a transmission thread for the file to be processed in the second server and to return the verification result; If the verification result received from the second server indicates that there is no transmission thread for the file to be processed in the second server, then the file to be processed is obtained from the file copy directory and transmitted to the second server. The file copy directory is a designated directory configured to store files to be copied.

2. The method according to claim 1, characterized in that, Obtaining the file to be processed from the file copy directory and transferring the file to be processed to the second server includes: The system sequentially acquires each batch of file data of the files to be processed in the file copy directory, and sequentially transmits each batch of file data to the second server, until N batches of file data of the files to be processed have been transmitted to the second server; where N is a positive integer greater than 1.

3. The method according to claim 2, characterized in that, After N batches of file data for the files to be processed have been transmitted to the second server, the method further includes: A notification message is sent to the second server, and the queue element in the transmission queue is removed, disconnecting the connection with the second server; wherein, the notification message is used to notify the second server that the transmission of the file to be processed has been completed.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: In response to an interface request sent by the batch job service process, the identification information of the file to be processed in the file copy directory is obtained, and the identification information is added as a queue element in the transmission queue; wherein, the interface request is used to indicate the processing of the file to be processed.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: Based on a timed thread, a second verification request is periodically sent to the second server, and the return information from the second server is received; wherein, the second verification request includes identification information corresponding to each file included in the file copy directory, and the second verification request is used to instruct the second server to check whether the file corresponding to each identification information exists in the second server; The identifier information corresponding to the file to be processed in the returned information is added as a queue element in the transmission queue; wherein, the returned information is used to indicate the identifier information corresponding to the file to be processed, and the file to be processed is a file that exists in the file copy directory but does not exist in the second server.

6. A file processing method, characterized in that, The method is applied to a second server, and the method includes: Receive a first verification request sent by a first server, wherein the first server is a server that transfers files with the second server, and the first verification request is used to instruct the second server to verify whether there is a transfer thread for the file to be processed in the second server and to return the verification result. Read the queue element carried by the first verification request and determine that the queue element is a verification keyword, wherein the queue element is the identification information of the file to be processed, and the identification information is stored in the transmission queue in the first server corresponding to the file transmission with the second server; Based on the verification keywords, each transmission thread running in the second server is verified, the verification result is determined, and the verification result is fed back to the first server; wherein, each transmission thread has identification information of the file being transmitted, and the verification result is used to indicate whether the second server has a transmission thread for the file to be processed; If the verification result indicates that there is no transmission thread for the file to be processed in the second server, then the file to be processed transmitted by the first server is received.

7. The method according to claim 6, characterized in that, Receiving the file to be processed transmitted by the first server includes: Start the target transfer thread for the file to be processed; The target transmission thread receives each batch of file data of the file to be processed from the first server and writes each batch of file data into the file directory corresponding to the file to be processed in sequence, until N batches of file data of the file to be processed have been transmitted to the second server; where N is a positive integer greater than 1.

8. The method according to claim 7, characterized in that, The method further includes: The system receives a notification message from the first server and removes the target transmission thread; wherein the notification message is used to notify the second server that the transmission of the file to be processed has been completed.

9. A server, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to read the computer program stored in the memory and execute the file processing method according to any one of claims 1-5 or any one of claims 6-8 according to the computer program in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the file processing method as described in any one of claims 1-5 or any one of claims 6-8.

Images