File synchronization method and system, electronic equipment and storage medium

Abstract

The invention provides a file synchronization method and system, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps of obtaining a target number of first to-be-synchronized files; storing the first file to be synchronized into a temporary folder in an archiving directory; and uploading the first file to be synchronized in the temporary folder to a distributed file system. According to the file synchronization method and device, the target number of first to-be-synchronized files can be transferred to the archiving catalog, and then the files in the archiving catalog are synchronized, so that the problems of high disk IO utilization rate, memory overflow, interface timeout and the like caused by reading all the files at a time are avoided, and the timeliness and stability of file synchronization are improved.

Description

Background Art

[0002] There are many data transmission methods in the aviation field. For example, the meteorological center can push the weather message data to the airline's FTP (File Transfer Protocol) server. The airline's system will pull the weather message file from the FTP server to synchronize the weather message file.

[0003] However, the meteorological center will continuously push new files to the FTP server. If all files are read at once, it will cause high disk IO (Input / Output) usage, memory overflow, interface timeout and other problems, which will eventually affect the timeliness and stability of file synchronization.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a file synchronization method, system, electronic device and storage medium, which at least to a certain extent overcome the problem that reading all files at once in the related technology will cause high disk IO usage, memory overflow, interface timeout and other problems, which ultimately affect the timeliness and stability of file synchronization.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a file synchronization method is provided, comprising:

[0008] Get the target number of first files to be synchronized;

[0009] Storing the first file to be synchronized in a temporary folder in the archive directory;

[0010] The first file to be synchronized in the temporary folder is uploaded to the distributed file system.

[0011] In some exemplary embodiments, before acquiring the target number of first files to be synchronized, the file synchronization method provided by the embodiment of the present disclosure further includes:

[0012] Acquire configuration information, where the configuration information includes at least one of an original file path corresponding to the first file to be synchronized, an archive file path corresponding to the archive directory, and target quantity information;

[0013] The original file path is used to obtain the first file to be synchronized, and the archived file path is used to store the first file to be synchronized in a temporary folder in the archive directory.

[0014] In some exemplary embodiments, the configuration information further includes at least one of an address, a port, and login information of the target server. The file synchronization method provided by the embodiment of the present disclosure further includes:

[0015] Log in to the target server according to at least one of the address, port, and login information of the target server;

[0016] The step of obtaining a target number of first files to be synchronized includes:

[0017] A target number of first files to be synchronized are obtained from the target server.

[0018] In some exemplary embodiments, uploading the first file to be synchronized in the temporary folder to the distributed file system includes:

[0019] Performing format conversion on the first file to be synchronized in the temporary folder to obtain a data stream in a target format;

[0020] The data stream in the target format is uploaded to a distributed file system.

[0021] In some exemplary embodiments, the file synchronization method provided by the embodiments of the present disclosure further includes:

[0022] In response to satisfying a preset condition, iteratively acquiring a target number of second files to be synchronized, storing the second files to be synchronized in a temporary folder in the archive directory, and uploading the second files to be synchronized in the temporary folder to the distributed file system.

[0023] In some exemplary embodiments, before acquiring the target number of first files to be synchronized, the file synchronization method provided by the embodiment of the present disclosure further includes:

[0024] A locking operation is performed to implement a single read of the first file to be synchronized.

[0025] According to another aspect of the present disclosure, a file synchronization system is provided, characterized in that it includes:

[0026] The task scheduler is used to obtain a target number of first files to be synchronized; store the first files to be synchronized in a temporary folder in an archive directory; and upload the first files to be synchronized in the temporary folder to a distributed file system;

[0027] The business system is used to parse and / or store in a database the first file to be synchronized in the distributed file system.

[0028] According to another aspect of the present disclosure, there is also provided a file synchronization device, characterized in that it includes:

[0029] A file acquisition module, used to acquire a target number of first files to be synchronized;

[0030] A file storage module, used for storing the first file to be synchronized in a temporary folder in the archive directory;

[0031] The file uploading module is used to upload the first file to be synchronized in the temporary folder to the distributed file system.

[0032] In some exemplary embodiments, the file synchronization device provided by the embodiments of the present disclosure further includes:

[0033] A configuration information acquisition module, used to acquire configuration information, wherein the configuration information includes at least one of an original file path corresponding to the first file to be synchronized, an archive file path corresponding to the archive directory, and target quantity information;

[0034] The original file path is used to obtain the first file to be synchronized, and the archived file path is used to store the first file to be synchronized in a temporary folder in the archive directory.

[0035] In some exemplary embodiments, the configuration information further includes at least one of an address, a port, and login information of the target server. The file synchronization device provided by the embodiment of the present disclosure further includes:

[0036] A target server login module, used to log in to the target server according to at least one of the address, port, and login information of the target server;

[0037] The file acquisition module is used to acquire a target number of first files to be synchronized from the target server.

[0038] In some exemplary embodiments, the file upload module is used to convert the format of the first file to be synchronized in the temporary folder to obtain a data stream in a target format; and upload the data stream in the target format to a distributed file system.

[0039] In some exemplary embodiments, the file acquisition module is further used to iteratively acquire a target number of second files to be synchronized in response to satisfying a preset condition;

[0040] The file storage module is further used to store the second file to be synchronized in a temporary folder in the archive directory;

[0041] The file upload module is also used to upload the second file to be synchronized in the temporary folder to the distributed file system.

[0042] In some exemplary embodiments, the file synchronization device provided by the embodiments of the present disclosure further includes:

[0043] The locking module is used to perform a locking operation to achieve a single read of the first file to be synchronized.

[0044] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned file synchronization methods by executing the executable instructions.

[0045] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the file synchronization method described in any one of the above is implemented.

[0046] According to another aspect of the present disclosure, a computer program product is provided, including: a computer program or instructions, wherein when the computer program or instructions are executed by a processor, any one of the above-mentioned file synchronization methods is implemented.

[0047] The technical solution provided in the disclosed embodiment can first transfer the target number of first files to be synchronized to the archive directory, and then synchronize the files in the archive directory, thereby avoiding problems such as high disk IO usage, memory overflow, interface timeout, etc. caused by reading all files at one time, thereby improving the timeliness and stability of file synchronization.

[0048] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0050] Figure 1 A schematic diagram showing a system structure in an embodiment of the present disclosure;

[0051] Figure 2 A flow chart of a file synchronization method in an embodiment of the present disclosure is shown;

[0052] Figure 3 A schematic diagram of a file transfer process in an embodiment of the present disclosure is shown;

[0053] Figure 4 A schematic diagram of a core service in an embodiment of the present disclosure is shown;

[0054] Figure 5 A schematic diagram of a file synchronization process in an embodiment of the present disclosure is shown;

[0055] Figure 6 A schematic diagram showing user-defined configuration information in an embodiment of the present disclosure;

[0056] Figure 7 A schematic diagram of a file synchronization device according to an embodiment of the present disclosure is shown;

[0057] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0058] Fig. 9 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0059] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0060] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0061] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0062] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the file synchronization method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0063] Among them, the terminal device 101 can send the first file to be synchronized to the server 103. Then, the server 103 can obtain the target number of first files to be synchronized. Then, the server 103 can also store the first file to be synchronized in a temporary folder in the archive directory. Finally, the server 103 uploads the first file to be synchronized in the temporary folder to the distributed file system.

[0064] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0065] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0066] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0067] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0068] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0069] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0070] Those skilled in the art will know that Figure 1 The number of terminal devices 101, networks 102 and servers 103 is only illustrative, and any number of terminal devices 101, networks 102 and servers 103 may be provided according to actual needs. The embodiments of the present disclosure are not limited to this.

[0071] Under the above system architecture, a file synchronization method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.

[0072] In some embodiments, the file synchronization method provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the file synchronization method provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the file synchronization method provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0073] Figure 2 A flow chart of a file synchronization method in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the file synchronization method provided in the embodiment of the present disclosure includes the following steps S202 to S206.

[0074] S202: Acquire a target number of first files to be synchronized.

[0075] The disclosed embodiment does not limit the file content of the first file to be synchronized, and the file content of the first file to be synchronized can be determined according to the application scenario. For example, when the application scenario is an aviation scenario, the first file to be synchronized can be weather message data, heading notification message data, etc. When the application scenario is an e-commerce field, the first file to be synchronized can be a product catalog file, inventory information, order data, etc.

[0076] In some possible implementations, the first file to be synchronized may be a file transmitted based on FTP. In other possible implementations, the first file to be synchronized may also be a file transmitted based on protocols such as HTTP and HTTPS.

[0077] Exemplarily, the target number can be used to describe the number of files synchronized at a time. The embodiment of the present disclosure does not limit the target number, and the target number can be set according to experience or application scenarios, for example, the target number can be 5, 6, 10, etc.

[0078] It should be noted that, taking the aviation field as an example, the meteorological center may continuously push files to be synchronized to the server. However, if a large number of files to be synchronized are synchronized at one time, it may cause problems such as memory overflow and interface timeout. Therefore, the first files to be synchronized of the target number can be synchronized first, so as to achieve incremental synchronization of files through multiple iterative executions. Among them, the first files to be synchronized are included in the files to be synchronized.

[0079] In some exemplary embodiments, before obtaining the target number of first files to be synchronized, the file synchronization method provided by the embodiments of the present disclosure may also include: obtaining configuration information, the configuration information including the original file path corresponding to the first file to be synchronized, the archived file path corresponding to the archive directory, and at least one of the target quantity information; wherein the original file path is used to obtain the first file to be synchronized, and the archived file path is used to store the first file to be synchronized in a temporary folder in the archive directory.

[0080] In an exemplary embodiment, the configuration information can be obtained through user preset configuration. Taking the aviation field as an example, the meteorological center can transmit the first file to be synchronized to a designated directory in the server based on FTP, wherein the server can be an FTP server, and the original file path corresponding to the first file to be synchronized can be the path of the designated directory in the server. The designated directory can be used to store various files to be synchronized.

[0081] For example, still taking the above-mentioned aviation field as an example, after obtaining the first file to be synchronized of the target data, the first file to be synchronized can be synchronized. At this time, the first file to be synchronized can be first transferred to the archive directory. The archive directory can be used to store the files that need to be synchronized this time. Therefore, the archive file path corresponding to the archive directory can be the path corresponding to the archive directory in the above-mentioned server.

[0082] In some exemplary embodiments, the original file path may be, for example, / files / txt / , and the archive file path may be, for example, / files / archive / .

[0083] In some possible implementations, the target quantity information may be used to indicate an upper limit on the number of files to be synchronized at a single time. In an exemplary embodiment, taking the target quantity information as 6 as an example, the target quantity may be less than or equal to 6.

[0084] In some other possible implementations, the target number information may be used to indicate the number of files to be synchronized at a time. Still taking the target number information as 6 as an example, if the target number needs to be equal to 6. In this case, if the total number of files to be synchronized is less than 6, file synchronization is not triggered.

[0085] In some exemplary embodiments, the configuration information further includes at least one of an address, a port, and login information of the target server.

[0086] In this case, the file synchronization method provided by the embodiment of the present disclosure may further include: logging into the target server according to at least one of the address, port, and login information of the target server. Wherein, obtaining the target number of first files to be synchronized includes: obtaining the target number of first files to be synchronized from the target server.

[0087] In an exemplary embodiment, the target server to be logged in can be determined according to the address and port of the target server. Exemplarily, the target server can be a server storing files to be synchronized, such as an FTP server. The login information can include at least one of a user name and a password. Therefore, the login of the target server can be achieved according to the address, port, and login information of the target server.

[0088] Then, the target number of first files to be synchronized can be acquired according to the original file path included in the configuration information. Then, the target number of first files to be synchronized can be transferred to a temporary folder in the archive directory according to the archive file path included in the configuration information.

[0089] In some exemplary embodiments, before acquiring the target number of first files to be synchronized, the file synchronization method provided by the embodiments of the present disclosure may further include: performing a locking operation to implement a single read of the first files to be synchronized.

[0090] It should be noted that, since the service for reading files in the target server may be distributed on more than two machine instances, there may be concurrency problems if the services of multiple machines read the first file to be synchronized at the same time. In order to prevent repeated reading problems, a distributed lock can be added.

[0091] In an exemplary embodiment, the FTP management service can be called based on the scheduling service in the task scheduler, thereby realizing the login of the FTP service. In this case, the distributed lock service can be called through the scheduling service, thereby realizing the locking operation.

[0092] In some possible implementations, the scheduling service may be locked to prevent other threads from concurrently reading files in a specified directory, or the FTP server may be locked to prevent multiple threads from logging into the FTP server and concurrently reading files.

[0093] S204: Store the first file to be synchronized in a temporary folder in the archive directory.

[0094] In some possible implementations, after obtaining the first files to be synchronized, the target number of first files to be synchronized may be obtained according to the original file path included in the configuration information. Then, the target number of first files to be synchronized may be transferred to a temporary folder in the archive directory according to the archive file path included in the configuration information.

[0095] In some exemplary embodiments, the target number of first files to be synchronized may be transferred to a dynamically generated temporary folder in the archive directory. The disclosed embodiment does not limit the naming method of the dynamically generated temporary folder. In an exemplary instance, the name of the temporary folder may be determined according to the timestamp of the current time. For example, the name of the temporary folder may be 2024080411020300. Alternatively, the temporary folder may be named according to the order in which the temporary folders are dynamically generated.

[0096] Exemplarily, a schematic diagram of a file transfer process for transferring a first to-be-synchronized file from a specified directory to an archive directory provided by an embodiment of the present disclosure may be as follows: Figure 3 shown.

[0097] in, Figure 3 The left side shows the specified directory, where the original file path is / files / txt / . Figure 3 The archive directory is shown on the right side, and the archive file path is / files / archive / .

[0098] like Figure 3 As shown, t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, 10 files to be synchronized are stored in the designated directory. Among them, t1, t2, t3, t4, t5, t6 can be the first files to be synchronized. Therefore, the embodiment of the present disclosure can transfer t1, t2, t3, t4, t5, t6 to a temporary folder named 2024080411020300 in the archive directory, while t7, t8, t9, t10 are still temporarily stored in the designated directory, and t7, t8, t9, t10 can be processed when the file synchronization operation is performed next time.

[0099] S206: Upload the first file to be synchronized in the temporary folder to the distributed file system.

[0100] In some exemplary embodiments, uploading the first file to be synchronized in the temporary folder to the distributed file system includes: converting the format of the first file to be synchronized in the temporary folder to obtain a data stream in a target format; and uploading the data stream in the target format to the distributed file system.

[0101] In an exemplary embodiment, the target format may be a binary format. Since the binary format is usually more compact than the text format and occupies less bandwidth, the disclosed embodiment can improve the transmission speed and reduce the error rate of data transmission through format conversion. In addition, binary transmission reduces the process of character encoding and decoding, so it can reduce transmission errors caused by encoding problems.

[0102] Taking meteorological message data as an example, since meteorological message data may include a large amount of numerical and image data, transmitting in binary format can ensure data integrity and transmission efficiency.

[0103] In some exemplary embodiments, the file synchronization method provided by the embodiments of the present disclosure may also include: in response to satisfying a preset condition, iteratively obtaining a target number of second files to be synchronized, storing the second files to be synchronized in a temporary folder in the archive directory, and uploading the second files to be synchronized in the temporary folder to the distributed file system.

[0104] In some possible implementations, the preset condition may be a time condition, for example, the file to be synchronized may be acquired once at a preset time interval, so if the time from the current time to the last time the file to be synchronized is acquired is not less than the preset time interval, it is determined that the preset condition is met. Exemplarily, the preset time interval may be 30 seconds, 1 minute, 5 minutes, etc. Taking the preset condition of executing once every minute as an example, the expression may be configured as "0* / 1***?".

[0105] Alternatively, the preset condition may be a time point, for example, a fixed time may be set every day to synchronize files, so if the current time is the preset synchronization time, it is determined that the preset condition is met. Exemplarily, the preset synchronization time may be, for example, 9:00, 15:00, etc.

[0106] In an exemplary embodiment, when the preset conditions are met, a small number of files can be read and synchronized at one time, thereby realizing asynchronous execution of file synchronization and avoiding full reading of all files in a specified directory, thereby reducing the impact on the disk and memory and improving the timeliness and stability of file synchronization.

[0107] It should be noted that after uploading the first file to be synchronized to the distributed file system, the first file to be synchronized in the distributed file system can be parsed and / or stored in a database. Since the parsing and storage processing takes a long time and is prone to timeout, uploading the first file to be synchronized to the distributed file system first can avoid the reduction in file synchronization reliability caused by timeout. In addition, there may be problems in which individual files cannot be processed during the processing. Therefore, the reliability of file synchronization can be further improved by first reading a certain number of first files to be synchronized and writing them to the distributed system and then performing file synchronization in an asynchronous manner. In addition, the distributed file system can retain the original files for a long time, which is convenient for subsequent reference and avoids excessive disk space occupation.

[0108] The method provided by the embodiment of the present disclosure can first transfer the target number of first files to be synchronized to the archive directory, and then synchronize the files in the archive directory, thereby avoiding problems such as high disk IO usage, memory overflow, interface timeout, etc. caused by reading all files at one time, and improving the timeliness and stability of file synchronization.

[0109] Exemplarily, a file synchronization system provided by an embodiment of the present disclosure may include a task scheduler and a business system.

[0110] The task scheduler is used to obtain the target number of first files to be synchronized; store the first files to be synchronized in a temporary folder in the archive directory; upload the first files to be synchronized in the temporary folder to the distributed file system. The business system is used to parse and / or store the first files to be synchronized in the distributed file system.

[0111] In an exemplary embodiment, if Figure 4 As shown, the present disclosure may include 6 core services, namely configuration management service, FTP management service, distributed lock service, file upload service, scheduling service, and task scheduler.

[0112] In a possible implementation, for Figure 4 The six core services shown in the figure, a schematic diagram of a file synchronization process provided by an embodiment of the present disclosure can be as follows Figure 5 shown.

[0113] In this Figure 5In the configuration management service, users can pre-define configuration information and configure the task scheduler's tasks. When the preset conditions are met, the task scheduler can trigger the scheduling service to perform asynchronous execution of file synchronization. Among them, the scheduling service can call the distributed lock service to perform a locking operation. If the lock is successful, other threads can no longer call the scheduling service, ensuring that only one thread reads the file to be synchronized. In addition, after the file synchronization is completed, the scheduling service can also call the distributed lock service to release the lock operation.

[0114] Exemplarily, the scheduling service can call the FTP management service to log in to the FTP server to read the first to-be-synchronized file of the target data in the specified directory, and transfer these files to a dynamically generated temporary folder in the archive directory. It should be noted that the FTP management service can read the configuration information in the configuration management service to implement the login of the FTP server.

[0115] Then, the scheduling service can package and compress the first file to be synchronized in the temporary folder into a binary data stream, and then call the file upload service to upload the binary data stream to the distributed file system to complete a single incremental file synchronization.

[0116] Finally, the task scheduler can also trigger a new scheduling service again when the preset conditions are met to repeat the steps executed by the above scheduling service, thereby achieving multiple file synchronizations.

[0117] In an exemplary embodiment, Figure 6 A schematic diagram showing user-defined configuration information.

[0118] In this Figure 6 In the configuration management service, the user can define the configuration information. For example, the configuration information may include the address of the target server, the port of the target server, the login information, the original file path, the archive file path, and the target quantity information. The login information may include at least one of a user name and a password.

[0119] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations, and various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of this disclosure have all been authorized.

[0120] Based on the same inventive concept, the present disclosure also provides a file synchronization device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0121] Figure 7 A schematic diagram of a file synchronization device in an embodiment of the present disclosure is shown. Figure 7 As shown, the device comprises:

[0122] The file acquisition module 701 is used to acquire a target number of first files to be synchronized;

[0123] The file storage module 702 is used to store the first file to be synchronized in a temporary folder in the archive directory;

[0124] The file uploading module 703 is used to upload the first file to be synchronized in the temporary folder to the distributed file system.

[0125] In some exemplary embodiments, the file synchronization device provided by the embodiments of the present disclosure further includes:

[0126] A configuration information acquisition module, used to acquire configuration information, the configuration information including at least one of an original file path corresponding to the first file to be synchronized, an archive file path corresponding to the archive directory, and target quantity information;

[0127] The original file path is used to obtain the first file to be synchronized, and the archived file path is used to store the first file to be synchronized in a temporary folder in the archive directory.

[0128] In some exemplary embodiments, the configuration information further includes at least one of an address, a port, and login information of the target server. The file synchronization device provided by the embodiment of the present disclosure further includes:

[0129] A target server login module, used to log in to the target server according to at least one of the address, port, and login information of the target server;

[0130] The file acquisition module 701 is used to acquire a target number of first files to be synchronized from the target server.

[0131] In some exemplary embodiments, the file upload module 703 is used to perform format conversion on the first file to be synchronized in the temporary folder to obtain a data stream in a target format; and upload the data stream in the target format to the distributed file system.

[0132] In some exemplary embodiments, the file acquisition module 701 is further configured to iteratively acquire a target number of second files to be synchronized in response to satisfying a preset condition;

[0133] The file storage module 702 is further used to store the second file to be synchronized in a temporary folder in the archive directory;

[0134] The file uploading module 703 is further configured to upload the second file to be synchronized in the temporary folder to the distributed file system.

[0135] In some exemplary embodiments, the file synchronization device provided by the embodiments of the present disclosure further includes:

[0136] The locking module is used to perform a locking operation to achieve a single read of the first file to be synchronized.

[0137] The device provided by the embodiment of the present disclosure can first transfer the target number of first files to be synchronized to the archive directory, and then synchronize the files in the archive directory, thereby avoiding problems such as high disk IO usage, memory overflow, interface timeout, etc. caused by reading all files at one time, thereby improving the timeliness and stability of file synchronization.

[0138] It should be noted that the examples and application scenarios implemented by the modules in the above-mentioned device embodiment are the same as those of the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0139] Those skilled in the art will appreciate that various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0140] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above file synchronization methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0141] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0142] like Figure 8As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0143] The storage unit stores a program code, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps of the above method embodiment:

[0144] Obtain a target number of first files to be synchronized; store the first files to be synchronized in a temporary folder in the archive directory; and upload the first files to be synchronized in the temporary folder to a distributed file system.

[0145] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0146] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0147] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0148] The electronic device 800 may also communicate with one or more external devices 840 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0149] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0150] Based on the same inventive concept, the embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, any of the above-mentioned file synchronization methods is implemented. Since the principle of solving the problem in the computer-readable storage medium embodiment is similar to that in the above-mentioned method embodiment, the implementation of the computer-readable storage medium embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0151] Fig. 9 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. Fig. 9 As shown, a program product capable of implementing the above method of the present disclosure is stored on the computer-readable storage medium 900. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0152] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0154] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0155] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0156] Based on the same inventive concept, a computer program product is also provided in the embodiment of the present disclosure, including a computer program product, including: a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the file synchronization method of any one of the above method embodiments is implemented. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0157] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0158] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0159] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0160] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope of the present disclosure is indicated by the appended claims.

Claims

1. A file synchronization method, characterized in that: include: Get the target number of first files to be synchronized; Storing the first file to be synchronized in a temporary folder in the archive directory; The first file to be synchronized in the temporary folder is uploaded to the distributed file system.

2. The file synchronization method according to claim 1, characterized in that: Before obtaining the target number of first files to be synchronized, the method further includes: Acquire configuration information, where the configuration information includes at least one of an original file path corresponding to the first file to be synchronized, an archive file path corresponding to the archive directory, and target quantity information; The original file path is used to obtain the first file to be synchronized, and the archived file path is used to store the first file to be synchronized in a temporary folder in the archive directory.

3. The file synchronization method according to claim 2, characterized in that: The configuration information also includes at least one of an address, a port, and login information of the target server, and the method further includes: Log in to the target server according to at least one of the address, port, and login information of the target server; The step of obtaining a target number of first files to be synchronized includes: A target number of first files to be synchronized are obtained from the target server.

4. The file synchronization method according to any one of claims 1 to 3, characterized in that: The step of uploading the first file to be synchronized in the temporary folder to the distributed file system comprises: Performing format conversion on the first file to be synchronized in the temporary folder to obtain a data stream in a target format; The data stream in the target format is uploaded to a distributed file system.

5. The file synchronization method according to any one of claims 1 to 3, characterized in that: The method further comprises: In response to satisfying a preset condition, iteratively acquiring a target number of second files to be synchronized, storing the second files to be synchronized in a temporary folder in the archive directory, and uploading the second files to be synchronized in the temporary folder to the distributed file system.

6. The file synchronization method according to any one of claims 1 to 3, characterized in that: Before obtaining the target number of first files to be synchronized, the method further includes: A locking operation is performed to implement a single read of the first file to be synchronized.

7. A file synchronization system, characterized in that: include: A task scheduler, used for obtaining a target number of first files to be synchronized; Storing the first file to be synchronized in a temporary folder in the archive directory; Uploading the first file to be synchronized in the temporary folder to the distributed file system; The business system is used to parse and / or store in a database the first file to be synchronized in the distributed file system.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the file synchronization method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the file synchronization method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the file synchronization method as described in any one of claims 1 to 6.

Images