File processing method and device based on rsync, equipment and storage medium

Abstract

The invention discloses a file processing method and device based on rsync, electronic equipment and a storage medium, and the method comprises the steps: generating an rsync command when a file backup request is received, and obtaining source file information contained in the file backup request; the source file corresponding to the source file information is subjected to sub-file identification, sub-file information of each sub-file contained in the source file is obtained, and the sub-file information comprises a file path and a file size; determining the number of threads for performing backup processing on the source file according to the file size in the sub-file information, and generating an xargs command according to the number of the threads; and performing thread calling according to the xargs command, and executing the rsync command to back up each sub-file in the source file into the storage environment according to the file path. When the file is processed, the parallel threads are determined according to the size of the file, the threads are distributed through the xargs command to execute the rsync command in parallel to backup the file, the file backup efficiency is improved, and waste of performance resources is avoided.

Description

Technical Field

[0001] The present application relates to the technical field of file synchronization processing, and in particular to a file processing method, device, electronic device and storage medium based on rsync. Background Art

[0002] At present, there are many backup tools based on Linux environment, such as UrBackup, Clonezilla, rsync, etc. Among them, rsync is still one of the biggest competitors for enterprise-level tasks. It provides the most advanced functions, such as incremental backup, options to update the entire directory tree, file system support for local and remote backup, and file permission retention. The current mainstream technology for using rsync for file backup and recovery is to execute the rsync command backup command to operate the target file after obtaining the file directory information, and perform full backup, incremental backup, file recovery and other operations on it.

[0003] Currently, the amount of core data in the Linux environment is large, and the number of files is large. Since rsync cannot perform concurrent synchronization processing, the rsync process has a large performance bottleneck and cannot maximize the IO performance of the storage device, resulting in low efficiency in processing file data. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a file processing method, device, electronic device and storage medium based on rsync to solve the technical problem of low efficiency in file backup synchronization processing in rsync in the related art.

[0005] In a first aspect, an embodiment of the present application provides a file processing method based on rsync, comprising:

[0006] When a file backup request is received, an rsync command is generated, and source file information included in the file backup request is obtained;

[0007] Identify subfiles of the source file corresponding to the source file information to obtain subfile information of each subfile contained in the source file, wherein the subfile information includes a file path and a file size;

[0008] Determine the number of threads for backing up the source file according to the file size in the sub-file information, and generate an xargs command according to the number of threads;

[0009] A thread call is performed according to the xargs command, and the rsync command is executed to back up each sub-file in the source file to a storage environment according to the file path.

[0010] In a second aspect, an embodiment of the present application provides a file processing device based on rsync, comprising:

[0011] A request receiving module, used for generating an rsync command when receiving a file backup request, and obtaining source file information included in the file backup request;

[0012] An information identification module, used to identify sub-files of a source file corresponding to the source file information, and obtain sub-file information of each sub-file contained in the source file, wherein the sub-file information includes a file path and a file size;

[0013] A thread determination module, used for determining the number of threads for backing up the source file according to the file size in the sub-file information, and generating an xargs command according to the number of threads;

[0014] The file processing module is used to perform thread calling according to the xargs command, and execute the rsync command to back up each sub-file in the source file to the storage environment according to the file path.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned rsync-based file processing methods when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned rsync-based file processing methods are implemented.

[0017] The embodiment of the present application provides a file processing method, device, electronic device and storage medium based on rsync. When performing file backup processing, a corresponding rsync command, i.e., a file synchronization command, is generated when a file backup request is received, and the source file information contained in the file backup request is obtained. Then, the source file corresponding to the source file information is identified and processed as a subfile, and the subfile information of each subfile contained in the source file is obtained, including the file path and the file size. Then, the number of threads for performing backup processing on the source file is determined according to the file size, so as to generate an xargs command according to the number of threads, and finally, a thread call is performed according to the xargs command, and the rsync command is executed to store each subfile in the source file in an environment according to the file path. When using rsync to perform synchronous processing such as backing up files, thread resources are allocated through the xargs command and the rsync command is synchronously executed to perform file backup processing, thereby improving the efficiency of file synchronous backup and other processing. At the same time, when determining the required parallel threads, it is determined based on the size of the source file, thereby avoiding the waste of performance resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of a file processing method based on rsync provided in an embodiment of the present application;

[0019] Figure 2 It is a block diagram schematic diagram of an application scenario of the rsync-based file processing method provided in an embodiment of the present application;

[0020] Figure 3 It is a flowchart of the steps of determining the number of threads provided in an embodiment of the present application;

[0021] Figure 4 This is a flowchart of the steps of obtaining the number of threads according to the comparison results provided in an embodiment of the present application;

[0022] Figure 5 It is another flowchart diagram of the steps of the rsync-based file processing method provided in an embodiment of the present application;

[0023] Figure 6 This is a structural diagram of a file processing device based on rsync provided in an embodiment of the present application;

[0024] Figure 7 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0025] Figure 8 This is another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] In the related art, the amount of core data in the Linux environment is large and the number of files is large. Since rsync cannot perform concurrent synchronization processing, only one thread can be used to perform synchronization processing such as backing up files, which makes the rsync process have a large performance bottleneck and cannot maximize the IO performance of the storage device, resulting in low efficiency in processing file data when using rsync.

[0030] In order to solve the technical problems existing in the related art, the present application embodiment provides a file processing method based on rsync, see Figure 1 , Figure 1 It is a flow chart of a file processing method based on rsync provided in an embodiment of the present application, and the method includes steps 101 to 104.

[0031] Step 101: when a file backup request is received, an rsync command is generated, and source file information included in the file backup request is obtained.

[0032] In one embodiment, when a file backup request is received, an rsync command, ie, a file synchronization command, is generated, and source file information included in the file backup request is obtained, wherein the source file corresponding to the source file information is a data file that needs to be backed up.

[0033] For example, refer to Figure 2 , Figure 2The present invention is a block diagram of an application scenario of a file processing method based on rsync provided by an embodiment of the present application, including a server, a production environment and a storage environment, wherein an agent is deployed in the production environment. In actual applications, when performing file backup processing, the server sends a request to the agent, so that the agent executes the received request in the production environment to realize backing up the files to be backed up in the storage environment, or when performing file reading processing, the server sends a request to the agent, so that the agent executes the received request in the production environment to realize reading the corresponding files from the storage environment.

[0034] In actual applications, taking file backup processing as an example, the agent receives the file backup request sent by the server to identify the source file information contained in the file backup request, where the source file information can be the unique identification information of the file, such as numbers, letters, or a combination of numbers and letters, etc. At the same time, an rsync command for file backup is generated, and when performing backup, the backup processing of the source file is realized by executing the rsync command.

[0035] Step 102, identifying sub-files of the source file corresponding to the source file information, and obtaining sub-file information of each sub-file contained in the source file, wherein the sub-file information includes a file path and a file size.

[0036] In one embodiment, after the source file information is obtained, the sub-files of the source file corresponding to the source file information are identified to obtain the sub-file information of each sub-file contained in the source file, and the obtained sub-file information includes the file path and file size of the sub-file.

[0037] Exemplarily, when identifying the source file corresponding to the source file information, the file identifier contained in the source file information can be identified, each file identifier corresponds to a sub-file, and when the included sub-files are identified, the file path and file size associated with each sub-file are obtained, wherein the file path and file size can be included in the file backup request, and the file path is the storage path of the sub-file in the storage environment.

[0038] Furthermore, for files stored in a storage environment, a corresponding association list can be constructed, and the file identifier and the file path can be associated and recorded in the association list. When reading a file, the file path to be read can be determined in the association list, and the file can be searched and read according to the file path, so as to complete operations such as reading the file in the production environment. Therefore, when obtaining sub-file information, it includes: querying the file identifier contained in the source file information to obtain the sub-file identifier contained in the source file information; obtaining a pre-recorded relationship list, and querying the relationship list according to the sub-file identifier to obtain the sub-file information corresponding to each sub-file identifier.

[0039] Step 103, determining the number of threads for backing up the source file according to the file size in the sub-file information, and generating an xargs command according to the number of threads.

[0040] In one embodiment, when performing file backup processing, in order to improve the efficiency of file backup, the number of threads when performing the backup processing on the source file can be determined according to the file size of each sub-file obtained, and the corresponding xargs command is generated according to the determined number of threads, wherein the xargs command is used to execute thread resource allocation and concurrently and synchronously perform file backup processing, and when performing file backup processing, it is completed by concurrently and synchronously executing the rsync command.

[0041] When determining the number of threads to back up the source file based on the file size, you can refer to Figure 3 , Figure 3 It is a flowchart of the steps for determining the number of threads provided in an embodiment of the present application, wherein the steps include steps 301 to 304.

[0042] Step 301, calculating the average file size of the sub-files contained in the source file according to the file size contained in the sub-file information;

[0043] Step 302, comparing the file size average with the first threshold, and determining the number of threads for backing up the source file according to the comparison result;

[0044] Step 303: if the file size average is greater than or equal to the first threshold, the sub-files contained in the source file are divided to obtain a first file set whose file size is greater than or equal to the file delimiter value and a second file set whose file size is less than the file delimiter value, and the number of threads for backing up the source file is determined based on the first file set and the second file set;

[0045] Step 304: If the file size average is smaller than the first threshold, the maximum file size among the file sizes is compared with the second threshold, and the number of threads for backing up the source file is determined according to the comparison result.

[0046] Exemplarily, when determining the number of threads based on sub-file information, the number is determined based on the file size of the sub-files. Specifically, based on the file size of all sub-files in the source file, the average file size of the sub-files is calculated, and then the obtained average file size is compared with a first threshold value to determine the number of threads for backing up the source file based on the comparison result.

[0047] When performing comparison processing, there are three size relationships between the mean file size and the first threshold, and when determining the number of threads, the size relationship is divided into: the mean file size is greater than or equal to the first threshold and the mean file size is less than the first threshold, and different size relationships correspond to different results.

[0048] When the mean file size is greater than or equal to a first threshold, the sub-files contained in the source file are divided based on the file size to obtain a first file set whose file size is greater than or equal to the file boundary value and a second file set whose file size is less than the file boundary value. Further processing is then performed based on the first file set and the second file set to determine the number of threads used to back up the source file, wherein the file boundary value is used to divide the file into large files and small files.

[0049] In addition, when the average file size is less than the first threshold, the maximum file size among the file sizes is compared with the file boundary value, and then the number of threads for backing up the source file is determined according to the further comparison result.

[0050] That is, when determining the number of threads for backing up the source file, a first comparison is first made based on the mean file size of the sub-files and the first threshold. When the mean file size is greater than the first threshold, it is determined that the source file contains at least one large file (for example, the file size of the large file is greater than or equal to 100G). At this time, in determining the number of threads, one large file can correspond to one thread, while for small files, one thread can be shared.

[0051] Therefore, by dividing the source files based on file size, a first file set containing large files and a second file set containing small files are obtained. Specifically, the file size is compared with the file boundary value. Files that are greater than or equal to the file boundary value are determined to be large files, otherwise they are determined to be small files. The specific size of the file boundary value is set according to actual conditions and is not specifically limited.

[0052] Furthermore, when determining the number of threads for backing up the source file, the relationship between the mean file size and the first threshold is used for processing, and different size relationships require further processing, including the two cases where the mean file size is greater than or equal to the first threshold and the mean file size is less than the first threshold.

[0053] For the case where the average file size is greater than or equal to the first threshold, you can refer to Figure 4 , Figure 4 It is a flowchart diagram of the step of obtaining the number of threads according to the comparison result provided in an embodiment of the present application, wherein the step includes step 401 to step 404.

[0054] Step 401, obtaining the number of sub-files included in the first file set, obtaining the number of first files, and using the number of first files as the number of first threads;

[0055] Step 402, obtaining the backup duration of each sub-file in the first file set, and obtaining a first maximum backup duration by comparing the durations;

[0056] Step 403, combining files according to the backup duration of each sub-file in the second file set to obtain a plurality of groups of combined files and a total backup duration of each combined file, wherein the difference between the total backup duration and the first maximum backup duration is less than a third threshold;

[0057] Step 404: Add the first thread quantity to the combined quantity of combined files to obtain the thread quantity for backing up the source file.

[0058] Specifically, when the file size mean is greater than or equal to a first threshold, the first file set and the second file set are obtained by dividing the sub-files in the source file. At this time, the number of sub-files contained in the first file set is obtained to obtain the first file number, and the first file number is used as the first thread number. At the same time, the backup duration of each sub-file in the first file set is obtained, and the first maximum backup duration is obtained by comparing the durations. Then, based on the first maximum backup duration, the sub-files in the second file set are combined according to the backup duration to obtain several groups of file combinations and the total backup duration of each file combination, wherein the difference between the total backup duration corresponding to each group and the first maximum backup duration is less than a third threshold, that is, to ensure that the total backup duration of each group is close to the first maximum backup duration.

[0059] For example, for the first file set, a thread can be called for each subfile in the first file set to perform backup processing, so the number of first threads is obtained by determining the number of subfiles in the first file set. For the subfiles included in the second file set, when the subfiles are backed up separately, the backup time is quite different from the backup time of the large file, such as the difference between 2 minutes and 1 hour. Therefore, the subfiles in the second file set can be combined for backup, that is, multiple subfiles can be serially backed up based on the same thread. At the same time, in order to improve the backup efficiency and reduce the backup time, when the second file set is divided, it is necessary to ensure that the backup time of each group is close to the first maximum backup time, so that when performing parallel synchronous backup processing, the backup efficiency is guaranteed while the backup time is effectively reduced.

[0060] For the case where the average file size is less than the first threshold, the maximum file size in the file size is compared with the second threshold, and then the number of threads for backing up the source file is determined based on the comparison result, specifically including: if the maximum file size is less than the file delimiter value, the number of threads for backing up the source file is determined to be one; if the maximum file size is greater than or equal to the file delimiter value, the first backup duration for backing up the source file is obtained, and the second backup duration predicted for the source file using a pre-trained backup duration prediction model is obtained, and the number of threads for backing up the source file is determined based on the first backup duration and the second backup duration.

[0061] Exemplarily, when the maximum file size in the source file is compared with the file delimiter value, if the maximum file size is smaller than the file delimiter value, the number of threads for backing up the source file is determined to be one; otherwise, a first backup duration for backing up the source file is obtained, and a second backup duration is obtained by predicting the source file using a pre-trained backup duration prediction model, and then the number of threads for backing up the source file is determined based on the first backup duration and the second backup duration.

[0062] In actual applications, when the file size mean is less than the first threshold, it is determined that there may be no large files. Therefore, it will be further compared with the file delimiter value. By comparing the maximum file size with the file delimiter value, it is determined whether there are large files in the source file. When it is determined that there are no large files, that is, the maximum file size is less than the file delimiter value, at this time, the number of threads for backing up the source file can be directly determined to be one, that is, one thread is used to perform serial backup processing on the source file. It should be noted that when it is determined that there are no large files through file size comparison, the backup time during the backup processing is shorter. If multiple threads are called to perform different parallel backup processing at this time, it will not have much impact on the backup efficiency and backup time, but will increase the application load.

[0063] When the maximum file size is greater than or equal to the file delimiter, it indicates that there is a large file in the source file. At this time, the first backup duration for backing up the source file can be obtained, and the second backup duration obtained by using the pre-trained backup duration prediction model can be obtained. Then, the number of threads can be determined by comparing the first backup duration with the second backup duration. It should be noted that the first backup duration is estimated by the system or operator based on historical backup situations. The backup duration of the file is determined by the file type, size, etc. In the historical backup records, there are certain backup records for different file types and file sizes. By checking the records, the first backup duration for backing up the source file can be estimated.

[0064] In order to improve the backup efficiency, the corresponding backup duration prediction model can be pre-trained based on the historical backup data information, and then when backing up the source file, the backup duration prediction model can be used to predict the duration and obtain the second backup duration for backing up the source file. When determining the number of threads based on the first backup duration and the second backup duration, if the difference between the first backup duration and the second backup duration is small and meets the set conditions, the source file can be directly backed up based on one thread, otherwise it is necessary to call multiple threads to back up the source file.

[0065] It should be noted that when the backup duration prediction model is used to obtain the second backup duration, it is a predicted result of the backup processing of the source file based on multi-threading. Therefore, when comparing the first backup duration with the second backup duration, it can be determined whether it is necessary to start multi-threading. If not, the number of threads is one. Otherwise, it needs to be further determined based on the source file itself.

[0066] Therefore, when determining the number of threads based on the first backup duration and the second backup duration, it includes: calculating the time change rate of the second backup duration compared with the first backup duration; if the time change rate of the first backup duration to the second backup duration is less than a third threshold, determining the number of threads to be one; if the time change rate of the first backup duration to the second backup duration is greater than or equal to the third threshold, dividing the sub-files contained in the source file to obtain a third file set whose file size is greater than or equal to the file boundary value and a fourth file set whose file size is less than the file boundary value, and determining the number of threads for backing up the source file based on the third file set and the fourth file set.

[0067] Specifically, the time change rate of the second backup duration compared with the first backup duration is calculated, and the number of threads is determined according to the time change rate. If the time change rate is less than the third threshold, the number of threads is determined to be one. If the time change rate is greater than or equal to the third threshold, the source file needs to be divided, and then the files are combined according to the backup duration of each sub-file to determine the required number of threads.

[0068] Among them, the calculation formula of the duration change rate is: duration change rate = (second backup duration - first backup duration) / first backup duration, or, duration change rate = (first backup duration - second backup duration) / second backup duration. When the duration change rate is greater than or equal to the third threshold, the processing process of the source file can refer to the processing method recorded in steps 401 to 404 to process the source file.

[0069] Further, after the thread data is determined, a corresponding xargs command is generated according to the number of threads, such as xargs -P, where P is the number of threads.

[0070] Step 104, calling a thread according to the xargs command, and executing the rsync command to back up each sub-file in the source file to the storage environment according to the file path.

[0071] In one embodiment, after the corresponding xargs command is generated, a thread is called according to the xargs command, and the pre-generated rsync command is executed to store each sub-file in the source file into the storage environment according to the obtained file path.

[0072] Furthermore, when reading files, corresponding files can also be read from the storage environment through multi-threaded parallel synchronization processing. Specifically, it includes: when a file reading request is received, a target rsync command is generated, and the target file information contained in the file reading request is obtained; the sub-files of the target data corresponding to the target file information are identified, and the target sub-file information of each sub-file contained in the target data is obtained, wherein the target sub-file information includes the target file path and the target file size; according to the target file size in the target sub-file information, the target number of threads for reading the source file is determined, and the target xargs command is generated according to the target number of threads; according to the target xargs command, a thread call is performed, and the target rsync command is executed to read the corresponding target sub-file from the storage environment based on the target file path.

[0073] Among them, when reading a file, similar to the process of backing up a file, the information of the target file to be read is obtained, including the file size, file path, etc., and whether to call multiple threads and the number of threads to be called are determined according to the file size, and then the file is read according to the determined method. Since the process of file reading and file backup is similar and part of the processing process is the same, the details are not repeated here.

[0074] Further, refer to Figure 5 , Figure 5 It is another flow chart of the steps of the file processing method based on rsync provided in an embodiment of the present application, wherein the steps include:

[0075] S1, start backup: upon receiving a file backup request, start the backup program to start backing up the source file;

[0076] S2, file traversal: traverse and accumulate the file size of each sub-file in the source file to obtain the average file size of the source file;

[0077] S3, size comparison: compare the obtained file size mean with the preset threshold to determine the size relationship between the file size mean and the preset threshold. If the file size mean is less than the preset threshold, enter S2, otherwise enter S4;

[0078] S4, executing the rsync command through the xargs command: determining the thread calling status according to the source file, and then executing the rsync command through the xargs command to back up the source file;

[0079] S5, return result: provide result feedback according to the backup processing result, including whether the backup is completed and the backup progress, etc.

[0080] It should be noted that the first threshold, the second threshold, the third threshold and the file delimiter mentioned in the above embodiments involve specific numerical limitations, which are set based on actual conditions and requirements and are not specifically limited.

[0081] In summary, the present application discloses a file processing method based on rsync. When performing file backup processing, a corresponding rsync command, i.e., a file synchronization command, is generated when a file backup request is received, and the source file information contained in the file backup request is obtained. Then, the source file corresponding to the source file information is identified and processed as a subfile, and the subfile information of each subfile contained in the source file is obtained, including the file path and the file size. Then, the number of threads for performing backup processing on the source file is determined according to the file size, so as to generate an xargs command according to the number of threads, and finally, a thread call is performed according to the xargs command, and the rsync command is executed to store each subfile in the source file in an environment according to the file path. When using rsync to perform synchronous processing such as backing up files, thread resources are allocated through the xargs command and the rsync command is synchronously executed to perform file backup processing, thereby improving the efficiency of file synchronous backup and other processing. At the same time, when determining the required parallel threads, the threads are determined based on the size of the source file, thereby avoiding the waste of performance resources.

[0082] According to the method described in the above embodiment, this embodiment will be further described from the perspective of an rsync-based file processing device. The rsync-based file processing device can be implemented as an independent entity or integrated in an electronic device, such as a terminal. The terminal may include a mobile phone, a tablet computer, etc.

[0083] See also Figure 6 , Figure 6 is a structural diagram of a file processing device based on rsync provided in an embodiment of the present application, such as Figure 6 As shown, the rsync-based file processing device 600 provided in the embodiment of the present application includes:

[0084] The request receiving module 601 is used to generate an rsync command when receiving a file backup request, and obtain source file information included in the file backup request;

[0085] The information identification module 602 is used to identify sub-files of the source file corresponding to the source file information, and obtain sub-file information of each sub-file contained in the source file, wherein the sub-file information includes a file path and a file size;

[0086] The thread determination module 603 is used to determine the number of threads for backing up the source file according to the file size in the sub-file information, and generate an xargs command according to the number of threads;

[0087] The file processing module 604 is used to perform thread calling according to the xargs command, and execute the rsync command to back up each sub-file in the source file to the storage environment according to the file path.

[0088] In one embodiment, the thread determination module 603 is further configured to:

[0089] According to the file sizes contained in the sub-file information, the average file sizes of the sub-files contained in the source file are calculated;

[0090] Compare the file size average with the first threshold, and determine the number of threads for backing up the source file according to the comparison result;

[0091] If the file size mean is greater than or equal to the first threshold, the sub-files contained in the source file are divided to obtain a first file set whose file size is greater than or equal to the file delimiter value and a second file set whose file size is less than the file delimiter value, and the number of threads for backing up the source file is determined based on the first file set and the second file set;

[0092] If the file size average is smaller than the first threshold, the maximum file size among the file sizes is compared with the second threshold, and the number of threads for backing up the source file is determined according to the comparison result.

[0093] In one embodiment, the thread determination module 603 is further configured to:

[0094] Obtain the number of sub-files included in the first file set to obtain the number of first files, and use the number of first files as the number of first threads;

[0095] Obtaining the backup duration of each sub-file in the first file set, and obtaining a first maximum backup duration by comparing the durations;

[0096] Combining files according to the backup duration of each sub-file in the second file set to obtain a plurality of groups of combined files and a total backup duration of each combined file, wherein a difference between the total backup duration and the first maximum backup duration is less than a third threshold;

[0097] The number of first threads is added to the number of combinations of combined files to obtain the number of threads for backing up the source files.

[0098] In one embodiment, the thread determination module 603 is further configured to:

[0099] If the maximum file size is smaller than the file delimiter value, the number of threads for backing up the source file is determined to be one;

[0100] If the maximum file size is greater than or equal to the file boundary value, the first backup duration for backing up the source file is obtained, and the second backup duration predicted for the source file using a pre-trained backup duration prediction model is obtained, and the number of threads for backing up the source file is determined based on the first backup duration and the second backup duration.

[0101] In one embodiment, the thread determination module 603 is further configured to:

[0102] Calculate the change rate of the second backup duration compared to the first backup duration;

[0103] If the duration change rate between the first backup duration and the second backup duration is less than a third threshold, determining the number of threads to be one;

[0104] If the time change rate between the first backup time and the second backup time is greater than or equal to a third threshold, the sub-files contained in the source file are divided to obtain a third file set whose file size is greater than or equal to the file boundary value and a fourth file set whose file size is less than the file boundary value, and the number of threads for backing up the source file is determined based on the third file set and the fourth file set.

[0105] In one embodiment, the rsync-based file processing device 600 further includes a file reading module for:

[0106] When a file read request is received, a target rsync command is generated, and target file information contained in the file read request is obtained;

[0107] Identify subfiles of target data corresponding to the target file information to obtain target subfile information of each subfile contained in the target data, wherein the target subfile information includes a target file path and a target file size;

[0108] According to the target file size in the target sub-file information, the target number of threads for reading and processing the source file is determined, and a target xargs command is generated according to the target number of threads;

[0109] A thread call is performed according to the target xargs command, and the target rsync command is executed to read the corresponding target sub-file from the storage environment based on the target file path.

[0110] In one embodiment, the information identification module 602 is further used to:

[0111] Querying the file identifier contained in the source file information to obtain the sub-file identifier contained in the source file information;

[0112] A pre-recorded relationship list is obtained, and a query is performed in the relationship list according to the sub-file identifier to obtain the sub-file information corresponding to each sub-file identifier.

[0113] Also, see Figure 7 , Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Figure 7 As shown, the electronic device 700 includes a processor 701 and a memory 702. The processor 701 is electrically connected to the memory 702.

[0114] The processor 701 is the control center of the electronic device 700. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 700 and processes data by running or loading applications stored in the memory 702 and calling data stored in the memory 702, thereby monitoring the electronic device 700 as a whole.

[0115] In this embodiment, the processor 701 in the electronic device 700 will load the instructions corresponding to the processes of one or more applications into the memory 702 according to the following steps, and the processor 701 will run the application stored in the memory 702, thereby implementing any step in the rsync-based file processing method provided in the above embodiment.

[0116] The electronic device 700 can implement the steps in any embodiment of the rsync-based file processing method provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any rsync-based file processing method provided in the embodiments of the present application. For details, please refer to the previous embodiments and will not be repeated here.

[0117] See also Figure 8 , Figure 8 is another structural schematic diagram of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, Figure 8 The specific structural block diagram of the electronic device provided in the embodiment of the present application is shown, and the electronic device can be used to implement the file processing method based on rsync provided in the above embodiment. The electronic device 800 can be a mobile terminal such as a smart phone or a laptop computer.

[0118] The RF circuit 810 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 810 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 810 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0119] The memory 820 can be used to store software programs and modules, such as the program instructions / modules corresponding to the rsync-based file processing method in the above-mentioned embodiment. The processor 880 executes various functional applications and the rsync-based file processing method by running the software programs and modules stored in the memory 820.

[0120] The memory 820 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 820 may further include a memory remotely arranged relative to the processor 880, and these remote memories may be connected to the electronic device 800 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0121] The input unit 830 can be used to receive uploaded digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 830 may include a touch-sensitive surface 831 and other input devices 832. The touch-sensitive surface 831, also known as a touch display screen or touchpad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 831), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 831 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 880, and can receive and execute commands sent by the processor 880. In addition, the touch-sensitive surface 831 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 831, the input unit 830 may further include other input devices 832. Specifically, the other input devices 832 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0122] The display unit 840 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 800, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 840 may include a display panel 841, and optionally, the display panel 841 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 831 may cover the display panel 841, and when the touch-sensitive surface 831 detects a touch operation on or near it, it is transmitted to the processor 880 to determine the type of the touch event, and then the processor 880 provides corresponding visual output on the display panel 841 according to the type of the touch event. Although in the figure, the touch-sensitive surface 831 and the display panel 841 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 831 can be integrated with the display panel 841 to implement input and output functions.

[0123] The electronic device 800 may also include at least one sensor 850, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 841 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 800, they will not be repeated here.

[0124] The audio circuit 860, the speaker 861, and the microphone 862 can provide an audio interface between the user and the electronic device 800. The audio circuit 860 can transmit the electrical signal converted from the received audio data to the speaker 861, which is converted into a sound signal for output; on the other hand, the microphone 862 converts the collected sound signal into an electrical signal, which is received by the audio circuit 860 and converted into audio data, and then the audio data is output to the processor 880 for processing, and then sent to another terminal through the RF circuit 810, or the audio data is output to the memory 820 for further processing. The audio circuit 860 may also include an earplug jack to provide communication between an external headset and the electronic device 800.

[0125] The electronic device 800 can help the user receive requests, send information, etc. through the transmission module 870 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 870 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 800 and can be omitted as needed without changing the essence of the invention.

[0126] The processor 880 is the control center of the electronic device 800. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 820, and calling data stored in the memory 820, it executes various functions of the electronic device 800 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 880 may include one or more processing cores; in some embodiments, the processor 880 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 880.

[0127] The electronic device 800 also includes a power supply 890 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 880 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 890 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0128] Although not shown, the electronic device 800 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein the one or more programs are stored in the memory, and are configured to be executed by one or more processors to implement any step of the file processing method based on rsync provided in the above embodiment.

[0129] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0130] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium, which stores multiple instructions, and the instructions can be executed by the processor to implement any step of the file processing method based on rsync provided in the above embodiments.

[0131] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0132] Since the instructions stored in the storage medium can execute the steps in any embodiment of the rsync-based file processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any rsync-based file processing method provided in the embodiments of the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0133] The above is a detailed introduction to a file processing method, device, electronic device and storage medium based on rsync provided by the embodiments of the present application. The principles and implementation methods of the present application are described in detail using specific examples herein. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application. Moreover, for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can be made, and these improvements and modifications are also regarded as the protection scope of the present application.

Claims

1. A file processing method based on rsync, characterized in that: include: When a file backup request is received, an rsync command is generated and the source file information included in the file backup request is obtained; Identifying subfiles of a source file corresponding to the source file information to obtain subfile information of each subfile contained in the source file, wherein the subfile information includes a file path and a file size; Determine the number of threads for backing up the source file based on the file size in the subfile information, and generate an xargs command based on the number of threads. A thread call is performed according to the xargs command, and the rsync command is executed to back up each sub-file in the source file to a storage environment according to the file path.

2. The method according to claim 1, wherein The step of determining the number of threads for backing up the source file according to the file size in the sub-file information includes: Calculating the average file size of the sub-files included in the source file based on the file sizes included in the sub-file information; Comparing the file size mean with a first threshold, and determining the number of threads for backing up the source file according to the comparison result; If the file size mean is greater than or equal to the first threshold, dividing the subfiles contained in the source file into a first file set whose file size is greater than or equal to the file delimiter value and a second file set whose file size is less than the file delimiter value, and determining the number of threads for backing up the source file based on the first file set and the second file set; If the file size average is smaller than the first threshold, the maximum file size among the file sizes is compared with the second threshold, and the number of threads for backing up the source file is determined according to the comparison result.

3. The method according to claim 2, wherein The determining, based on the first file set and the second file set, the number of threads for backing up the source files includes: Obtaining the number of sub-files included in the first file set to obtain a first file number, and using the first file number as a first thread number; Obtaining a backup duration for each subfile in the first file set, and obtaining a first maximum backup duration by comparing the durations; Combining files based on the backup duration of each subfile in the second file set to obtain a plurality of groups of combined files and a total backup duration of each combined file, wherein a difference between the total backup duration and the first maximum backup duration is less than a third threshold; The number of the first threads is added to the number of combinations of the combined files to obtain the number of threads for backing up the source file.

4. The method according to claim 2, wherein The determining the number of threads for backing up the source file according to the comparison result includes: If the maximum file size is smaller than the file delimiter value, determining the number of threads for backing up the source file to be one; If the maximum file size is greater than or equal to the file delimiter value, a first backup duration for backing up the source file is obtained, and a second backup duration predicted for the source file using a pre-trained backup duration prediction model is obtained, and the number of threads for backing up the source file is determined based on the first backup duration and the second backup duration.

5. The method according to claim 4, wherein The determining the number of threads for backing up the source file according to the first backup duration and the second backup duration includes: Calculate the change rate of the second backup duration compared to the first backup duration; If the duration change rate between the first backup duration and the second backup duration is less than a third threshold, determining the number of threads to be one; If the change rate of the first backup duration and the second backup duration is greater than or equal to a third threshold, the sub-files contained in the source file are divided to obtain a third file set whose file size is greater than or equal to the file delimiter value and a fourth file set whose file size is less than the file delimiter value, and the number of threads for backing up the source file is determined based on the third file set and the fourth file set.

6. The method according to claim 1, wherein The method further comprises: When a file read request is received, a target rsync command is generated, and target file information included in the file read request is obtained; Identifying subfiles of target data corresponding to the target file information to obtain target subfile information of each subfile contained in the target data, wherein the target subfile information includes a target file path and a target file size; Determine the target number of threads for reading the source file based on the target file size in the target subfile information, and generate a target xargs command based on the target number of threads. A thread call is performed according to the target xargs command, and the target rsync command is executed to read a corresponding target sub-file from a storage environment based on the target file path.

7. The method according to claim 1, wherein The step of identifying subfiles of the source file corresponding to the source file information and obtaining subfile information of each subfile contained in the source file includes: Querying the file identifier contained in the source file information to obtain the sub-file identifier contained in the source file information; A pre-recorded relationship list is obtained, and a query is performed in the relationship list according to the sub-file identifier to obtain sub-file information corresponding to each sub-file identifier.

8. A file processing device based on rsync, characterized in that: include: A request receiving module is used to generate an rsync command when receiving a file backup request, and obtain the source file information included in the file backup request; an information identification module, configured to identify subfiles of a source file corresponding to the source file information, and obtain subfile information of each subfile contained in the source file, wherein the subfile information includes a file path and a file size; A thread determination module is used to determine the number of threads for backing up the source file according to the file size in the sub-file information, and generate an xargs command according to the number of threads; The file processing module is used to perform thread calling according to the xargs command, and execute the rsync command to back up each sub-file in the source file to a storage environment according to the file path.

9. An electronic device, characterized in that: The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images