File processing method and device, server and readable storage medium

Abstract

The application provides a file processing method and device, a server and a readable storage medium, and relates to the field of file processing. First, the server acquires the total length of the deleted small files to be processed on a virtual file; then, the server judges whether the data file needs to be migrated according to the total length; if yes, the server migrates the small files not deleted in the data file to a newly created data file, and establishes the mapping relationship between the newly created data file and the virtual file. The data file is a small file pre-created by the server for storage, the data file corresponds to the virtual file displayed on the client, and the URL of the small file is established based on the offset of the small file in the virtual file. Since the offset of the small file in the virtual file does not change before and after the migration, the URL will not change, so that the space utilization of the data file can be improved by migrating the small file, and the URL of the small file does not need to be changed by the client, thereby ensuring the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of file processing, in particular to a file processing method and device, a server and a readable storage medium. BACKGROUND

[0002] The existing file system generally writes multiple small files into a large file, and establishes a URL containing the large file and the position of the small file in the large file for each small file, and sends the URL of the small file to the client, so that the client can read the small file according to the URL.

[0003] When a certain number of small files in the large file are deleted, the client migrates the small files in the large file that are not deleted to a new large file in order, and deletes the original large file, thereby improving the space utilization. Since the large file and the position of the small file in the large file are changed, the URL of the small file is also changed, and the client needs to be notified to update the URL of the small file, which affects the user experience. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the embodiments of the present application provide a file processing method, device, server and readable storage medium, which can change the URL of the small file after migration, so as to not need to notify the client to update the URL of the small file, and guarantee the user experience.

[0005] The technical scheme of the embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides a file processing method applied to a server in a file system, the file system further comprising a client in communication connection with the server, the server pre-creating a data file for storing small files, the data file corresponding to a virtual file displayed on the client, the URL of the small file being established based on the offset of the small file in the virtual file, and the method comprising:

[0007] Obtaining the total length of the deleted small files to be processed on the virtual file;

[0008] According to the total length, determining whether the data file needs to be migrated;

[0009] If yes, migrating the small files in the data file that are not deleted to a newly created data file, and establishing a mapping relationship between the newly created data file and the virtual file.

[0010] Optionally, the server caches a first set and a second set, the first set including all unprocessed deleted small files on the virtual file, and the second set including all processed deleted small files on the virtual file.

[0011] The step of obtaining the total length of the to-be-processed deleted small files on the virtual file includes:

[0012] According to the first set and the second set, at least one of the to-be-processed deleted small files is determined, each of the to-be-processed deleted small files existing in the first set and not existing in the second set.

[0013] The total length is obtained by calculating the sum of the lengths of all the to-be-processed deleted small files.

[0014] Optionally, the method further includes:

[0015] All the to-be-processed deleted small files are added to the second set.

[0016] The first set is emptied.

[0017] Optionally, the method further includes:

[0018] A file processing request for a to-be-processed small file is received from the client.

[0019] According to the file processing request, a target virtual file in which the to-be-processed small file is located and an offset of the target virtual file in the target virtual file are determined.

[0020] A target data file corresponding to the target virtual file is determined.

[0021] According to the offset and the length of each processed deleted small file on the target virtual file, and the offset of the to-be-processed small file in the target virtual file, an offset of the to-be-processed small file in the target data file is determined.

[0022] The to-be-processed small file is processed according to the offset of the to-be-processed small file in the target data file and the length of the to-be-processed small file, and the client is responded.

[0023] Optionally, the file processing request is used for reading or deleting the to-be-processed small file, and the file processing request includes a URL of the to-be-processed small file.

[0024] The step of determining, according to the file processing request, the target virtual file in which the to-be-processed small file is located and the offset of the target virtual file in the target virtual file includes:

[0025] parsing the URL of the to-be-processed small file to obtain a target virtual file where the to-be-processed small file is located and an offset of the target virtual file in the target virtual file.

[0026] Optionally, the file processing request is used for writing the to-be-processed small file, and the step of determining, according to the file processing request, the target virtual file where the to-be-processed small file is located and the offset of the target virtual file in the target virtual file includes:

[0027] If there is a target virtual file with a remaining space greater than the length of the to-be-processed small file, determining the offset of the to-be-processed small file in the target virtual file according to the offset and the length of the last small file in the target virtual file.

[0028] If there is no target virtual file, creating a new virtual file and taking the newly created virtual file as the target virtual file.

[0029] Optionally, the file request is used for writing the to-be-processed small file, and the step of responding to the client includes:

[0030] generating a URL of the to-be-processed small file according to the offset of the to-be-processed small file in the target virtual file.

[0031] returning the URL of the to-be-processed small file to the client to respond to the client.

[0032] In a second aspect, an embodiment of the present application provides a file processing device applied to a server in a file system, the file system further including a client in communication connection with the server, the server pre-creating a data file used for storing small files, the data file corresponding to a virtual file displayed on the client, a URL of the small file being established based on an offset of the small file in the virtual file, and the device including:

[0033] an obtaining module, configured to obtain a total length of to-be-processed deleted small files on the virtual file;

[0034] a processing module, configured to:

[0035] determine, according to the total length, whether the data file needs to be migrated;

[0036] if yes, migrating the non-deleted small files in the data file to a newly created data file and establishing a mapping relationship between the newly created data file and the virtual file.

[0037] In a third aspect, an embodiment of the present application provides a server, comprising a memory and a processor, the memory stores a computer program, and the processor implements the file processing method according to the first aspect when executing the computer program.

[0038] In a fourth aspect, the present application provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the file processing method according to the first aspect.

[0039] Compared with the prior art, the file processing method, device, server and readable storage medium provided by the embodiment of the present application firstly acquire the total length of the deleted small files to be processed on the virtual file by the server; then, the server judges whether the data file needs to be migrated according to the total length; if yes, the server migrates the small files not deleted in the data file to a newly created data file, and establishes a mapping relationship between the newly created data file and the virtual file, wherein the data file is a small file pre-created by the server for storing small files, the data file corresponds to the virtual file displayed on the client, and the URL of the small file is established based on the offset of the small file in the virtual file. Since the offset of the small file in the virtual file does not change before and after the migration of the small file, the URL of the small file will not change, so that the space utilization of the data file can be improved by migrating the small file, and the URL of the small file does not need to be changed by the client, thereby ensuring the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0041] Figure 1 A structural schematic block diagram of a file system provided by an embodiment of the present application;

[0042] Figure 2 A structural schematic block diagram of a server provided by an embodiment of the present application;

[0043] Figure 3 A flowchart of a file processing method provided by an embodiment of the present application Figure 1 ;

[0044] Figure 4 A flowchart of an implementation of step S101 provided by an embodiment of the present application;

[0045] Figure 5A specific example diagram of a data file migration process provided by an embodiment of the present application is provided.

[0046] Figure 6 A flowchart of a file processing method provided by an embodiment of the present application is provided. Figure 2

[0047] Figure 7 A specific example diagram of a change process of a mapping relationship between a virtual file and a data file provided by an embodiment of the present application is provided.

[0048] Figure 8 A schematic diagram of a multi-level cache mechanism about a second set provided by an embodiment of the present application is provided.

[0049] Figure 9 A functional unit block diagram of a file processing apparatus provided by an embodiment of the present application is provided.

[0050] Icon: 100-file system; 110-client; 120-server; 121-memory; 122-processor; 130-metadata server; 200-file processing apparatus; 201-obtaining module; 202-processing module. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0053] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0054] In addition, if the terms "first", "second", etc. are used, they are only used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0055] It should be noted that: in the case of no conflict, the features in the embodiments of the present application can be combined with each other.

[0056] ​Before introducing a file processing method provided by an embodiment of the present application, a file system 100 is introduced first, please refer to Figure 1 The file system includes a client 110, a server 120 and a metadata server 130. The client is in communication connection with the server, and the server is in communication connection with the metadata server.

[0057] The client 110 can show virtual files corresponding to data files on the server, and send a file processing request to the server, so that the server writes, reads and deletes small files (files with a length less than a preset file length, which can be set artificially according to requirements). The client can be a server, a personal computer (PC), a notebook computer, etc. The client 110 can also be one or more program modules on a device, or a virtual machine or a container running on a device. The client can also be a cluster composed of multiple devices, for example, it can be a collective name of multiple program modules distributed on multiple devices.

[0058] The server 120 can receive the file processing request sent by the client, write, read and delete small files, and migrate the small files that have not been deleted in the data file when the total length of the small files to be processed and deleted in the data file reaches a certain value. The server 120 can be a server, a PC, a notebook computer, etc. The server 120 can be a physical server, or a logical server divided from a physical server.

[0059] Further, please refer to Figure 2 , Figure 2 A structural schematic block diagram of the server 120 provided by an embodiment of the present application is shown, which can include a memory 121 and a processor 122.

[0060] The processor 122 can be a general central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the file processing method provided by the following method embodiments.

[0061] The memory 121 can be a ROM, or other type of static storage device that can store static information and instructions; a RAM, or other type of dynamic storage device that can store information and instructions; an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The memory 121 can exist independently and be connected to the processor 122 through a communication bus. The memory 121 can also be integrated with the processor 122. The memory 121 is configured to store machine executable instructions for implementing the solutions of the present application. The processor 122 is configured to execute the machine executable instructions stored in the memory 121 to implement the method embodiments described below.

[0062] The metadata server 130 can be configured to store the mapping relationship between the virtual file displayed on the client and the data file on the server, the offset and length of the processed deleted small file on the virtual file, and the offset and length of the unprocessed deleted small file on the virtual file. The metadata server 130 can also be one or more program modules on a device, or a virtual machine or container running on a device. The metadata server can also be a cluster composed of multiple devices, for example, it can be a collective name of multiple program modules distributed on multiple devices.

[0063] The application applied to the file processing method provided by the embodiments of the present application will be described below. Figure 1 And Figure 2 The file processing method of the server 120 in the embodiments of the present application will be described in detail.

[0064] Please refer to Figure 3 The file processing method includes steps S101-S103.

[0065] S101, obtaining the total length of the deleted small file to be processed on the virtual file.

[0066] The virtual file is displayed on the client 110, and the virtual file corresponds to the data file for storing the small file created by the server 120 in advance, that is, there is a mapping relationship between the two, and the content of the mapping relationship includes the path of the virtual file, the path of the data file, the migration times, the current length of the virtual file, the length of the data file after the last migration, the total length of the processed deleted small file, and the total length of the unprocessed deleted small file.

[0067] The "unprocessed deleted small file" and the "processed deleted small file" are divided according to whether the small file in the data file is migrated or not. It can be understood that the small file deleted after the migration and before the next migration is the unprocessed deleted small file, and the folder deleted before the migration is the processed deleted small file.

[0068] The content of the mapping relationship is stored in the metadata server 130. After the server 120 performs the deletion and migration operations on the small file in the data file, the server 120 sends an update request to the metadata server 130, so that the metadata server 130 updates the specific content of the mapping relationship.

[0069] As shown in FIG. 1, step S101 includes sub-steps S101-1-S101-2. Figure 4

[0070] S101-1, determining at least one to-be-processed deleted small file according to the first set and the second set;

[0071] S101-2, calculating the sum of the lengths of all to-be-processed deleted small files to obtain a total length.

[0072] The server 120 caches the first set and the second set. The first set includes all unprocessed deleted small files on the virtual file, and the second set includes all processed deleted small files on the virtual file. Each to-be-processed deleted small file exists in the first set and does not exist in the second set.

[0073] The first set and the second set cached by the server 120 can be loaded from the metadata server 130. The prerequisite for "loading" is that the server 120 detects that the total length of the unprocessed deleted small files in the virtual file reaches a set value. The detection method can be that the server 120 interacts with the metadata server 130 to obtain the total length of the unprocessed deleted small files recorded in the content of the mapping relationship between the virtual file and the data file stored in the metadata server 130.

[0074] Since the small file on the virtual file can be repeatedly deleted, when the server 120 detects that the total length of the unprocessed deleted small files in the virtual file reaches the set value, the loaded first set and second set need to be intersected to determine whether the intersection is empty. When the intersection is empty, it means that there is no small file that is repeatedly deleted. When the intersection is not empty, it means that there is a small file that is repeatedly deleted.

[0075] In the implementation of the present application, each unprocessed deleted small file in the first set that does not belong to the intersection of the first set and the second set is regarded as a to-be-processed deleted small file.

[0076] ​After all the to-be-processed deleted small files are screened out, the sum of the lengths of all the to-be-processed deleted small files is taken as the total length of the to-be-processed deleted small files.

[0077] S102, judging whether the data file needs to be migrated according to the total length.

[0078] The migration of the data file is determined by judging whether the total length of the to-be-processed deleted small files reaches a preset value. In the embodiment of the application, when the total length of the to-be-processed deleted small files reaches the preset value, step S103 is executed, and when the total length of the to-be-processed deleted small files reaches the preset value, step S101 is returned.

[0079] S103, migrating the non-deleted small files in the data file to a newly created data file, and establishing a mapping relationship between the newly created data file and the virtual file.

[0080] In order to further introduce the above technical solution, the embodiment of the application is further described based on the example shown in Figure 5 .

[0081] As shown in Figure 5 , the virtual file A displayed on the client 110 corresponds to the data file a on the server 120, and the virtual file A writes 6 small files with a length of 1, and the offsets of the 6 small files in the virtual file A are 0, 1, 2, 3, 4 and 5 in turn. Similarly, the offsets of the 6 small files on the data file a are also 0, 1, 2, 3, 4 and 5 in turn.

[0082] The 2nd, 4th and 5th small files (the offsets in the virtual file A are 1, 3 and 4 respectively) are deleted in turn, at this time, the total length of the unprocessed deleted small files recorded in the mapping relationship between the virtual file A and the data file a stored in the metadata server 130 is 3.

[0083] The server 120 detects that the total length of the unprocessed deleted small files reaches a preset value by interacting with the metadata server 130, and loads the first set and the second set stored in the metadata server 130. The first set includes the identifier of each unprocessed deleted small file, and the second set includes the identifier of each processed deleted small file, wherein the identifier of the deleted small file is composed of the name of the virtual file where the small file is located, the offset of the small file in the virtual file and the length of the small file, for example, A-4-1, indicating that the deleted small file is on the virtual file A, the offset of the deleted small file in the virtual file A is 4, and the length of the deleted small file is 1.

[0084] It can be understood that, based on Figure 5In the example, the first set includes three unprocessed deleted small files identified as A-1-1, A-3-1, and A-4-1. Optionally, since the fourth and fifth small files occupy consecutive positions in the virtual file, they can be merged. Therefore, the first set includes unprocessed deleted small files identified as A-1-1 and A-3-2. Since the data file corresponding to virtual file A has not yet been migrated, the second set is empty.

[0085] Since the second set is empty, the identifiers of the deleted small files to be processed are A-1-1 and A-3-2. Therefore, the total length of the deleted small files to be processed is determined to be 3, which also reaches the set value. Data file a needs to be migrated.

[0086] like Figure 5 As shown, three small files that were not deleted are migrated from data file a to data file a' in sequence. Specifically, the small file with offset 0 in virtual file A has offset 0 in data file a', the small file with offset 2 in virtual file A has offset 1 in data file a', and the small file with offset 5 in virtual file A has offset 2 in data file a'.

[0087] After the migration is completed, a mapping relationship between data file a' and virtual file A is established. In one possible implementation, server 120 can request metadata server 130 to modify the data file path in the content of the mapping relationship between virtual file A and data file a stored therein, that is, change the path of data file a to the path of data file a', and change the migration count from 0 to 1, change the length of the data file after the most recent migration from 6 to 3, change the total length of the processed deleted small files from 0 to 3, and change the total length of the unprocessed deleted small files from 3 to 0.

[0088] After completing step S103, the following process can also be executed:

[0089] First, add all pending deleted small files to the second set;

[0090] Then, clear the first set.

[0091] based on Figure 5 In the example, after migrating small files from data file a to data file a', the labels A-1-1 and A-3-2 of the small files to be deleted are added to the second set to identify the small files to be deleted during the next migration. The first set is then cleared to record the identifiers of the small files deleted between the current migration and the next migration.

[0092] Based on the same inventive concept as the above embodiment, in order to enable normal reading, deleting or writing of the small file in the above embodiment, the embodiment of the present application further provides an implementation of processing the small file.

[0093] Please refer to Figure 6 The file processing method further includes steps S201-S202.

[0094] S201, receiving a file processing request for a small file to be processed sent by the client 110.

[0095] The file processing request is used for reading, deleting or writing the small file to be processed.

[0096] S202, determining a target virtual file in which the small file to be processed is located and an offset of the small file to be processed in the target virtual file according to the file processing request.

[0097] For the case that the file processing request is used for reading or deleting the small file to be processed, the file processing request includes a URL of the small file to be processed. The implementation process of step S202 can be as follows:

[0098] The URL of the small file to be processed is parsed to obtain the target virtual file in which the small file to be processed is located and the offset of the small file to be processed in the target virtual file.

[0099] For example, the parsing result of the URL of the small file to be processed in the file processing request sent by the client 110 is A-2-1, that is, the target virtual file in which the small file to be processed is located is virtual file A, the offset of the small file to be processed in virtual file A is 2, and the length of the small file to be processed is 1.

[0100] For the case that the file processing request is used for writing the small file to be processed, the implementation process of step S202 can be as follows:

[0101] If there is a target virtual file with a remaining space greater than the length of the small file to be processed, the offset of the small file to be processed in the target virtual file is determined according to the offset and length of the last small file in the target virtual file;

[0102] If there is no target virtual file, a virtual file is newly created, and the newly created virtual file is used as the target virtual file.

[0103] For example, the length of the small file to be processed is 2, the remaining space of the current virtual file A is 1, the remaining space of virtual file B is 5, and the remaining space of virtual file C is 0. At this time, virtual file B can be used as the target virtual file. The offset of the last small file in virtual file B is 9, and the length is 1. Therefore, the offset of the small file to be processed is 10.

[0104] S203, determine the target data file corresponding to the target virtual file.

[0105] In the embodiment of the application, the server 120 can send a query request to the metadata server 130, so that the metadata server 130 determines the path of the target data file based on the content of the stored mapping relationship between the target virtual file and the target data file.

[0106] For example, the target virtual file is virtual file A, the server 120 sends a query request to the metadata server 130, the metadata server 130 determines that the data file corresponding to the virtual file A is a', and determines the path of the data file a' according to the mapping relationship between the virtual file A and the data file a'.

[0107] S204, determine the offset of the to-be-processed small file in the target data file according to the offset and length of each processed deleted small file on the target virtual file and the offset of the to-be-processed small file in the target virtual file.

[0108] The offset and length of each processed deleted small file on the target virtual file are obtained from a second set of the target virtual file, and the second set is stored in the metadata server.

[0109] For example, the target virtual file where the to-be-processed small file is located is virtual file A, the data file corresponding to the virtual file A is a', the offset of the to-be-processed small file on the virtual file A is 2, the small file with an offset of 1 and a length of 1 and the small file with an offset of 3 and a length of 2 on the virtual file A are processed deleted small files, and the total length of the processed deleted small files before the to-be-processed small file on the virtual file A is 1, so the offset of the to-be-processed small file in the data file a' is "the offset of the to-be-processed small file on the virtual file A" minus "the total length of the processed deleted small files before the to-be-processed small file on the virtual file A", that is, 2-1=1.

[0110] S205, process the to-be-processed small file according to the offset of the to-be-processed small file in the target data file and the length of the to-be-processed small file, and respond to the client.

[0111] For example, the offset of the to-be-processed small file in the data file a' is 1, and the length of the to-be-processed small file is 1, if the file processing request is to read the to-be-processed small file, the data file a' is read from the offset of 1 to the offset of 1+1=2, so as to realize the reading of the to-be-processed small file.

[0112] For the case where the file processing request is used to write the to-be-processed small file, the implementation process of "responding to the client" in step S205 is as follows:

[0113] First, a URL of the to-be-processed small file is generated according to an offset of the to-be-processed small file in the target virtual file;

[0114] Then, the URL of the to-be-processed small file is returned to the client to respond to the client 110.

[0115] To introduce the above technical solutions in more detail, embodiments of the present application are based on Figure 7 as shown in the examples.

[0116] As Figure 7 shown, the data file corresponding to the virtual file A has migrated twice, wherein, before the first migration, the data file corresponding to the virtual file A is a, after the first migration, the data file corresponding to the virtual file A is a', and after the second migration, the data file corresponding to the virtual file A is a".

[0117] Before the first migration, the small files with offsets 1, 3, 4 and 7 on the virtual file A are deleted, and after the first migration, the small files with offsets 1, 3, 4 and 7 on the virtual file A are deleted as processed deleted small files.

[0118] After the first migration, three small files with a length of 1 are written on the virtual file A, and the offsets of the three newly written small files on the virtual file A are 9, 10 and 11.

[0119] According to the offsets and lengths of all processed deleted small files, it is determined that the total length of the processed deleted small files located before each newly written small file on the virtual file A is 4, and then the offsets of the three newly written small files on the data file a' are calculated as 9-4=5, 10-4=6 and 11-4=7.

[0120] The small files with offsets 5 and 10 on the virtual file A are deleted, and according to the offsets and lengths of all processed deleted small files, it is determined that the total length of the processed deleted small files located before the small file with offset 5 to be deleted on the virtual file A is 3, and the total length of the processed deleted small files located before the small file with offset 10 to be deleted on the virtual file A is 4, and then the offsets of the two small files to be deleted on the data file a' are calculated as 5-3=2 and 10-4=6 respectively.

[0121] After the second migration, the small files with offsets 5 and 10 on the virtual file A are deleted as processed deleted small files.

[0122] As Figure 7 can be seen, no matter how many times the data file corresponding to the virtual file A is migrated, the offset of the small file on the virtual file will not be affected, and thus the client does not need to change the URL of the small file.

[0123] Optionally, for the case of small file reading, the client 110 can load the second set including each processed deleted small file on the virtual file to the local memory from the server 120, and the client 110 can determine the offset of the small file to be read in the target data file according to the target virtual file determined according to the URL of the small file to be read, the offset of the small file in the target virtual file, and the second set of the target virtual file cached in the local memory. Therefore, the embodiment of the application also provides a multi-level cache mechanism for the second set as shown in Figure 8 .

[0124] If the second set of the target virtual file is not cached in the local memory of the client 110, the second set of the target virtual file is loaded from the server 120, and if the second set of the target virtual file is not cached in the server 120, the second set of the target virtual file is loaded from the metadata server 130, so as to effectively balance the transmission frequency between the server 120 and the metadata server 130 and the transmission frequency between the client 110 and the server 120, and further improve the reading performance of the small file.

[0125] Compared with the prior art, the embodiment of the application has the following effects:

[0126] (1) The server 120 pre-creates a data file for storing small files, the data file corresponds to a virtual file displayed on the client 110, and the URL of the small file is established based on the offset of the small file in the virtual file, so that when the data file is migrated, the client 110 does not need to be notified to change the URL, and the user experience is guaranteed.

[0127] (2) For the case of small file reading, a multi-level cache for the second set is established to balance the transmission frequency between the server 120 and the metadata server 130 and the transmission frequency between the client 110 and the server 120, and further improve the reading performance of the small file.

[0128] In order to perform the above-mentioned method embodiments and corresponding steps in various possible implementations, an implementation of a file processing device is given as follows. Please refer to Figure 9 , the file processing device 200 is applied to the server in Figure 1 and Figure 2 , and can include an acquisition module 201 and a processing module 202.

[0129] The acquisition module 201 is used to acquire the total length of the deleted small file to be processed on the virtual file.

[0130] The processing module 202 is used to determine whether the data file needs to be migrated according to the total length, if yes, migrate the non-deleted small file in the data file to a newly created data file, and establish a mapping relationship between the newly created data file and the virtual file.

[0131] Optionally, the server caches a first set and a second set, the first set including all unprocessed deleted small files on the virtual file, and the second set including all processed deleted small files on the virtual file, and the processing module 202 is specifically configured to determine at least one to-be-processed deleted small file according to the first set and the second set, each to-be-processed deleted small file existing in the first set and not existing in the second set; and calculate a sum of lengths of all to-be-processed deleted small files to obtain a total length.

[0132] Optionally, the processing module 202 is further configured to add all to-be-processed deleted small files into the second set; and empty the first set.

[0133] Optionally, the processing module 202 is further configured to receive a file processing request for the to-be-processed small file sent by the client; determine a target virtual file in which the to-be-processed small file is located and an offset of the target virtual file in the target virtual file according to the file processing request; determine a target data file corresponding to the target virtual file; determine an offset of the to-be-processed small file in the target data file according to an offset and a length of each processed deleted small file on the target virtual file and the offset of the to-be-processed small file in the target virtual file; process the to-be-processed small file according to the offset of the to-be-processed small file in the target data file and the length of the to-be-processed small file, and respond to the client.

[0134] Optionally, the file processing request is used for reading or deleting the to-be-processed small file, and the file processing request includes a URL of the to-be-processed small file, and the processing module 202 is specifically configured to parse the URL of the to-be-processed small file to obtain the target virtual file in which the to-be-processed small file is located and the offset of the target virtual file in the target virtual file when determining the target virtual file in which the to-be-processed small file is located and the offset of the target virtual file in the target virtual file according to the file processing request.

[0135] Optionally, the file processing request is used for writing the to-be-processed small file, and the processing module 202 is specifically configured to, when determining the target virtual file in which the to-be-processed small file is located and the offset of the target virtual file in the target virtual file according to the file processing request, determine the offset of the to-be-processed small file in the target virtual file according to an offset and a length of a last small file in the target virtual file if there is a target virtual file with a remaining space greater than the length of the to-be-processed small file; and create a new virtual file as the target virtual file if there is no target virtual file.

[0136] Optionally, the file processing request is used for writing the to-be-processed small file, and the processing module 202 is specifically configured to generate a URL of the to-be-processed small file according to the offset of the to-be-processed small file in the target virtual file when responding to the client; and return the URL of the to-be-processed small file to the client to respond to the client.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the file processing apparatus 200 described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0138] The embodiment of the present application further provides a computer readable storage medium containing a computer program, which can be used to execute the related operations in the file processing method provided by the foregoing method embodiments when executed.

[0139] To sum up, the embodiment of the present application provides a file processing method, device, server and readable storage medium, first, the server acquires the total length of the deleted small files to be processed on the virtual file; then, the server judges whether the data file needs to be migrated according to the total length; if yes, the server migrates the small files not deleted in the data file to a newly created data file, and establishes the mapping relationship between the newly created data file and the virtual file, wherein the data file is a small file pre-created by the server for storing small files, the data file corresponds to the virtual file displayed on the client, and the URL of the small file is established based on the offset of the small file in the virtual file, so that the space utilization of the data file can be improved by migrating the small file, and the URL of the small file does not need to be changed by the client, thereby ensuring the user experience.

[0140] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A file processing method, characterized in that, A server-side application in a file system, the file system also including a client communicating with the server, the server pre-creating a data file for storing small files, the data file corresponding to a virtual file displayed on the client, the URL of the small file being established based on the offset of the small file in the virtual file, the method including: Obtain the total length of the small files to be deleted on the virtual file; Based on the total length, determine whether the data file needs to be migrated; If so, migrate the small files that were not deleted from the data file to the newly created data file, and establish a mapping relationship between the newly created data file and the virtual file.

2. The method as described in claim 1, characterized in that, The server-side cache has a first set and a second set. The first set includes all unprocessed deleted small files on the virtual file, and the second set includes all processed deleted small files on the virtual file. The step of obtaining the total length of the small files to be deleted on the virtual file includes: Based on the first set and the second set, at least one of the pending deleted small files is determined, and each pending deleted small file exists in the first set but not in the second set; The total length is obtained by summing the lengths of all the small files to be deleted.

3. The method as described in claim 2, characterized in that, The method further includes: Add all the pending deleted small files to the second set; Clear the first set.

4. The method as described in claim 1, characterized in that, The method further includes: Receive file processing requests for small files to be processed sent by the client; Based on the file processing request, determine the target virtual file where the small file to be processed is located and the offset of the target virtual file in the target virtual file; Determine the target data file corresponding to the target virtual file; Based on the offset and length of each processed deleted small file on the target virtual file, and the offset of the small file to be processed in the target virtual file, determine the offset of the small file to be processed in the target data file; The small file to be processed is processed according to its offset in the target data file and its length, and the client is responded to.

5. The method as described in claim 4, characterized in that, The file processing request is used to read or delete the small file to be processed, and the file processing request includes the URL of the small file to be processed; The step of determining the target virtual file containing the small file to be processed and the offset of the target virtual file within the target virtual file according to the file processing request includes: The URL of the small file to be processed is parsed to obtain the target virtual file where the small file to be processed is located and the offset of the target virtual file in the target virtual file.

6. The method as described in claim 4, characterized in that, The file processing request is used to write the small file to be processed. The step of determining the target virtual file where the small file to be processed is located and the offset of the target virtual file in the target virtual file according to the file processing request includes: If there is a target virtual file with remaining space greater than the length of the small file to be processed, then the offset of the small file to be processed in the target virtual file is determined based on the offset and length of the last small file in the target virtual file. If the target virtual file does not exist, a new virtual file is created, and the newly created virtual file is used as the target virtual file.

7. The method as described in claim 4, characterized in that, The file request is used to write the small file to be processed, and the steps of responding to the client include: Generate the URL of the small file to be processed based on its offset in the target virtual file; The URL of the small file to be processed is returned to the client in response to the client.

8. A document processing device, characterized in that, A server-side application in a file system, the file system further including a client communicating with the server, the server pre-creating a data file for storing small files, the data file corresponding to a virtual file displayed on the client, the URL of the small file being established based on the offset of the small file in the virtual file, the device comprising: The acquisition module is used to acquire the total length of the small files to be deleted on the virtual file. Processing module, used for: Based on the total length, determine whether the data file needs to be migrated; If so, migrate the small files that were not deleted from the data file to the newly created data file, and establish a mapping relationship between the newly created data file and the virtual file.

9. A server-side component, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the file processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the file processing method as described in any one of claims 1 to 7.

Images