A data processing method and storage system

Abstract

The present application provides a data processing method and storage system, which relate to the field of storage. The method includes: a metadata server receives a first instruction from a recycle bin management server; based on the first instruction, the first data is identified as an abnormal state; when the system overhead meets a preset condition, the first data identified as an abnormal state is cleared from the recycle bin. Based on the data processing method provided by the present application, expired but not cleared data is identified as an abnormal state and is not included in the tenant's billing space, and when the system overhead meets the preset condition, the data identified as an abnormal state is cleared. Therefore, the strategy of clearing expired data in the recycle bin is optimized, while reducing the impact on system performance, without increasing the tenant's storage space occupation time, avoiding unnecessary expenses for the tenant.

Description

Technical Field

[0001] The present application relates to the field of storage, and in particular to a data processing method and a storage system. Background Art

[0002] When tenants use cloud space to store data, they may accidentally delete data. In order to prevent tenants from accidentally deleting data and causing tenant data loss, a recycle bin function is set up in the cloud space. Data deleted by tenants will go to the recycle bin, and tenants can also restore accidentally deleted data from the recycle bin. At the same time, the recycle bin also has an automatic cleanup function. When the data is retained in the recycle bin for a period of time exceeding the time set by the tenant, the recycle bin will automatically clean up the expired data.

[0003] After the recycle bin is enabled, the storage system still charges the storage space occupied by the data in the recycle bin at the normal storage price. After the data is cleared from the storage space of the recycle bin, the storage space is released and the billing stops. Therefore, if the recycle bin does not clear expired data in time, it will occupy the tenant's billing space and add extra costs to the tenant. Therefore, the recycle bin needs to clear expired data in real time. However, clearing the recycle bin data requires modifying the metadata of the file system, which will also trigger the deletion of metadata and data, and will occupy a certain amount of system overhead. If the background is also clearing the recycle bin data while the user is accessing the system, the system overhead will be very large and the system performance will be affected, such as affecting the processing of data link creation, deletion and rewrite requests.

[0004] Therefore, a method needs to be provided to optimize the strategy of cleaning up expired data in the recycle bin and minimize the impact of automatic cleaning of expired data on tenants. Summary of the invention

[0005] The present application provides a data processing method and a storage system, in order to optimize the strategy of cleaning up expired data in a recycle bin and minimize the impact of automatic cleaning up of expired data on tenants.

[0006] In a first aspect, the present application provides a data processing method, which is applied to a storage system, wherein the storage system is used to provide data storage services for one or more tenants, and the storage system includes a recycle bin management server and a metadata server, and the method includes: the metadata server receives a first instruction from the recycle bin management server, the first instruction is used to instruct to identify first data in a recycle bin of a first tenant as being in an abnormal state; wherein the first data is data that has been stored in the recycle bin for a preset period of time but has not been cleared, and the storage space occupied by the data identified as being in an abnormal state in the recycle bin is not included in the billing space of the first tenant; the metadata server identifies the first data as being in an abnormal state based on the first instruction; and the metadata server clears the first data identified as being in an abnormal state from the recycle bin when the system overhead meets a preset condition.

[0007] In a second aspect, the present application provides a data processing method, the method comprising: a recycle bin management server generates a first instruction based on a time index of first data; wherein the first data is data that has been stored in a recycle bin of a first tenant for a preset period of time but has not been cleaned up, and the first instruction is used to indicate that the first data is identified as an abnormal state, and the storage space occupied by the data identified as an abnormal state in the recycle bin is not included in the billing space of the first tenant; the recycle bin management server sends the first instruction to the metadata server.

[0008] In a third aspect, the present application provides a data processing method, applied to a metadata server, the method comprising: the metadata server receives a first instruction from a recycle bin management server, the first instruction being used to instruct that first data in a recycle bin of a first tenant be identified as being in an abnormal state; wherein the first data is data that has been stored in the recycle bin for a preset period of time but has not been cleaned up, and the storage space occupied by the data identified as being in an abnormal state in the recycle bin is not included in the billing space of the first tenant; the metadata server identifies the first data as being in an abnormal state based on the first instruction; and the metadata server cleans up the first data identified as being in an abnormal state from the recycle bin when the system overhead meets a preset condition.

[0009] In a fourth aspect, the present application provides a storage system for providing data storage services for one or more tenants, the storage system comprising a recycle bin management server and a metadata server; wherein the recycle bin management server is used to send a first instruction to the metadata server, the first instruction being used to instruct that first data in a recycle bin of a first tenant be identified as being in an abnormal state; wherein the first data is data that has been stored in the recycle bin for a preset period of time but has not been cleaned up, and the storage space occupied by the data identified as being in an abnormal state in the recycle bin is not included in the billing space of the first tenant; the metadata server is used to identify the first data as being in an abnormal state based on the first instruction; and is used to clean up the first data from the recycle bin based on a second instruction when the system overhead meets a preset condition.

[0010] In a fifth aspect, the present application provides a metadata server for implementing the method described in the second aspect.

[0011] In a sixth aspect, the present application provides a recycling bin management server for implementing the method described in the third aspect.

[0012] In a seventh aspect, the present application provides an electronic device, comprising a processor, wherein the processor is used to call a computer program (also referred to as code, or instruction) so that the electronic device implements the method described in any one of the first to third aspects.

[0013] In an eighth aspect, the present application provides a computer-readable storage medium, comprising a computer program, which, when executed on a computer, enables the computer to execute the method described in any one of the first to third aspects.

[0014] In a ninth aspect, the present application provides a computer program product, comprising: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method described in any one of the first to third aspects.

[0015] In an embodiment of the present application, the first data that has expired but not been cleared is marked as abnormal through the metadata server and is not included in the tenant's paid space. When the system overhead meets the preset conditions, the data marked as abnormal is cleared. Therefore, the strategy of cleaning up expired data in the recycle bin is optimized. While reducing the impact on system performance, it does not increase the tenant's occupancy time of the storage space, thereby avoiding unnecessary expenses for the tenant. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the structure of a storage system provided in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of a process for deleting metadata provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of a process for releasing space provided in an embodiment of the present application;

[0019] Figure 4 A schematic diagram of the structure of the log structure merge tree LSM-Tree provided in an embodiment of the present application;

[0020] Figure 5 A schematic flow chart of a data processing method provided in an embodiment of the present application;

[0021] Figure 6 A schematic diagram of the process flow of the recycle bin management service and metadata service provided in the embodiment of the present application;

[0022] Figure 7 A schematic diagram of a process for querying the directory structure of a recycle bin provided in an embodiment of the present application;

[0023] Figure 8 A schematic block diagram of a recycle bin management server provided in an embodiment of the present application;

[0024] Fig. 9 A schematic block diagram of a metadata server provided in an embodiment of the present application;

[0025] Fig.10A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them.

[0027] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0028] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, the term "includes" and any variations thereof are intended to cover non-exclusive inclusions.

[0029] Figure 1 This is a schematic diagram of the structure of the storage system provided in the embodiment of the present application. Figure 1 As shown, the storage system 110 includes a recycle bin management server 111 and a metadata server 112. The recycle bin management server 111 can be used to generate instructions to instruct the metadata server 112 to execute a clearing action of metadata in the recycle bin; the metadata server 112 includes a key-value (KV) database, which is a database that stores data in key-value pairs, each key corresponds to a unique value, and is suitable for querying data through a primary key.

[0030] In an embodiment of the present application, the KV database stores data of each tenant, for example, including but not limited to metadata of the file system and metadata of the recycle bin. Different tenants rent different storage spaces and use different file systems and recycle bins. Of course, the same tenant can also use multiple file systems, which is not limited in this application. In the following, for the sake of distinction and explanation, the file system and the recycle bin are distinguished based on the tenant granularity.

[0031] It should be understood that Figure 1 In the figure, the recycle bin management server 111 and the metadata server 112 are shown separately for the sake of distinction. In an actual scenario, the recycle bin management server 111 and the metadata server 112 can be two independent physical devices, or two different logical partitions on the same physical device. This embodiment of the application does not limit this.

[0032] In order to prevent the data stored by tenants in the storage system from being lost due to accidental deletion, a recycle bin function is designed in the storage system so that tenants can restore the accidentally deleted data. At the same time, the recycle bin also has an automatic cleanup function. When the data is retained in the recycle bin for more than the preset time, the recycle bin management server will send a command to the metadata server to instruct the metadata server to clear the expired data.

[0033] Currently, the storage system charges the storage space occupied by the data in the recycle bin at the normal storage price. After the data is cleared from the storage space of the recycle bin, the storage space is released and the charging stops. Therefore, if the recycle bin does not clear the expired data in time, it will occupy the tenant's billing space and add extra costs to the tenant. Therefore, the recycle bin needs to clear the expired data in real time. However, clearing the recycle bin data requires modifying the metadata of the file system, and will also trigger the deletion of metadata and data, which will occupy a certain amount of system overhead.

[0034] Cleaning up the Recycle Bin data includes two processes: deleting metadata and freeing up space. The process of deleting metadata is as follows: Figure 2 As shown, the process of freeing up space is as follows Figure 3 shown.

[0035] Specifically, the process of deleting metadata may include:

[0036] Step 201, deleting file object metadata;

[0037] Step 202, delete the file object index record;

[0038] Step 203, modify the modification time and statistical information of the parent directory;

[0039] Step 204, modifying file system statistics;

[0040] Step 205: Record the file deletion event.

[0041] The process of freeing up space can specifically include:

[0042] Step 301, obtaining a file deletion event;

[0043] Step 302, determining the storage location of the data according to the file layout record;

[0044] Step 303, deleting data;

[0045] Step 304, modify the storage management record of the system;

[0046] Step 305: Delete the layout record of the file.

[0047] Currently, disks widely use the log structured merge tree (LSM-Tree) structure to store data. The structure of LSM-Tree is as follows: Figure 4 As shown. The structure shows random access memory (RAM) and disk, where the disk can be further divided into multiple levels, such as level 0, level 1 and level 2 as shown in the figure. The total size of files in each layer is predefined, and the size of the next layer is larger than the previous layer. When the memory is full, the files in the memory will be written to the disk. The files in the disk are merged downward layer by layer. Once the total size of files in a certain layer exceeds a predefined threshold, a file can be selected to merge with the files in the next layer (compaction).

[0048] In the above two processes, the modification and deletion operations, i.e., steps 201, 202, 203, 204, 303, 304, and 305, will be written to the memory table (memtable) in RAM, and the data will be actually modified or deleted only after multiple merges. The query request needs to traverse the memtable in RAM, the sorted string table (SStable) of level 0 on the disk, and the lower SStable containing the query data. Therefore, if the background is also cleaning up the recycle bin data while the user is accessing the system, the system overhead will be very large and the system performance will be affected, such as affecting the processing of data link creation, deletion, and rewrite requests.

[0049] In view of this, the present application provides a data processing method. When data in the recycle bin expires, on the one hand, the expired data is marked and not included in the tenant's billing space. On the other hand, the expired data is cleared only when the storage system enters a low period. In this way, the strategy of clearing expired data in the recycle bin is optimized, and while reducing the impact on system performance, it does not increase the tenant's storage space occupation time, avoiding unnecessary expenses for the tenant.

[0050] The data processing method and storage system provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0051] It should be noted that the data processing method provided in the embodiment of the present application can be applied to a storage system. The storage system can be used to provide storage services for one or more tenants, and each tenant can use the storage space provided by the storage system to store their own data. The storage system may include a recycle bin management server and a metadata server. The following embodiment will take the interaction between the two as an example to describe the method provided in the embodiment of the present application.

[0052] For the convenience of distinction and explanation, the first data of the first tenant is used as an example to describe the method provided by the embodiment of the present application. The first data is data that has been stored in the recycle bin for a preset period of time but has not been cleaned up. Simply put, the first data is expired and uncleaned data. It can be understood that the first data can be data from any tenant. It is defined as the data of the first tenant in the text for the convenience of distinction.

[0053] Figure 5 The following is a schematic flow chart of a data processing method provided in an embodiment of the present application. Figure 5 As shown, the method may include steps 501 to 504. Figure 2 Detailed description of each step.

[0054] In step 501, the recycle bin management server generates a first instruction according to a time index of first data.

[0055] As mentioned above, the first data is data that has been stored in the recycle bin for a preset period of time but has not been cleaned up. The preset period of time may be set by the tenant or by the system default, and this application does not limit this. The recycle bin management server may determine whether the data is expired and not cleaned up based on the time when the data was placed in the recycle bin and the preset period of time.

[0056] Here, data being put into the recycle bin may mean that the user deletes the data from the file system. In order to avoid accidental deletion, the system automatically moves the data into the recycle bin for temporary storage for a period of time. In an embodiment of the present application, the operation of deleting data from the file system can be recorded. For example, the time when data 1 (i.e., an example of the first data) is put into the recycle bin is 1:00 on June 15, 2021, which can be recorded as 2021061501; for another example, the time when data 2 (i.e., another example of the first data) is put into the recycle bin is 2:00 on June 15, 2021, which can be recorded as 2021061502.

[0057] Since the tenant may delete unnecessary data at any time, that is, new data may be stored in the recycle bin at any time, the recycle bin management server may periodically count the data in the recycle bin to determine the newly generated expired and uncleaned data.

[0058] In an embodiment of the present application, the time index of the first data may be related to the time when the first data is placed in the recycle bin. Each time index may correspond to a time node. In this way, the recycle bin management server may generate a first instruction based on the time index of the first data, so as to instruct the metadata server to mark the first data as abnormal through the first instruction.

[0059] Here, the time index can be created with different time lengths as granularity. For example, with hours as the granularity, a time index is created for each hour, corresponding to the data put into the recycle bin within the hour.

[0060] In one example, data 1 was put into the recycle bin at 1:00 on June 15, 2021, and the corresponding time index is 1024; data 2 was put into the recycle bin at 2:00 on June 15, 2021, and the corresponding time index is 1025.

[0061] In the embodiment of the present application, the granularity of the time index can be the same as the billing method. For example, if the billing for the tenant's use of storage space is charged in hours, the granularity of the time index can be hours. Of course, the billing for the tenant's use of storage space can also be charged in units of other time lengths, and the time index can also be distinguished by other time lengths as granularity, which is not limited in the present application.

[0062] Optionally, step 501 includes: the recycle bin management server determines the time index of the first data based on a preset duration and a first mapping relationship, the first mapping relationship includes a correspondence between at least one time index and at least one time, and the time in at least one time represents the time when the data under the corresponding time index is placed into the recycle bin; the recycle bin management server generates a first instruction based on the time index of the first data.

[0063] Table 1 shows an example of the first mapping relationship.

[0064] Table 1

[0065] Time Index time 1024 2021061501 1025 2021061502

[0066] It should be understood that the first mapping relationship shown in Table 1 is given based on the data 1 and data 2 in the above example, but this should not constitute any limitation to the present application. The present application does not limit the specific content of the first mapping relationship.

[0067] Taking the mapping relationship between time index 1024 and time 2021061501 as an example, assuming that the preset duration is 1 day, at 1:00 on June 16, 2021, the storage time of data 1 with time 2021061501 in the recycle bin reaches the preset duration. The recycle bin management server determines the time index 1024 of data 1 from time 2021061501 according to the first mapping relationship, and then generates a first instruction based on the time index 1024 of data 1.

[0068] It should be noted that the above-mentioned first mapping relationship can be recorded in the metadata server, for example, it can be the metadata of the recycle bin. The recycle bin management server can read the relevant data in the metadata server through the access interface provided by the metadata server, and then determine the time index of the first data, and then generate the first instruction according to the time index of the first data. The storage space occupied by the data identified as abnormal in the recycle bin is not included in the billing space of the first tenant. In other words, the storage space occupied by expired and uncleaned data is not included in the billing space of the tenant. In other words, the tenant does not need to pay extra for expired and uncleaned data. In contrast, the storage space occupied by the data identified as normal in the recycle bin is included in the billing space of the first tenant, and the tenant needs to pay for it. Here, the normal data is the data in the recycle bin that has not expired, that is, the data that has not been stored in the recycle bin for a preset period of time.

[0069] In step 502, the recycle bin management server sends the first instruction to the metadata server. Accordingly, the metadata server receives the first instruction from the recycle bin management server.

[0070] In step 503, the metadata server identifies the first data as being in an abnormal state based on the first instruction.

[0071] The recycle bin management server may trigger the metadata server to identify the first data as being in an abnormal state by sending the first instruction to the metadata server.

[0072] Optionally, step 503 includes: based on the first instruction, the metadata server identifies the state of the first data in the second mapping relationship as an abnormal state, the second mapping relationship includes a correspondence between at least one time index and at least one state, the at least one state includes a normal state and an abnormal state, and the storage space occupied by the data identified as a normal state in the recycle bin is included in the billing space of the first tenant.

[0073] In one example, the second mapping relationship is a mapping relationship between time index 1024 and state. Before the metadata server receives the first instruction, the second mapping relationship of data 1 is: time index 1024, state "normal state"; after the metadata server receives the first instruction, the second mapping relationship of data 1 is changed to: time index 1024, state "abnormal state".

[0074] Table 2 shows an example of the second mapping relationship.

[0075] Table 2

[0076] Time Index state 1024 Abnormal 1025 Normal

[0077] It should be understood that the first mapping relationship and the second mapping relationship may be stored as two independent mapping relationships, or may be stored as one mapping relationship. For example, Table 3 shows another example of the first mapping relationship and the second mapping relationship.

[0078] Table 3

[0079] Time Index state time 1024 Abnormal 2021061501 1025 Normal 2021061502

[0080] Optionally, the first instruction also indicates the amount of storage space occupied by the first data; after step 501, the method further includes: the metadata server excludes the amount of storage space occupied by the data to be cleaned from the amount of storage space occupied by the file system and recycle bin of the first tenant to obtain the billing space of the first tenant.

[0081] Here, the data to be cleaned up may include the first data determined to be expired in the above step 501, and the data that has been determined to be expired but has not been cleaned up before. Since the storage time of these data in the recycle bin reaches or exceeds the preset time length, they should be cleaned up. However, due to the introduction of the mechanism introduced in this application, the metadata server will only clean it up when the system overhead meets the preset conditions, so the storage time in the recycle bin may exceed or has exceeded the preset time length. In order to avoid unnecessary expenses for tenants, the storage space occupied by the data to be cleaned up is excluded from the billing space of the first tenant. That is, the billing space of the tenant = the storage space occupied by the tenant's file system + the storage space occupied by the recycle bin - the storage space occupied by the data to be cleaned up.

[0082] In one example, the unit of storage space is gigabyte (GB). If the first instruction indicates that data 1 occupies 1GB of storage space; the first tenant's file system occupies 10GB of storage space, the recycle bin occupies 5G of storage space, and the data to be cleaned up counted by the metadata processor before receiving the first instruction occupies 2GB of storage space. After receiving the first instruction, the metadata processor can add the 1GB of storage space occupied by data 1 to the 2GB of storage space occupied by the data to be cleaned up that was previously counted, and obtain 3GB. Then the billing space for the first tenant is: 10GB+5GB-3GB=12GB.

[0083] Optionally, the abnormal state includes a freeze state and a cleanup state.

[0084] It should be understood that both the frozen state and the cleanup state are abnormal states, and the storage space occupied by the data marked as frozen state or cleanup state is not included in the billing space. In the embodiment of the present application, the frozen state and the cleanup state are used to distinguish the states before and after metering.

[0085] Exemplarily, step 503 includes: before the metadata server excludes the storage space occupied by the first data from the storage space occupied by the file system and recycle bin of the first tenant, the metadata server identifies the state of the first data as a frozen state; after the metadata server excludes the storage space occupied by the first data from the storage space occupied by the file system and recycle bin of the first tenant, the metadata server identifies the state of the first data as a clean state.

[0086] For example, after receiving the first instruction, the metadata server marks the time index 1024 of data 1 as frozen in the second mapping relationship before executing the action of cleaning data 1. The metadata server adds the storage space occupied by data 1 to the storage space occupied by the data to be cleaned, and after updating the billing space, marks the time index 1024 of data 1 in the second mapping relationship as cleaned.

[0087] In this way, the atomicity of the metadata processor's processing of the first data can be ensured, and when a replay is requested, repeated accumulation of the storage space occupied by the first data can be avoided, which would lead to an error in the measurement result of the billing space.

[0088] It should be understood that the frozen state and the cleanup state are only two states introduced to facilitate the distinction between before and after billing, and should not constitute any limitation to this application. This application does not exclude the possibility of introducing other states to distinguish between the two states before and after billing.

[0089] In step 504, when the system overhead meets a preset condition, the first data identified as being in an abnormal state is cleared from the recycle bin.

[0090] Since cleaning data from the recycle bin requires a certain amount of system overhead, thus affecting the creation, deletion, and read and write functions of data links in the storage system, data cleaning is best performed during the low-volume period of the storage system.

[0091] Whether the storage system is in a low period can be determined based on the system overhead. If the system overhead is low, it can be considered to be in a low period. Therefore, the metadata server can monitor the current system overhead in real time, and when the system overhead is low, the data identified as abnormal state is cleared from the recycle bin. Here, the data identified as abnormal state includes the first data.

[0092] As mentioned above, in order to distinguish the states before and after metering, the frozen state and the cleanup state are introduced. It is understandable that since the data needs to be metered in the billing space before being cleaned, when the frozen state and the cleanup state are introduced, the cleaned data is usually the data in the cleanup state.

[0093] Here, the system overhead includes the foreground system overhead and the background system overhead. The foreground system overhead specifically includes the system overhead caused by user access, such as the system overhead caused by operations such as writing, deleting, modifying, and querying; the background system overhead specifically includes the system overhead caused by the aforementioned clearing of expired data in the recycle bin.

[0094] The metadata server can clear the data to be cleared from the recycle bin when the foreground system overhead and / or the background system overhead is small.

[0095] Exemplarily, the foreground system overhead can be measured by the foreground input output per second (IOPS), the foreground query per second (QPS), etc. For example, a threshold is set for the foreground IOPS or QPS, for example, the foreground IOPS corresponds to a first preset threshold, and the foreground QPS corresponds to a second preset threshold.

[0096] If the foreground IOPS is used as the judgment basis, the metadata server can determine that the foreground system overhead is smaller when the foreground IOPS is lower than the first preset threshold; if the foreground QPS is used as the judgment basis, the metadata server can determine that the foreground system overhead is smaller when the foreground QPS is lower than the second preset threshold.

[0097] It should be understood that foreground IOPS and foreground QPS are merely two examples used to characterize the system overhead of the foreground, and the present application includes but is not limited to them.

[0098] The system overhead of the background can be measured by the amount of unmerged data. For example, a third preset threshold is set for the amount of unmerged data. If the amount of unmerged data is lower than the third preset threshold, the metadata database determines that the system overhead of the background is small.

[0099] The metadata server can clean up the data to be cleaned up from the recycle bin when the system overhead of either the foreground or the background is small, or can clean up the data to be cleaned up from the recycle bin when the system overhead of both the foreground and the background is small.

[0100] That is, the above-mentioned preset conditions include one or more of the following: the input and output volume of the foreground per second is lower than the first preset threshold, the query rate of the foreground per second is lower than the second preset threshold, and the amount of unmerged data in the memory is higher than the third preset threshold. As for which one or several items to select in actual application, and whether to add more preset conditions, it can be set by the staff, and this application does not impose any restrictions on this. The metadata server can execute the above step 204 according to the status of each data in the second mapping relationship, or it can execute the above step 204 in response to the call of the recycle bin management server.

[0101] Optionally, before step 504, the method also includes: the recycle bin management server generates a second instruction based on N data to be cleaned, the second instruction is used to instruct to clean up N data to be cleaned, the N data to be cleaned include the first data, and N is a preset value; the recycle bin management server sends the second instruction to the metadata server.

[0102] Since the recycle bin management server may determine one or more expired and uncleaned data every time it performs statistics, these expired and uncleaned data can be identified as abnormal and become data to be cleaned after being determined. If instructions are sent for each data separately, it may cause a large signaling overhead. Therefore, the cleaning requests for N data to be cleaned can be aggregated into one instruction and sent to the metadata server. That is, the recycle bin management server can generate a second instruction based on the N data to be cleaned to instruct the cleaning of N data to be cleaned. Where N can be the system default or the tenant setting, and this application does not limit this.

[0103] It should be noted that data may include data in a file and data in a directory. A data may specifically refer to data in a file (or data in a file directory) or data in a directory.

[0104] It should also be noted that the recycling bin management server marking the first data as abnormal and the recycling bin management server calling the metadata server to clean up the first data can be regarded as two independent processes, and the two are not coupled to each other.

[0105] An example, Figure 6 The figure is a flow chart of the recycle bin management service and the metadata service, wherein the recycle bin management service can be provided by the recycle bin management server, the metadata service can be provided by the metadata server, and the metadata server can provide the metadata service in response to the call of the recycle bin management server.

[0106] In the Recycle Bin Management Service process, first select a file system to clean up, then obtain the hourly directory and extract the directory being cleaned, then obtain the files and subdirectories under the directory to be cleaned, and finally call the metadata service to perform the cleanup. In the metadata service process, first receive the file / subdirectory cleanup request, then determine whether it can be cleaned based on the status of the hourly directory to which it belongs, then determine whether the cleanup request can be executed based on the QPS of the current foreground request and the number of level 0 files and other indicators, finally clean up the files / subdirectories and return the cleanup results to the Recycle Bin Management Server. If the cleanup is successful, you can repeatedly call the metadata service to perform the cleanup to ensure that the cleanup is successful; if the system is busy and the cleanup fails, select another file system to clean up.

[0107] In the above steps, the hourly directory may be the content shown in Table 1, Table 2 or Table 3.

[0108] Optionally, the second instruction carries data identifiers of N data to be processed. In an embodiment of the present application, the data identifiers may include: a time index node (inode), a parent directory index node, a parent directory entry (dentryid), and a file name of the data to be cleaned.

[0109] The format of data identification is: <hour directory inode>#<original parent directory inode>_ <dentryid>_<original file name>, where the hourly directory inode is an example of a time inode with hourly granularity.

[0110] In one example, the data identifier of the data with a file name of "file2" is "1024#23_27_file2". In another example, the data identifier of the data with a file name of "dir4" is "1025#20_23_dir4".

[0111] It should be understood that each data identifier can be used to uniquely identify a piece of data and determine the location of the data in the storage space. Therefore, based on the data identifiers of the N pieces of data to be processed, the metadata server can determine which data needs to be deleted and their locations.

[0112] The metadata server counts the input and output requests and query requests of the foreground, as well as the amount of unmerged data in the background. Among them, the input and output requests and query requests of the foreground can be used to determine whether the current storage system is busy, and the number of unmerged files in the background can be used to predict the performance impact that subsequent foreground input and output requests may have on the storage system. Therefore, the above three preset conditions can be used to determine whether the storage system is in a low period.

[0113] As mentioned above, data marked as abnormal cannot be accessed, while data marked as normal can be accessed.

[0114] The "cannot be accessed" and "can be accessed" here are for tenants. If the length of time that data 1 exists in the recycle bin does not reach the preset length of time, then data 1 is still stored in the recycle bin and can still be accessed by the tenant; if the length of time that data 1 exists in the recycle bin reaches the preset length of time, data 1 should be cleared for the tenant. Although data 1 may not be cleared in the metadata server for the time being, the tenant cannot access data 1.

[0115] Optionally, the method also includes: the metadata server receives a query request, the query request carries a directory structure index of the second data, and is used to request a query for the second data; the metadata server determines a time index of the first data based on the directory structure index of the second data and a third mapping relationship, the third mapping relationship including a one-to-one correspondence between at least one directory structure index and at least one time index; the metadata server determines a state of the second data based on the second mapping relationship; the metadata server rejects the query request if the state of the second data is abnormal; or the metadata server searches for the second data based on the directory structure index and the time index if the state of the second data is normal.

[0116] Table 4 below shows an example of the third mapping relationship. The third mapping relationship shown in Table 4 can be called a directory structure table.

[0117] Table 4

[0118] Directory Structure Index Time Index 2 1025 20 1025 22 1025 23 1024，1025 26 1024

[0119] The process of querying the directory structure of the recycle bin is as follows: Figure 7 Specifically, the process of querying the directory structure of the recycle bin may include:

[0120] Step 701, obtain the hourly directory of the deleted files / subdirectories in the storage directory;

[0121] Step 702, filtering hourly directories in abnormal states;

[0122] Step 703, query the files and directories under the normal state hour directory.

[0123] For example, assume that the directory structure indexes of data 1 and data 2 are both "23". If the tenant queries "23" at 1:00 on June 16, 2021, the metadata server can determine the time indexes are 1024 and 1025 through the third mapping relationship, and then query the second mapping relationship. The state corresponding to time index 1024 is abnormal, and the state corresponding to time index 1025 is normal. Then, data 1 with time index 1024 is filtered out, and data 2 with time index 1025 is returned to the tenant.

[0124] Based on the above method, the first data that has expired but not been cleaned up is marked as abnormal through the metadata server and is not included in the tenant's paid space. When the system overhead meets the preset conditions, the data marked as abnormal is cleared. Therefore, the strategy of cleaning up expired data in the recycle bin is optimized. While reducing the impact on system performance, it does not increase the tenant's occupancy time of storage space, avoiding unnecessary expenses for the tenant.

[0125] Figure 8 A schematic block diagram of a recycle bin management server provided in an embodiment of the present application. The recycle bin management server 800 may be Figure 1 The recycle bin management server 111 in the embodiment of the present invention, or a server having the same function as the recycle bin management server 111. Figure 8 As shown, the recycle bin management server 800 may include a processing module 810 and a sending module 820. The processing module 810 may be used to generate a first instruction according to a time index of the first data, wherein the first data is data that has been stored in the recycle bin of the first tenant for a preset period of time but has not been cleaned up, and the first instruction is used to indicate that the first data is identified as an abnormal state, and the storage space occupied by the data identified as an abnormal state in the recycle bin is not included in the billing space of the first tenant; the sending module 820 may be used to send the first instruction to the metadata server.

[0126] Optionally, the processing module 810 can also be used to determine the time index of the first data based on a preset duration and a first mapping relationship; the storage space occupied by the data identified as abnormal in the recycle bin is not included in the billing space of the first tenant, and the first mapping relationship includes a correspondence between at least one time index and at least one time, and the time in at least one time represents the time when the data under the corresponding time index is stored in the recycle bin.

[0127] Optionally, the processing module 810 can also be used to generate a second instruction based on N data to be cleaned, the second instruction is used to instruct the cleaning of N data to be cleaned, the N data to be cleaned include the first data, and N is a preset value; the sending module 820 can also be used to send the second instruction to the metadata server.

[0128] Fig. 9 A schematic block diagram of a metadata server provided in an embodiment of the present application. The metadata server 900 may be Figure 1 The metadata server 112 in , or a server having the same function. Fig. 9 As shown, the metadata server 900 may include a receiving module 910 and a processing module 920. The receiving module 910 may be used to receive a first instruction from a recycle bin management server, the first instruction being used to instruct that first data in the recycle bin of the first tenant be identified as abnormal, wherein the first data is data that has been stored in the recycle bin for a preset period of time but has not been cleared, and the storage space occupied by the data identified as abnormal in the recycle bin is not included in the billing space of the first tenant; the processing module 920 may be used to identify the first data as abnormal based on the first instruction; and clear the first data identified as abnormal from the recycle bin when the system overhead meets a preset condition.

[0129] Optionally, the processing module 920 may also be used to exclude the storage space occupied by the data to be cleaned from the storage space occupied by the file system and recycle bin of the first tenant to obtain the billing space of the first tenant; the data to be cleaned includes the first data.

[0130] Optionally, the processing module 920 can also be used to identify the state of the first data as a frozen state before excluding the storage space occupied by the first data from the storage space occupied by the file system and recycle bin of the first tenant; and identify the state of the first data as a clean state after excluding the storage space occupied by the first data from the storage space occupied by the file system and recycle bin of the first tenant.

[0131] Optionally, the processing module 920 may also be configured to, when the system overhead meets a preset condition, clear N data to be cleared from the recycle bin based on the second instruction.

[0132] Optionally, the receiving module 910 can also be used to receive a query request, the query request carrying a directory structure index of the second data, for requesting to query the second data; the processing module 920 can also be used to determine the time index of the first data based on the directory structure index of the second data and a third mapping relationship, the third mapping relationship including a one-to-one correspondence between at least one directory structure index and at least one time index; determine the state of the second data based on the second mapping relationship; reject the query request if the state of the second data is abnormal; or search for the second data based on the directory structure index and the time index if the state of the second data is normal.

[0133] Fig.10 is a schematic diagram of the structure of the data processing device provided by the present application. In one embodiment, the data processing device may be a recycling bin management server. In another embodiment, the data processing device may be a metadata server. Fig.10 As shown, the data processing device 1000 may include at least one processor 1010, which is used to implement the data processing function in the method provided in this application. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0134] The data processing device 1000 may also include a memory 1020 for storing program instructions and / or data. The memory 1020 is coupled to the processor 1010. The coupling in this application is an indirect coupling or communication connection between devices, units or modules, which may be electrical, mechanical or other forms, for information exchange between devices, units or modules. The processor 1010 may operate in conjunction with the memory 1020. The processor 1010 may execute program instructions stored in the memory 1020. At least one of the at least one memory may be included in the processor.

[0135] The data processing device 1000 may also include a communication interface 1030 for communicating with other devices via a transmission medium, so that the device in the data processing device 1000 can communicate with other devices. The communication interface 1030 may be, for example, a transceiver, an interface, a bus, a circuit, or a device capable of implementing transceiver functions. The processor 1010 may use the communication interface 1030 to send and receive data and / or information, and to implement Figure 5 The data processing method in the corresponding embodiment.

[0136] The specific connection medium between the processor 1010, the memory 1020 and the communication interface 1030 is not limited in this application. Fig.10 In the embodiment, the processor 1010, the memory 1020 and the communication interface 1030 are connected via a bus 1040. The bus 1040 is connected to the Fig.10 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0137] In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0138] According to the method provided by the present application, the present application also provides an electronic device, the electronic device includes a processor, which is used to call a computer program so that the electronic device executes Figures 5 to 7 The method executed by the recycle bin management server and / or the method executed by the metadata server in the illustrated embodiment. Optionally, the electronic device further comprises a memory for storing the computer program.

[0139] According to the method provided by the present application, the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer executes Figures 5 to 7 The method in the illustrated embodiment is executed by the recycle bin management server or the method executed by the metadata server.

[0140] According to the method provided by the present application, the present application also provides a computer program product, the computer program product comprising: computer program code. When the computer program code is run on a computer, the computer executes Figures 5 to 7 The method in the illustrated embodiment is executed by the recycle bin management server or the method executed by the metadata server.

[0141] The technical solution provided in this application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a terminal device or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center through a wired, such as coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium such as a digital video disc (DVD), or a semiconductor medium, etc.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.< / dentryid>

Claims

1. A data processing method, characterized in that: Applied in a storage system, the storage system is used to provide storage services for one or more tenants, the storage system includes a recycling bin management server and a metadata server, and the method includes: The metadata server receives a first instruction from the recycle bin management server, the first instruction being used to instruct to mark first data in the recycle bin of the first tenant as being in an abnormal state; wherein the first data is data that has been stored in the recycle bin for a preset period of time but has not been cleared, and the storage space occupied by the data marked as being in an abnormal state in the recycle bin is not counted into the billing space of the first tenant; The metadata server identifies the first data as being in an abnormal state based on the first instruction; The metadata server clears the first data identified as being in an abnormal state from the recycle bin when the system overhead meets a preset condition; Among them, data marked as abnormal cannot be accessed, and data marked as normal can be accessed.

2. The method according to claim 1, characterized in that The first instruction carries a time index of the first data, where the time index is related to a time when the first data is put into the recycle bin; as well as The method further comprises: The recycle bin management server determines a time index of the first data according to the preset duration and a first mapping relationship, wherein the first mapping relationship includes a correspondence between at least one time index and at least one time, and the time in the at least one time indicates the time when the data under the corresponding time index is put into the recycle bin; The recycle bin management server generates the first instruction according to a time index of the first data.

3. The method according to claim 1, characterized in that The first instruction further indicates the amount of the storage space occupied by the first data; The method further comprises: The metadata server excludes the amount of storage space occupied by the data to be cleaned from the amount of storage space occupied by the file system and the recycle bin of the first tenant to obtain the billing space of the first tenant; the data to be cleaned includes the first data.

4. The method according to claim 3, characterized in that The abnormal state includes a frozen state and a cleanup state, and The metadata server identifies the state of the first data as an abnormal state based on the first instruction, including: The metadata server identifies the state of the first data as a frozen state before excluding the amount of the storage space occupied by the first data from the amount of the storage space occupied by the file system of the first tenant and the recycle bin; After excluding the amount of the storage space occupied by the first data from the amount of the storage space occupied by the file system of the first tenant and the recycle bin, the metadata server identifies the state of the first data as a clean state.

5. The method according to claim 1, characterized in that The method further comprises: The recycle bin management server generates a second instruction based on N data to be cleaned up, wherein the second instruction is used to instruct to clean up the N data to be cleaned up, wherein the N data to be cleaned up include the first data, and N is a preset value; The recycle bin management server sends the second instruction to the metadata server; and When the system overhead meets a preset condition, the metadata server clears the first data identified as being in an abnormal state from the recycle bin, including: When the system overhead meets a preset condition, the metadata server clears the N data to be cleared from the recycle bin based on the second instruction.

6. The method according to claim 5, characterized in that The second instruction carries data identifiers of the N data to be cleaned, and each data identifier includes a time index node, a parent directory index node, a parent directory entry and a file name of the data to be cleaned.

7. The method according to claim 1, characterized in that The preset conditions include one or more of the following: The input and output volume of the front-end per second is lower than the first preset threshold. The query rate per second of the front desk is lower than the second preset threshold, and The amount of unmerged data in the memory is lower than a third preset threshold.

8. The method according to claim 1, characterized in that The method further comprises: The metadata server receives a query request, where the query request carries a directory structure index of the second data and is used to request a query for the second data; The metadata server determines the time index of the first data according to the directory structure index of the second data and a third mapping relationship, wherein the third mapping relationship includes a one-to-one correspondence between at least one directory structure index and at least one time index; The metadata server determines the state of the second data according to a second mapping relationship, where the second mapping relationship includes a correspondence between at least one time index and at least one state; The metadata server rejects the query request when the state of the second data is abnormal; or When the state of the second data is normal, the metadata server searches for the second data based on the directory structure index and the time index.

9. A storage system, characterized in that: Used to provide data storage services for one or more tenants, the storage system includes a recycling bin management server and a metadata server; wherein, The recycle bin management server is used to send a first instruction to the metadata server, wherein the first instruction is used to instruct to mark first data in the recycle bin of the first tenant as abnormal; wherein the first data is data that has been stored in the recycle bin for a preset period of time but has not been cleared, and the storage space occupied by the data marked as abnormal in the recycle bin is not included in the billing space of the first tenant; The metadata server is used to identify the first data as an abnormal state based on the first instruction; and is used to clear the first data from the recycle bin based on the first instruction when the system overhead meets a preset condition; Among them, data marked as abnormal cannot be accessed, and data marked as normal can be accessed.

10. An electronic device, characterized in that: The electronic device comprises a processor, wherein the processor is used to call a computer program so that the electronic device implements the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 8 is implemented.

Images