Metadata garbage collection method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure provide a metadata garbage collection method and device, electronic equipment and storage medium. In response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace is detected, and a target storage segment is determined. The metadata persistent storage area includes storage segments that store metadata based on a sequential writing method. The target storage segment is a storage segment to be garbage collected. Metadata is used to implement data recovery of corresponding business data. For the target storage segment, a corresponding metadata snapshot is generated and stored in the metadata persistent storage area. The metadata snapshot is used to record the latest generated metadata in the target storage segment. The data in the target storage segment is cleared. Data recovery of the storage device can be achieved, the utilization rate and space recovery efficiency of the storage space can be improved, and the operation stability of the storage device can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of cloud storage, and particularly relate to a metadata garbage collection method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In the technical field of cloud storage, a storage device based on zoned namespaces (ZNS) technology divides a storage space into multiple segments for management, thereby optimizing the storage and access process of data and improving the data read / write performance.

[0003] In the prior art, in order to increase the user storage space of a storage device, only a small amount of space is usually reserved to store metadata for data recovery. In addition, in a storage device based on the zoned namespaces technology, data can only be sequentially written and cannot be overwritten, which leads to insufficient metadata storage space and low space recycling efficiency, thereby affecting the operational stability of the storage device. SUMMARY

[0004] Embodiments of the present disclosure provide a metadata garbage collection method and device, an electronic device, and a storage medium to overcome the problems of insufficient metadata storage space and low space recycling efficiency.

[0005] In a first aspect, embodiments of the present disclosure provide a metadata garbage collection method, comprising:

[0006] In response to a garbage collection instruction, a metadata persistent storage area in a storage device based on zoned namespaces is detected, and a target storage segment is determined, wherein the metadata persistent storage area includes storage segments that store metadata based on sequential writing, the target storage segment is a storage segment to be subjected to garbage collection, and the metadata is used to implement data recovery of corresponding business data; a corresponding metadata snapshot is generated for the target storage segment, and the metadata snapshot is stored in the metadata persistent storage area, the metadata snapshot is used to record the latest generated metadata in the target storage segment; and data in the target storage segment is cleared.

[0007] In a second aspect, embodiments of the present disclosure provide a metadata garbage collection device, comprising:

[0008] A detection module is configured to, in response to a garbage collection instruction, detect a metadata persistent storage area in a storage device based on zoned namespaces, and determine a target storage segment, wherein the metadata persistent storage area includes storage segments that store metadata based on sequential writing, the target storage segment is a storage segment to be subjected to garbage collection, and the metadata is used to implement data recovery of corresponding business data.

[0009] generating, for the target storage segment, a corresponding metadata snapshot, and storing the metadata snapshot into the metadata persistent storage, the metadata snapshot being used to record the newly generated metadata in the target storage segment;

[0010] cleaning up data in the target storage segment.

[0011] In a third aspect, an electronic device is provided, which includes a processor and a memory.

[0012] The memory stores computer-executable instructions.

[0013] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the metadata garbage collection method according to the first aspect and various possible designs of the first aspect.

[0014] In a fourth aspect, a computer-readable storage medium is provided, which stores computer-executable instructions. When a processor executes the computer-executable instructions, the metadata garbage collection method according to the first aspect and various possible designs of the first aspect is implemented.

[0015] In a fifth aspect, a computer program product is provided, which includes a computer program. When a processor executes the computer program, the metadata garbage collection method according to the first aspect and various possible designs of the first aspect is implemented.

[0016] The metadata garbage collection method, device, electronic equipment and storage medium provided by the embodiment, in response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace is detected, a target storage segment is determined, wherein the metadata persistent storage area includes a storage segment storing metadata based on a sequential writing manner, the target storage segment is a storage segment to be garbage collected, and the metadata is used to realize data recovery of corresponding business data; for the target storage segment, a corresponding metadata snapshot is generated, and the metadata snapshot is stored in the metadata persistent storage area, the metadata snapshot is used to record the latest generated metadata in the target storage segment; and data in the target storage segment is cleared. By determining the target storage segment in the metadata persistent storage area that needs to be garbage collected, triggering a snapshot generation event for the target storage segment, generating a data snapshot of the metadata in the target storage segment, i.e. a metadata snapshot, then storing the metadata snapshot in the metadata persistent storage area, and clearing the data in the target storage segment, the garbage collection of the target storage segment is completed, at the same time, by saving the metadata snapshot in the metadata persistent storage area, data recovery of the storage device can be realized, the utilization rate and space recovery efficiency of the storage space are improved, and the operation stability of the storage device is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 An application scenario diagram of the metadata garbage collection method provided by the embodiment of the present disclosure is shown in the figure.

[0019] Figure 2 A flowchart of the metadata garbage collection method provided by the embodiment of the present disclosure is shown in the figure. Figure 1 ;

[0020] Figure 3 A schematic diagram of a metadata persistent storage area provided by the embodiment of the present disclosure is shown in the figure.

[0021] Figure 4 A process diagram for determining a target storage segment provided by the embodiment of the present disclosure is shown in the figure.

[0022] Figure 5 A flowchart of the specific implementation of step S102 in the embodiment shown in the figure. Figure 2

[0023] Figure 6 ​Flowchart of the metadata garbage collection method provided in the embodiments of this disclosure Figure 2 ;

[0024] Figure 7 for Figure 6 A flowchart illustrating the specific implementation of step S200 in the illustrated embodiment;

[0025] Figure 8 This is a schematic diagram illustrating the process of persisting metadata snapshots according to an embodiment of the present disclosure;

[0026] Figure 9 for Figure 6 A flowchart illustrating the specific implementation of step S203 in the illustrated embodiment;

[0027] Figure 10 This is a structural block diagram of the metadata garbage collection device provided in the embodiments of this disclosure;

[0028] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0029] Figure 12 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0032] The application scenarios of the embodiments of this disclosure are explained below:

[0033] Figure 1An application scenario diagram of the metadata garbage collection method provided by the embodiments of the present disclosure is shown in the figure. The metadata garbage collection method provided by the embodiments of the present disclosure can be applied in the application scenario of cloud storage, more specifically, in the application scenario of distributed cloud storage. The execution subject of the present embodiment can be a storage node in a distributed cloud storage system, such as a storage device or other electronic devices with similar functions. In some embodiments, the storage device can implement the metadata garbage collection method provided by the embodiments of the present disclosure by running various computer executable instructions or computer programs. For example, the computer executable instructions can be program-level commands, machine instructions, or software instructions. The computer program can be a native program or a software module in the operating system; it can be a local application program, that is, a program that needs to be installed in the operating system to run, or it can be a small program embedded in any APP, that is, a program running based on the browser environment. In summary, the above-mentioned computer executable instructions can be any form of instructions, and the above-mentioned computer programs can be any form of application programs, modules or plug-ins, and the specific implementation form can be configured as needed. Further, the storage device can execute the metadata garbage collection method provided by the embodiments of the present disclosure by running the computer executable instructions or computer programs set locally, or by calling the computer executable instructions or computer programs set in the server outside. In some embodiments, the above-mentioned server can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud storage, cloud communication, cloud database, cloud computing, cloud function, network service, middleware service, domain name service, security service, content distribution network (Content Delivery Network, CDN), and big data and artificial intelligence platform, etc. Basic cloud computing services.

[0034] Reference Figure 1 As shown in the figure, in a partitioned namespace-based storage device, the storage space is abstracted into a plurality of continuous storage segments (Segments) through a specific interface. The storage segment is a mapping of the physical storage medium at the software layer, wherein most of the storage segments are used to store service data, i.e., a service data storage area, such as Segment_00 to Segment_95 shown in the figure; and a small part of the storage segments are used to store metadata corresponding to the service data, such as Segment_96 to Segment_99, i.e., a metadata persistent storage area. After the partition is formatted, the above-mentioned space division method is also determined, and the space size of the storage segment used to store service data and the storage segment used to store metadata will not change.

[0035] Due to the technical characteristics of the zoning namespace (ZNS) technology, the metadata can be written into the storage segment sequentially as the business data, and the operation record log is usually written into the storage segment. After the storage segment is full, if the space in the storage segment needs to be used again, the effective data in the storage segment needs to be migrated first, and then the storage segment is erased before it can be reused. However, due to the small storage space for storing the metadata corresponding to the business data, when the business data is frequently modified, a large number of operation record logs will be generated, so that the corresponding storage segment is filled to trigger the garbage collection process, and the frequent garbage collection process will cause the record log to be repeatedly moved, thereby generating additional resource overhead, resulting in the problems of insufficient metadata storage space and low space recycling efficiency.

[0036] The embodiments of the present disclosure provide a metadata garbage collection method to solve the above problems.

[0037] Reference Figure 2 , Figure 2 The flowchart of the metadata garbage collection method provided by the embodiments of the present disclosure is shown in Figure 1 The method of the present embodiment can be applied in a storage device, and the metadata garbage collection method comprises the following steps:

[0038] Step S101: In response to a garbage collection instruction, a metadata persistent storage area in a zoning namespace-based storage device is detected, and a target storage segment is determined, wherein the metadata persistent storage area includes storage segments that store metadata in a sequential writing manner, and the target storage segment is a storage segment to be subjected to garbage collection.

[0039] Exemplarily, reference is made to Figure 1As shown in the application scenario schematic diagram, the execution subject of the method provided in this embodiment can be a storage device or a control unit in the storage device (hereinafter, a request of the control unit as the execution subject is taken as an example for description), wherein one or more storage segments are arranged in the storage device, and the storage device controls the one or more storage segments through the control unit, so as to implement the metadata garbage collection method provided in this embodiment. Specifically, the control unit detects a metadata persistent storage area in a storage device based on a zoning namespace (ZNS) in response to a garbage collection instruction, and the metadata persistent storage area is a set of storage spaces in the storage device for storing metadata related information. The metadata persistent storage area includes one or more storage segments (Segments), and each storage segment is used to store corresponding metadata. The metadata in the storage segment is stored based on a sequential writing manner. The control unit detects each storage segment in the metadata persistent storage area through a preset detection logic, and determines one or more storage segments as target storage segments. Specifically, the control unit determines the target storage segments by detecting the storage load of each storage segment. In a possible implementation manner, the storage load represents the proportion of invalid data in the storage segment. In this case, one or more storage segments with the largest proportion of invalid data are determined as the target storage segments. Invalid data refers to expired metadata in the data segment. The higher the proportion of invalid data is, the more garbage collection is needed to release the storage space.

[0040] In a possible implementation manner, Figure 3 A schematic diagram of a metadata persistent storage area provided in an embodiment of the present disclosure is as follows, Figure 3As shown, the metadata persistent storage includes a first metadata persistent storage and a second metadata persistent storage, the first metadata persistent storage is used to store historical operation records for business data, wherein the operation records record operation records for creation, freeze, discard and deletion of chunk files of a single storage engine, and operation records for opening, closing and erasing of storage segments, wherein the chunk file is a subunit of the storage segment for storing business data, which is a concept in the prior art and will not be described here. Through the operation records, operation reproduction of the business data can be realized, and thus data recovery of the business data can be realized. Therefore, the first metadata persistent storage for storing the historical operation records, that is, a journal segment. The second metadata persistent storage is used to store checkpoint data for metadata, that is, a metadata checkpoint. Wherein, generating a checkpoint is a data persistence technology, and generating a checkpoint for the above-mentioned storage segment or chunk file will persist the related information of the metadata of the storage segment or chunk file, thereby realizing data recovery of the business data, at the same time, the metadata checkpoint has a better data recovery priority, therefore, the metadata checkpoint will invalidate the operation log of the previously generated storage segment or chunk file. The second metadata persistent storage is used to store the metadata checkpoint generated by the system, that is, a metadata checkpoint segment (matasegment).

[0041] Correspondingly, the specific implementation of step S101 includes: in response to the first garbage collection instruction, detecting the first metadata persistent storage, determining the target storage segment in the first metadata persistent storage, and / or in response to the second garbage collection instruction, detecting the second metadata persistent storage, and determining the target storage segment in the second metadata persistent storage. That is, when the garbage collection instruction responded by the control unit is the first garbage collection instruction, the first metadata persistent storage is detected, and the target storage segment in the first metadata persistent storage is determined, so that in the subsequent steps, the garbage collection of the first metadata persistent storage is completed; when the garbage collection instruction responded by the control unit is the second garbage collection instruction, the second metadata persistent storage is detected, and the target storage segment in the second metadata persistent storage is determined, so that in the subsequent steps, the garbage collection of the second metadata persistent storage is completed.

[0042] It can be understood that in another possible implementation, the garbage collection instruction can also simultaneously include the first garbage collection instruction and the second garbage collection instruction, that is, the control unit detects the target storage segment in the first metadata persistent storage area and the target storage segment in the second metadata persistent storage area respectively by responding to the garbage collection instruction, thereby completing the garbage collection of the first metadata persistent storage area and the second metadata persistent storage area. This can be set as needed, and will not be repeated here.

[0043] Further, for the case of garbage collecting the first metadata persistent storage area in response to the first garbage collection instruction, when the first data storage area is detected, the data write time stamps corresponding to each storage segment in the first metadata persistent storage area are obtained, and the storage segment corresponding to the earliest write time stamp is determined as the target storage segment; and for the case of garbage collecting the second metadata persistent storage area in response to the second garbage collection instruction, when the second data storage area is detected, the storage loads corresponding to each storage segment in the second metadata persistent storage area are obtained, and at least one storage segment with the largest storage load is determined as the target storage segment.

[0044] Figure 4 A process schematic diagram for determining a target storage segment provided by the embodiments of the present disclosure is as shown in Figure 4As shown, when the control unit detects the first data storage area in response to the first garbage collection instruction, since the first data storage area is used to store the historical operation records for the business data, the historical operation records have the time sequence attribute, and therefore when data recovery is performed using the historical operation records, the playback must be strictly in chronological order, and therefore when garbage collection is performed on the storage segments (shown in the figure as journal_segment_1, journal_segment_2, etc., with storage loads of storage load=30%, storage load=76%) in the first data storage area, the earliest storage segment is preferentially selected, that is, the data write timestamp corresponding to each storage segment in the first metadata persistent storage area is obtained, and the storage segment corresponding to the earliest write timestamp is determined as the target storage segment, for example, the storage segment journal_segment_1 shown in the figure. When the control unit detects the second data storage area in response to the second garbage collection instruction, since the second data storage area is used to store the metadata snapshots for the metadata, through the metadata snapshots, data recovery can be performed on the business data corresponding to the metadata, and is not affected by the log time, and therefore the target storage segment can be directly determined through the storage load of each storage segment in the second data storage, that is, one or more storage segments with the largest storage load are determined as the target storage segment, for example, the two storage segments meta_segment_1 (storage load=60%) and meta_segment_2 (storage load=75%) with the largest storage load are determined as the target storage segment.

[0045] Further, the specific implementation manner of determining the at least one storage segment with the largest storage load as the target storage segment includes:

[0046] Step S1011: Obtain the comprehensive load value of the second metadata persistent storage area, and determine the target number according to the comprehensive load value and the preset load mapping information, wherein the preset load mapping information represents the mapping relationship between the comprehensive load value and the number of storage segments.

[0047] Step S1012: Determine the target storage segment as the target number of storage segments in the second metadata persistent storage area with the largest storage load.

[0048] Exemplarily, before determining the target storage segment, the control unit first acquires a comprehensive load value of the second metadata persistent storage area, that is, the current usage amount of the storage segment for storing the metadata snapshot, and then determines the number of the target storage segments, that is, the target number, from the second metadata persistent storage area according to the comprehensive load value. The comprehensive load value is in a positive proportional relationship with the target number, that is, within a certain range, the greater the comprehensive load value, the greater the target number. For example, when the comprehensive load value is 60%, the target number is 1, that is, one storage segment with the largest storage load is selected as the target storage segment; when the comprehensive load value is 80%, the target number is 3, that is, three storage segments with the largest storage load are selected as the target storage segments. Through the steps of this embodiment, the garbage cleaning amount can be increased when the comprehensive load value of the second metadata persistent storage area is large, so as to quickly reduce the comprehensive load value and improve the efficiency of garbage cleaning.

[0049] Step S102: For the target storage segment, a corresponding metadata snapshot is generated, and the metadata snapshot is stored in the metadata persistent storage area. The metadata snapshot is used to realize data recovery of the business data corresponding to the metadata by recording the latest generated metadata in the target storage segment.

[0050] Step S103: The data in the target storage segment is cleared.

[0051] Exemplarily, after determining the target storage segment in the metadata persistent storage area, a corresponding metadata snapshot is generated for the target storage segment. The implementation process of generating the metadata snapshot for the target storage segment can be realized through a functional interface provided by the ZNS system, and the specific process of generating the metadata snapshot is not described herein. Then, the metadata snapshot is stored in the metadata persistent storage area to realize the persistence of the metadata snapshot. After the metadata snapshot is persisted, when the system restarts and needs to recover the business data, the corresponding metadata snapshot can be read from the above metadata persistent storage area (persistent storage area), and the business data is recovered to the state at the time when the metadata snapshot is generated based on the metadata snapshot, so as to realize the recovery of the business data. After the above metadata snapshot generation and persistence process is completed, the data in the above target storage segment can be cleared, or the target storage segment can be deleted, so as to realize the purpose of garbage collection.

[0052] In a possible implementation, the storage device based on the partition namespace includes at least one single-machine storage engine file, and the single-machine storage engine file is used to store the business data corresponding to the metadata. Before step S102, the method further includes: acquiring validity information of the single-machine storage engine file corresponding to the target storage segment, and the validity information is used to indicate the validity of the single-machine storage engine file.

[0053] Correspondingly, the step S102 includes: according to the validity information of the single storage engine file, if the single storage engine file is in the valid state, generating the corresponding metadata snapshot for the target storage segment.

[0054] Exemplarily, in the step of the embodiment, the control unit first acquires the single storage engine file corresponding to the metadata in the target storage segment before generating the metadata snapshot, so as to detect the validity information of the single storage engine file. If the single storage engine file is deleted or moved to other storage devices or media, it is judged that the single storage engine file is in the invalid state. In this case, the subsequent step of generating the metadata snapshot can be not performed, so as to save resource overhead. On the other hand, if the single storage engine file normally exists, it is judged that the single storage engine file is in the valid state. In this case, the corresponding function interface is called for the single storage engine file, so that the corresponding metadata snapshot can be generated.

[0055] Further, in a possible implementation manner, as shown in Figure 5 the step S102 includes:

[0056] Step S1021: acquiring the storage priority of the first metadata persistent storage area and the second metadata persistent storage area;

[0057] Step S1022: determining the proportion value of the first metadata snapshot and the second metadata snapshot according to the storage priority, wherein the first metadata snapshot is the metadata snapshot generated based on the target storage segment in the first metadata persistent storage area; and the second metadata snapshot is the metadata snapshot generated based on the target storage segment in the second metadata persistent storage area.

[0058] Step S1023: executing the metadata migration request based on the proportion value through the pre-configured task queue, and generating the corresponding number of the first metadata snapshot and the second metadata snapshot in a unit time.

[0059] Step S1024: storing the first metadata snapshot and the second metadata snapshot into the metadata persistent storage area.

[0060] Exemplarily, the process of generating the metadata snapshot is equivalent to the process of generating the metadata snapshot, so as to consume a certain amount of computing resources, for example. In the embodiment step, in the process of generating the metadata snapshot, the storage priorities of the first metadata persistent storage area and the second metadata persistent storage area are first determined, and then the corresponding proportional values are converted based on the storage priorities. Then, based on the proportional values, the metadata migration request is executed through the pre-configured task queue, and a corresponding number of first metadata snapshots and second metadata snapshots are generated in a unit of time. Specifically, the first metadata snapshot and the second metadata snapshot are the latest metadata information of the migration chunk involved. This process is equivalent to generating events for the snapshots corresponding to the target storage segments in the first metadata persistent storage area and the second metadata persistent storage area based on the proportional values, respectively allocating corresponding computing resources, thereby realizing dynamic adjustment of the garbage collection process and improving the overall garbage collection efficiency of the storage device.

[0061] Further, the embodiment step further includes:

[0062] Step S1025: Obtain the comprehensive load values corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0063] Step S1026: Adjust the storage priorities of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load values corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0064] Exemplarily, based on the above embodiment step, the control unit can further adjust the storage priorities of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load values corresponding to the first metadata persistent storage area and the second metadata persistent storage area. For example, when the comprehensive load value of the first metadata persistent storage area is greater than a preset load proportion, and / or the comprehensive load value of the first metadata persistent storage area is greater than a preset multiple of the comprehensive load value of the second metadata persistent storage area, the corresponding storage priority of the first metadata persistent storage area is increased. For another example, when the comprehensive load value of the second metadata persistent storage area is greater than a preset load proportion, and / or the comprehensive load value of the second metadata persistent storage area is greater than a preset multiple of the comprehensive load value of the first metadata persistent storage area, the corresponding storage priority of the second metadata persistent storage area is increased. The specific implementation manner can be set according to the resource configuration and specific requirements of the storage device, which will not be described here.

[0065] In this embodiment, by detecting a metadata persistent storage area in the partition-based namespace storage device in response to the garbage collection instruction, a target storage segment is determined, wherein the metadata persistent storage area includes a storage segment storing metadata in a sequential writing manner, the target storage segment is a storage segment to be subjected to garbage collection, and the metadata is used to implement data recovery of corresponding service data; a corresponding metadata snapshot is generated for the target storage segment, and the metadata snapshot is stored in the metadata persistent storage area, the metadata snapshot is used to record the latest generated metadata in the target storage segment; and data in the target storage segment is cleared. By determining the target storage segment in the metadata persistent storage area to be subjected to garbage collection, triggering a snapshot generation event for the target storage segment, generating a data snapshot of the metadata in the target storage segment, i.e., the metadata snapshot, and then storing the metadata snapshot in the metadata persistent storage area and clearing the data in the target storage segment, garbage collection of the target storage segment is completed, and at the same time, data recovery of the storage device is implemented, the utilization rate and space recovery efficiency of the storage space are improved, and the operation stability of the storage device is improved.

[0066] Reference Figure 6 , Figure 6 Flowchart of a metadata garbage collection method provided by the embodiments of the present disclosure Figure 2 The embodiments of the present disclosure provide a metadata garbage collection method. Figure 2 On the basis of the embodiments shown in the embodiments, steps S102-103 are further refined, and the metadata garbage collection method comprises:

[0067] Step S200: generating a garbage collection instruction, the garbage collection instruction comprising a first garbage collection instruction and / or a second garbage collection instruction.

[0068] Exemplarily, the storage device generates a garbage collection instruction by using a preset program logic of a control unit during operation, for example, generates a garbage collection instruction based on a preset time interval, so as to trigger a subsequent metadata garbage collection process; or generates a garbage collection instruction by detecting an index in the storage device.

[0069] In a possible implementation manner, as shown in Figure 7 , the specific implementation manner of step S200 comprises:

[0070] Step S2001: obtaining a comprehensive load value of the metadata persistent storage area;

[0071] Step S2002: determining a corresponding trigger recovery load interval according to the comprehensive load value, wherein the comprehensive load value is inversely proportional to an interval median of the trigger recovery load interval;

[0072] Step S2003: When the storage load of at least one storage segment in the metadata persistent storage area is located in the trigger recovery load interval, a garbage collection instruction is generated.

[0073] Exemplarily, the control unit first acquires a comprehensive load value of the metadata persistent storage area, which can be the data storage amount, occupancy amount, occupancy rate, etc. of the metadata persistent storage area. Then, according to the comprehensive load value, a corresponding trigger recovery load interval is determined, wherein the comprehensive load value is inversely proportional to the interval value of the trigger recovery load interval. Specifically, the higher the comprehensive load value, the lower the interval value of the corresponding trigger recovery load interval. Here, the interval value of the trigger recovery load interval can refer to the upper limit value, or the lower limit value, or the interval value of the trigger recovery load interval, without limitation. For example, the interval value of the trigger recovery load interval is the lower limit value of the trigger recovery load interval. For example, when the comprehensive load value is 50%, the corresponding trigger recovery load interval is 70%, i.e. the lower limit value of the trigger recovery load interval is 70%, and the upper limit value is fixed at 100%. Then, based on the trigger recovery load interval and the storage load of each storage segment in the metadata persistent storage area, a garbage collection instruction is generated, i.e. when the storage load of a storage segment is located in the trigger recovery load interval (i.e. greater than 70%), a garbage collection instruction is generated. When the comprehensive load value is 70%, the corresponding trigger recovery load interval is 50%, i.e. the lower limit value of the trigger recovery load interval is 50%, and the upper limit value is fixed at 100%. Then, based on the trigger recovery load interval and the storage load of each storage segment in the metadata persistent storage area, a garbage collection instruction is generated, i.e. when the storage load of a storage segment is located in the trigger recovery load interval (i.e. greater than 50%), a garbage collection instruction is generated.

[0074] In the step of the embodiment, the corresponding trigger recovery load interval is dynamically determined according to the comprehensive load value of the metadata persistent storage area, so as to dynamically adjust the garbage collection timing and frequency, avoid performance loss caused by excessive garbage collection, and improve the overall read-write efficiency of the storage device.

[0075] Step S201: In response to the first garbage collection instruction, the first metadata persistent storage area is detected to determine a target storage segment in the first metadata persistent storage area, and / or in response to the second garbage collection instruction, the second metadata persistent storage area is detected to determine a target storage segment in the second metadata persistent storage area.

[0076] Step S202: For the target storage segment in the first metadata persistent storage area, a first metadata snapshot is generated and stored in the second metadata persistent storage area.

[0077] Step S203: generating a second metadata snapshot in the second metadata persistent storage area, for the target storage segment in the second metadata persistent storage area.

[0078] For example, after the garbage collection instruction is generated, the control unit executes the corresponding steps according to the specific implementation content of the garbage collection instruction, that is, the first garbage collection instruction or the second garbage collection instruction, to determine the target storage segment in the first metadata persistent storage area and / or the target storage segment in the second metadata persistent storage area. Then, in response to the first garbage collection instruction, a first metadata snapshot is generated for the target storage segment in the first metadata persistent storage area, and the first metadata snapshot is stored in the second metadata persistent storage area; and in response to the second garbage collection instruction, a second metadata snapshot is generated in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area. Figure 8 A process diagram for persisting metadata snapshots provided by an embodiment of the present disclosure is shown in FIG. 2. As shown in FIG. 2, for example, in response to the first garbage collection instruction, a first metadata snapshot is generated for the target storage segment in the first metadata persistent storage area (journal_segment), and then the first metadata snapshot is stored in the second metadata persistent storage area (meta_segment); and in response to the second garbage collection instruction, a second metadata snapshot is directly generated in the second metadata persistent storage area (meta_segment) for the target storage segment in the second metadata persistent storage area (journal_segment). Figure 8

[0079] In other possible implementations, steps S202 and S203 can be executed separately, that is, in one possible embodiment, by executing steps S200, S201, S202, and S204, the garbage collection of the first metadata persistent storage area is completed; or in another possible embodiment, by executing steps S200, S201, S203, and S204, the garbage collection of the second metadata persistent storage area is completed. As described in the above embodiments, steps S202 and S203 are executed simultaneously, which is based on the specific garbage collection instruction, and will not be described again.

[0080] For example, the target storage segment includes at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; in the case of generating a second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area, as shown in FIG. 3, the specific implementation of step S203 includes: Figure 9 For example, the target storage segment includes at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; in the case of generating a second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area, as shown in FIG. 3, the specific implementation of step S203 includes:​

[0081] Step S203-1: generating the metadata snapshot of each target storage segment in the second metadata persistent storage area, and obtaining the data hotness corresponding to each target storage segment, the data hotness representing the update probability of the metadata corresponding to the target storage segment.

[0082] Step S203-2: storing the metadata snapshot corresponding to the target storage segment with the data hotness greater than the hotness threshold to the first sub-storage area.

[0083] Step S203-3: storing the metadata snapshot corresponding to the target storage segment with the data hotness not greater than the hotness threshold to the second sub-storage area.

[0084] Exemplarily, in a possible case, the second metadata persistent storage area contains a plurality of target storage segments, and the first sub-storage area and the first sub-storage area for storing the metadata snapshot; in this case, when the snapshot generation event for the target storage segment is triggered, first, the data hotness corresponding to each target storage segment is determined, the data hotness representing the update probability of the metadata corresponding to the target storage segment; in short, for the metadata (corresponding to the storage segment) that is no longer updated or has a small probability of being updated, the generated metadata snapshot can be stored to the second sub-storage area, and the frequency of detecting the second sub-storage area is reduced, so as to reduce the number and frequency of generating new metadata snapshots, thereby saving the computing resource overhead; while for the metadata (corresponding to the storage segment) that is frequently updated or has a large probability of being updated, the generated metadata snapshot can be stored to the first sub-storage area, thereby ensuring the real-time performance of data recovery and improving the data security.

[0085] Step S204: clearing the data in the target storage segment.

[0086] In this embodiment, the implementation manner of step S204 is the same as that of step S103 in the embodiment of the present disclosure, which will not be repeated here. Figure 2

[0087] The metadata garbage collection method corresponding to the above embodiment, Figure 10 A structure block diagram of the metadata garbage collection device provided by the embodiment of the present disclosure is shown. The method introduced in the above embodiment can be executed by the metadata garbage collection device, which can be realized by software and / or hardware, and can be integrated in an electronic device with certain data processing function. The electronic device can include but is not limited to a mobile terminal with large data processing capacity, and a desktop computer, a supercomputer and other fixed terminals with large data processing capacity.

[0088] For ease of illustration, only parts related to the embodiments of the present disclosure are shown. For details, refer to the description of the above embodiments. Figure 10 ​The metadata garbage collection apparatus 3 comprises:

[0089] The detection module 31 is configured to, in response to a garbage collection instruction, detect a metadata persistent storage area in the storage device based on the partition namespace, and determine a target storage segment, wherein the metadata persistent storage area comprises storage segments storing metadata in a sequential writing manner, and the target storage segment is a storage segment to be subjected to garbage collection, and the metadata is used to implement data recovery of corresponding service data.

[0090] The generation module 32 is configured to generate a corresponding metadata snapshot for the target storage segment, and store the metadata snapshot in the metadata persistent storage area, wherein the metadata snapshot is used to record the latest generated metadata in the target storage segment.

[0091] The cleaning module 33 is configured to clean data in the target storage segment.

[0092] According to one or more embodiments of the present disclosure, the metadata persistent storage area comprises a first metadata persistent storage area and a second metadata persistent storage area, wherein the first metadata persistent storage area is used to store historical operation records of service data, and the second metadata persistent storage area is used to store metadata snapshots of metadata; the detection module 31 is specifically configured to: in response to a first garbage collection instruction, detect the first metadata persistent storage area, and determine a target storage segment in the first metadata persistent storage area, and / or in response to a second garbage collection instruction, detect the second metadata persistent storage area, and determine a target storage segment in the second metadata persistent storage area.

[0093] According to one or more embodiments of the present disclosure, when the detection module 31 detects the first metadata persistent storage area in response to the first garbage collection instruction, and determines a target storage segment in the first metadata persistent storage area, the detection module 31 is specifically configured to: in response to the first garbage collection instruction, acquire data write time stamps corresponding to each storage segment in the first metadata persistent storage area, and determine a storage segment corresponding to the earliest write time stamp as the target storage segment; when the detection module 31 detects the second metadata persistent storage area in response to the second garbage collection instruction, and determines a target storage segment in the second metadata persistent storage area, the detection module 31 is specifically configured to: in response to the second garbage collection instruction, acquire storage loads corresponding to each storage segment in the second metadata persistent storage area, and determine at least one storage segment with the largest storage load as the target storage segment, wherein the storage load is used to represent a proportion of invalid data in the storage segment.

[0094] According to one or more embodiments of the present disclosure, when the detection module 31 determines the at least one storage segment with the largest storage load as the target storage segment, the detection module 31 is specifically configured to: obtain a comprehensive load value of the second metadata persistent storage area, and determine a target number according to the comprehensive load value; and determine the target number of storage segments with the largest storage load in the second metadata persistent storage area as the target storage segment.

[0095] According to one or more embodiments of the present disclosure, the generation module 32 is specifically configured to: generate a first metadata snapshot for the target storage segment in the first metadata persistent storage area, and store the first metadata snapshot to the second metadata persistent storage area, and / or generate a second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area.

[0096] According to one or more embodiments of the present disclosure, the target storage segment includes at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; when the generation module 32 generates the second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area, the generation module 32 is specifically configured to: generate a metadata snapshot of each target storage segment in the second metadata persistent storage area, and obtain a data heat corresponding to each target storage segment, the data heat representing an update probability of metadata corresponding to the target storage segment; store the metadata snapshot corresponding to the target storage segment with the data heat greater than a heat threshold to the first sub-storage area; and store the metadata snapshot corresponding to the target storage segment with the data heat not greater than the heat threshold to the second sub-storage area.

[0097] According to one or more embodiments of the present disclosure, the generation module 32 is specifically configured to: obtain a storage priority of the first metadata persistent storage area and the second metadata persistent storage area; determine a proportion value of the first metadata snapshot and the second metadata snapshot according to the storage priority, wherein the first metadata snapshot is a metadata snapshot generated based on the target storage segment in the first metadata persistent storage area, and the second metadata snapshot is a metadata snapshot generated based on the target storage segment in the second metadata persistent storage area; and execute a metadata migration request based on the proportion value through a pre-configured task queue, to generate a corresponding number of the first metadata snapshot and the second metadata snapshot in a unit time.

[0098] According to one or more embodiments of the present disclosure, the generation module 32 is further configured to: obtain a comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area; and adjust the storage priority of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0099] According to one or more embodiments of this disclosure, before detecting the metadata persistent storage area in the storage device based on the partition namespace and determining the target storage segment in response to a garbage collection instruction, the detection module 31 is further configured to: obtain the comprehensive load value of the metadata persistent storage area; determine the corresponding trigger garbage collection load interval based on the comprehensive load value, wherein the comprehensive load value is inversely proportional to the median value of the trigger garbage collection load interval; and generate a garbage collection instruction based on the trigger garbage collection load interval and the storage load of each storage segment in the metadata persistent storage area.

[0100] According to one or more embodiments of this disclosure, a storage device based on a partition namespace includes at least one single-machine storage engine file. The generation module 32 is further configured to: obtain validity information of the single-machine storage engine file corresponding to the target storage segment; when the generation module 32 generates a corresponding metadata snapshot for the target storage segment, it is specifically configured to: generate a metadata snapshot corresponding to the single-machine storage engine file based on the validity information of the single-machine storage engine file.

[0101] The detection module 31, generation module 32, and cleaning module 33 are connected in sequence. The metadata garbage collection device 3 provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0102] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 11 As shown, the electronic device 4 includes:

[0103] Processor 41, and memory 42 communicatively connected to processor 41;

[0104] Memory 42 stores instructions executed by the computer;

[0105] The processor 41 executes computer execution instructions stored in the memory 42 to achieve, for example, Figures 2-9 Metadata garbage collection method in the illustrated embodiment.

[0106] Optionally, the processor 41 and the memory 42 are connected via a bus 43.

[0107] For relevant instructions, please refer to the corresponding text. Figures 2-9 The relevant descriptions and effects of the steps in the corresponding embodiments are understood, and will not be elaborated on here.

[0108] This disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement this disclosure. Figures 2-9 The metadata garbage collection method provided in any of the corresponding embodiments.

[0109] The embodiment of the present disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements the metadata garbage collection method of any of the embodiments Figures 2-9 The metadata garbage collection method of any of the embodiments.

[0110] To implement the above-mentioned embodiments, the embodiment of the present disclosure further provides an electronic device.

[0111] Reference Figure 12 , which shows a structural diagram of an electronic device 900 suitable for implementing the embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 12 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0112] As shown in Figure 12 , the electronic device 900 can include a processing device (such as a central processor, a graphics processor, etc.) 901, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or loaded into a random access memory (RAM) 903 from a storage device 908. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0113] Generally, the following devices can be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 908 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 909. The communication devices 909 can allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 12The electronic device 900 is illustrated with various means for performing various functions, but it is to be understood that not all of the means illustrated are required to perform the functions. Some of the means can be implemented by software, hardware, or a combination of software and hardware.

[0114] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from the network via the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0115] It is noted that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can, for example and without limitation, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave transporting the program code, in which the computer readable program code can be used by or in connection with an instruction execution system, apparatus or device. The computer readable signal medium can also be any computer readable medium that can transmit, propagate or transport program code used by or in connection with an instruction execution system, apparatus or device. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to wire, cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0116] The computer readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and not be assembled into the electronic device.

[0117] The computer readable medium described above can carry one or more programs, which when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0118] Computer program code for carrying out operations of the present disclosure can be written in any one or more programming languages, including object oriented programming languages such as Java, Smalltalk, C++, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0119] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware-based systems and computer instructions.

[0120] The units or modules involved in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the name of the unit or module does not constitute a limitation on the unit itself.

[0121] The functionality described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0122] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more of: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] In a first aspect, according to one or more embodiments of the present disclosure, a metadata garbage collection method is provided, comprising:

[0124] In response to a garbage collection instruction, a metadata persistent storage area in a partition-based namespace storage device is detected, and a target storage segment is determined, wherein the metadata persistent storage area includes storage segments storing metadata in a sequential writing manner, and the target storage segment is a storage segment to be subjected to garbage collection; for the target storage segment, a corresponding metadata snapshot is generated and stored in the metadata persistent storage area, the metadata snapshot is used to realize data recovery of business data corresponding to the metadata by recording the latest generated metadata in the target storage segment; and data in the target storage segment is cleared.

[0125] According to one or more embodiments of the present disclosure, the metadata persistent storage includes a first metadata persistent storage and a second metadata persistent storage, wherein the first metadata persistent storage is configured to store historical operation records for business data, and the second metadata persistent storage is configured to store metadata snapshots for metadata; and the detecting, in response to a garbage collection instruction, a metadata persistent storage in a partitioned namespace-based storage device and determining a target storage segment includes: in response to a first garbage collection instruction, detecting the first metadata persistent storage and determining a target storage segment in the first metadata persistent storage, and / or in response to a second garbage collection instruction, detecting the second metadata persistent storage and determining a target storage segment in the second metadata persistent storage.

[0126] According to one or more embodiments of the present disclosure, the detecting, in response to a first garbage collection instruction, the first metadata persistent storage and determining a target storage segment in the first metadata persistent storage includes: in response to the first garbage collection instruction, obtaining data write time stamps corresponding to each storage segment in the first metadata persistent storage, and determining, as the target storage segment, a storage segment corresponding to the earliest write time stamp; and the detecting, in response to a second garbage collection instruction, the second metadata persistent storage and determining a target storage segment in the second metadata persistent storage includes: in response to the second garbage collection instruction, obtaining storage loads corresponding to each storage segment in the second metadata persistent storage, and determining, as the target storage segment, at least one storage segment with the largest storage load, wherein the storage load represents a proportion of invalid data in the storage segment.

[0127] According to one or more embodiments of the present disclosure, the determining, as the target storage segment, the at least one storage segment with the largest storage load includes: obtaining a comprehensive load value of the second metadata persistent storage, and determining, according to the comprehensive load value and preset load mapping information, a target number, wherein the preset load mapping information represents a mapping relationship between the comprehensive load value and the number of storage segments; and determining, as the target storage segment, the target number of storage segments in the second metadata persistent storage with the largest storage load.

[0128] According to one or more embodiments of the present disclosure, the generating, for the target storage segment, a corresponding metadata snapshot and storing the metadata snapshot into the metadata persistent storage includes: generating, for the target storage segment in the first metadata persistent storage, a first metadata snapshot and storing the first metadata snapshot into the second metadata persistent storage, and / or generating, for the target storage segment in the second metadata persistent storage, a second metadata snapshot in the second metadata persistent storage.

[0129] According to one or more embodiments of the present disclosure, the target storage segments include at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; and the generating, for the target storage segments in the second metadata persistent storage area, a second metadata snapshot in the second metadata persistent storage area includes:

[0130] generating a metadata snapshot of each of the target storage segments in the second metadata persistent storage area, and obtaining a data heat corresponding to each of the target storage segments, the data heat representing an update probability of metadata corresponding to the target storage segment; storing the metadata snapshot corresponding to the target storage segment with the data heat greater than a heat threshold value to the first sub-storage area; and storing the metadata snapshot corresponding to the target storage segment with the data heat not greater than the heat threshold value to the second sub-storage area.

[0131] According to one or more embodiments of the present disclosure, the generating, for the target storage segments, a corresponding metadata snapshot includes: obtaining a storage priority of the first metadata persistent storage area and the second metadata persistent storage area; determining a proportion value of a first metadata snapshot and a second metadata snapshot according to the storage priority, wherein the first metadata snapshot is a metadata snapshot generated based on a target storage segment in the first metadata persistent storage area; and the second metadata snapshot is a metadata snapshot generated based on a target storage segment in the second metadata persistent storage area; and executing a metadata migration request based on the proportion value through a pre-configured task queue to generate a corresponding number of first metadata snapshots and second metadata snapshots in a unit of time.

[0132] According to one or more embodiments of the present disclosure, the method further includes: obtaining a comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area; and adjusting the storage priority of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0133] According to one or more embodiments of the present disclosure, before the determining, in response to a garbage collection instruction, a target storage segment in a metadata persistent storage area of a partitioned namespace, the method further includes: obtaining a comprehensive load value of the metadata persistent storage area; determining a trigger recovery load interval according to the comprehensive load value, wherein the comprehensive load value is inversely proportional to an interval median of the trigger recovery load interval; and generating the garbage collection instruction based on the trigger recovery load interval and a storage load of each storage segment in the metadata persistent storage area.

[0134] According to one or more embodiments of the present disclosure, the partition-based namespace-based storage device includes at least one single-machine storage engine file, and the single-machine storage engine file is used to store service data corresponding to the metadata; the method further includes: obtaining validity information of a single-machine storage engine file corresponding to the target storage segment; and generating a corresponding metadata snapshot for the target storage segment includes: according to the validity information of the single-machine storage engine file, if the single-machine storage engine file is in a valid state, generating a corresponding metadata snapshot for the target storage segment.

[0135] In a second aspect, according to one or more embodiments of the present disclosure, a metadata garbage collection apparatus is provided, including:

[0136] The detection module is configured to, in response to a garbage collection instruction, detect a metadata persistent storage area in a partition-based namespace-based storage device, and determine a target storage segment, wherein the metadata persistent storage area includes storage segments storing metadata in a sequential writing manner, the target storage segment is a storage segment to be subjected to garbage collection, and the metadata is used to implement data recovery of corresponding service data.

[0137] The generation module is configured to generate a corresponding metadata snapshot for the target storage segment, and store the metadata snapshot in the metadata persistent storage area, wherein the metadata snapshot is used to record newly generated metadata in the target storage segment.

[0138] The cleaning module is configured to clean data in the target storage segment.

[0139] According to one or more embodiments of the present disclosure, the metadata persistent storage area includes a first metadata persistent storage area and a second metadata persistent storage area, wherein the first metadata persistent storage area is used to store historical operation records of service data, and the second metadata persistent storage area is used to store metadata snapshots of metadata; and the detection module is specifically configured to: in response to a first garbage collection instruction, detect the first metadata persistent storage area, and determine a target storage segment in the first metadata persistent storage area; and / or in response to a second garbage collection instruction, detect the second metadata persistent storage area, and determine a target storage segment in the second metadata persistent storage area.

[0140] According to one or more embodiments of the present disclosure, when detecting the target storage segment in the first metadata persistent storage area in response to the first garbage collection instruction, the detection module is specifically configured to: in response to the first garbage collection instruction, acquire data write time stamps corresponding to each storage segment in the first metadata persistent storage area, and determine a storage segment corresponding to the earliest write time stamp as the target storage segment; and when detecting the target storage segment in the second metadata persistent storage area in response to the second garbage collection instruction, the detection module is specifically configured to: in response to the second garbage collection instruction, acquire storage loads corresponding to each storage segment in the second metadata persistent storage area, and determine at least one storage segment with the largest storage load as the target storage segment, where the storage load represents a proportion of invalid data in the storage segment.

[0141] According to one or more embodiments of the present disclosure, when determining the at least one storage segment with the largest storage load as the target storage segment, the detection module is specifically configured to: acquire a comprehensive load value of the second metadata persistent storage area, and determine a target number according to the comprehensive load value and preset load mapping information, where the preset load mapping information represents a mapping relationship between the comprehensive load value and the number of storage segments; and determine the target number of storage segments with the largest storage load in the second metadata persistent storage area as the target storage segment.

[0142] According to one or more embodiments of the present disclosure, the generation module is specifically configured to: generate a first metadata snapshot for the target storage segment in the first metadata persistent storage area, and store the first metadata snapshot to the second metadata persistent storage area, and / or generate a second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area.

[0143] According to one or more embodiments of the present disclosure, the target storage segment includes at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; when generating the second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area, the generation module is specifically configured to: generate metadata snapshots of each target storage segment in the second metadata persistent storage area, and acquire data hotness corresponding to each target storage segment, where the data hotness represents an update probability of metadata corresponding to the target storage segment; store the metadata snapshot corresponding to the target storage segment with the data hotness greater than a hotness threshold to the first sub-storage area; and store the metadata snapshot corresponding to the target storage segment with the data hotness not greater than the hotness threshold to the second sub-storage area.

[0144] According to one or more embodiments of the present disclosure, the generating module is specifically configured to: acquire storage priorities of the first metadata persistent storage area and the second metadata persistent storage area; determine a proportion value of single first metadata snapshot and second metadata snapshot according to the storage priorities, wherein the first metadata snapshot is a metadata snapshot generated based on a target storage segment in the first metadata persistent storage area; the second metadata snapshot is a metadata snapshot generated based on a target storage segment in the second metadata persistent storage area; and perform metadata migration requests based on the proportion value through a pre-configured task queue to generate corresponding numbers of first metadata snapshots and second metadata snapshots in a unit time.

[0145] According to one or more embodiments of the present disclosure, the generating module is further configured to: acquire comprehensive load values corresponding to the first metadata persistent storage area and the second metadata persistent storage area; and adjust the storage priorities of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load values corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0146] According to one or more embodiments of the present disclosure, before the detecting module detects the metadata persistent storage area in the partitioned namespace-based storage device in response to the garbage collection instruction and determines the target storage segment, the detecting module is further configured to: acquire a comprehensive load value of the metadata persistent storage area; determine a corresponding trigger recovery load interval according to the comprehensive load value, wherein the comprehensive load value is inversely proportional to an interval median value of the trigger recovery load interval; and generate the garbage collection instruction based on the trigger recovery load interval and storage loads of each storage segment in the metadata persistent storage area.

[0147] According to one or more embodiments of the present disclosure, the partitioned namespace-based storage device includes at least one single-machine storage engine file, and the generating module is further configured to: acquire validity information of the single-machine storage engine file corresponding to the target storage segment; and when generating the corresponding metadata snapshot for the target storage segment, the generating module is specifically configured to: generate a metadata snapshot corresponding to the single-machine storage engine file according to the validity information of the single-machine storage engine file.

[0148] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, which includes at least one processor and a memory.

[0149] The memory stores computer execution instructions.

[0150] The at least one processor executes the computer-executed instructions stored in the memory, so that the at least one processor executes the metadata garbage collection method as described in the first aspect and various possible designs of the first aspect.

[0151] In a fourth aspect, a computer-readable storage medium is provided according to one or more embodiments of the present disclosure, and the computer-readable storage medium has stored therein computer-executed instructions which, when executed by a processor, implement the metadata garbage collection method as described in the first aspect and various possible designs of the first aspect.

[0152] In a fifth aspect, a computer program product is provided according to one or more embodiments of the present disclosure, and the computer program product includes a computer program which, when executed by a processor, implements the metadata garbage collection method as described in the first aspect and various possible designs of the first aspect.

[0153] The above description merely illustrates the preferred embodiments of the present disclosure and the principles of the applied technologies. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the above disclosed concepts. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0154] In addition, although each operation is described in a particular order, this should not be understood as requiring the operations to be performed in the particular order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments. The various features described in the context of the above embodiments can be combined in any suitable sub-combination.

[0155] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A metadata garbage collection method, characterized in that, include: In response to a garbage collection command, the persistent metadata storage area in the storage device based on the partition namespace is detected, and the target storage segment is determined. The persistent metadata storage area includes a storage segment that stores metadata in a sequential write manner, and the target storage segment is the storage segment to be garbage collected. For the target storage segment, a corresponding metadata snapshot is generated and stored in the metadata persistent storage area. The metadata snapshot is used to restore the business data corresponding to the metadata by recording the latest generated metadata in the target storage segment. Clear the data in the target storage segment; The metadata persistent storage area includes a first metadata persistent storage area for storing historical operation records of business data and a second metadata persistent storage area for storing metadata snapshots. The step of detecting the metadata persistent storage area in the storage device based on the partition namespace and determining the target storage segment in response to a garbage collection instruction includes: In response to a first garbage collection instruction, the system obtains the data write timestamps corresponding to each storage segment in the first metadata persistent storage area and determines the storage segment corresponding to the earliest write timestamp as the target storage segment. Alternatively, in response to a second garbage collection instruction, the system obtains the storage load corresponding to each storage segment in the second metadata persistent storage area and determines at least one storage segment with the largest storage load as the target storage segment. The storage load is used to characterize the proportion of invalid data in the storage segment.

2. The method according to claim 1, characterized in that, The step of determining at least one storage segment with the highest storage load as the target storage segment includes: Obtain the overall load value of the second metadata persistent storage area, and determine the target quantity based on the overall load value; The target storage segment is determined as the storage segment with the largest storage load in the second metadata persistent storage area.

3. The method according to claim 1, characterized in that, The step of generating a corresponding metadata snapshot for the target storage segment and storing the metadata snapshot in the metadata persistent storage area includes: For the target storage segment in the first metadata persistent storage area, generate a first metadata snapshot and store the first metadata snapshot in the second metadata persistent storage area, and / or; For the target storage segment in the second metadata persistent storage area, a second metadata snapshot is generated in the second metadata persistent storage area.

4. The method according to claim 3, characterized in that, The target storage segment includes at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; The step of generating a second metadata snapshot in the second metadata persistent storage area for a target storage segment in the second metadata persistent storage area includes: Generate metadata snapshots for each of the target storage segments in the second metadata persistent storage area, and obtain the data heat corresponding to each target storage segment, wherein the data heat represents the update probability of the metadata corresponding to the target storage segment; The target storage segment with data popularity greater than the popularity threshold, corresponding to the metadata snapshot, is stored in the first sub-storage area; The target storage segment with a data popularity not greater than the popularity threshold is corresponding to the metadata snapshot and stored in the second sub-storage area.

5. The method according to claim 1, characterized in that, The step of generating a corresponding metadata snapshot for the target storage segment includes: Obtain the storage priorities of the first metadata persistent storage area and the second metadata persistent storage area; Based on the storage priority, the ratio of the first metadata snapshot to the second metadata snapshot is determined, wherein the first metadata snapshot is a metadata snapshot generated based on the target storage segment within the first metadata persistent storage area; and the second metadata snapshot is a metadata snapshot generated based on the target storage segment within the second metadata persistent storage area. Using a pre-configured task queue, metadata migration requests are executed based on the ratio value, generating a corresponding number of first and second metadata snapshots within a unit of time.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the combined load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area; Based on the combined load values ​​corresponding to the first and second metadata persistent storage areas, adjust the storage priorities of the first and second metadata persistent storage areas.

7. The method according to claim 1, characterized in that, Before detecting the metadata persistent storage area in the storage device based on the partition namespace in response to a garbage collection instruction and determining the target storage segment, the method further includes: Obtain the overall load value of the metadata persistent storage area; Based on the comprehensive load value, a corresponding trigger recovery load range is determined, wherein the comprehensive load value is inversely proportional to the median value of the trigger recovery load range; The garbage collection instruction is generated when the storage load of at least one storage segment in the metadata persistent storage area is within the trigger recycling load range.

8. The method according to claim 1, characterized in that, The partition-based namespace storage device includes at least one single-machine storage engine file, which is used to store business data corresponding to the metadata. The method further includes: Obtain the validity information of the single-machine storage engine file corresponding to the target storage segment; the validity information is used to indicate the validity of the single-machine storage engine file. The step of generating a corresponding metadata snapshot for the target storage segment includes: Based on the validity information of the single-machine storage engine file, if the single-machine storage engine file is in a valid state, a corresponding metadata snapshot is generated for the target storage segment.

9. A metadata garbage collection device, characterized in that, include: The detection module is used to detect the persistent metadata storage area in the storage device based on the partition namespace in response to the garbage collection command, and determine the target storage segment. The persistent metadata storage area includes a storage segment that stores metadata in a sequential write manner. The target storage segment is the storage segment to be garbage collected. The metadata is used to realize the data recovery of the corresponding business data. The generation module is used to generate a corresponding metadata snapshot for the target storage segment and store the metadata snapshot in the metadata persistent storage area. The metadata snapshot is used to record the latest generated metadata in the target storage segment. The cleanup module is used to clear the data in the target storage segment; The metadata persistent storage area includes a first metadata persistent storage area for storing historical operation records of business data and a second metadata persistent storage area for storing metadata snapshots. The detection module is specifically configured to: in response to a first garbage collection instruction, obtain the data write timestamps corresponding to each storage segment in the first metadata persistent storage area, and determine the storage segment corresponding to the earliest write timestamp as the target storage segment; or, in response to a second garbage collection instruction, obtain the storage load corresponding to each storage segment in the second metadata persistent storage area, and determine at least one storage segment with the largest storage load as the target storage segment, wherein the storage load is used to characterize the proportion of invalid data in the storage segment.

10. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the metadata garbage collection method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the metadata garbage collection method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the metadata garbage collection method as described in any one of claims 1 to 8.

Images