Metadata garbage collection method and device, electronic equipment and storage medium

By detecting the metadata persistent storage area in the partitioned namespace-based storage device, determining the target storage segment, generating metadata snapshots and clearing data, the problems of insufficient metadata storage space and low recycling efficiency are solved, and the stability and data recovery capabilities of the storage device are improved.

CN119576226AActive Publication Date: 2025-03-07BEIJING VOLCANO ENGINE TECH CO LTD

Patent Information

Application Number
CN202411622074.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-13
Publication Date
2025-03-07
Estimated Expiration
2044-11-13

AI Technical Summary

Technical Problem

In storage devices based on partition namespaces, the metadata storage space is insufficient and the space recycling efficiency is low, which affects the operating stability of the storage device.

Method used

By detecting the metadata persistent storage area in the storage device, determining the target storage segment, generating a metadata snapshot and storing, and clearing the data in the target storage segment to achieve garbage collection.

Benefits of technology

It improves the utilization rate of storage space and space recycling efficiency, enhances the operation stability of storage devices, and ensures the ability to recover data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119576226A_ABST
    Figure CN119576226A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a metadata garbage collection method and device, electronic equipment and a storage medium, by responding to a garbage collection instruction, a metadata persistent storage area in storage equipment based on partition namespace is detected, and a target storage segment is determined, the metadata persistent storage area comprises a storage section for storing metadata based on a sequential writing mode, the target storage section is a storage section to be subjected to garbage collection, and the metadata is used for realizing data recovery of corresponding business data; aiming at the target storage section, generating a corresponding metadata snapshot, and storing the metadata snapshot into a metadata persistent storage area, the metadata snapshot being used for recording newly generated metadata in the target storage section; and clearing the data in the target storage section. Data recovery of the storage device can be realized, the utilization rate and the space recovery efficiency of the storage space are improved, and the operation stability of the storage device is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of cloud storage technology, and in particular to a metadata garbage collection method, device, electronic device, and storage medium. Background Art

[0002] In the field of cloud storage technology, storage devices based on Zoned Namespaces (ZNS) technology divide the storage space into multiple storage segments for management, thereby optimizing the data storage and access process and improving data read and write performance.

[0003] In the prior art, in order to increase the user storage space of the storage device, usually only a small portion of the space is reserved for storing metadata for data recovery. At the same time, because data can only be written sequentially and cannot be overwritten in storage devices based on partition namespace technology, this leads to insufficient storage space for metadata and low space recovery efficiency, which affects the operating stability of the storage device. Summary of the invention

[0004] The embodiments of the present disclosure provide a metadata garbage collection method, device, electronic device and storage medium to overcome the problems of insufficient metadata storage space and low space recovery efficiency.

[0005] In a first aspect, an embodiment of the present disclosure provides a metadata garbage collection method, comprising:

[0006] In response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace is detected, and a target storage segment is determined, wherein the metadata persistent storage area includes a storage segment for storing metadata based on a sequential write method, the target storage segment is a storage segment to be garbage collected, and the metadata is used to achieve data recovery of corresponding business data; for the target storage segment, a corresponding metadata snapshot is generated, and the metadata snapshot is stored in the metadata persistent storage area, and the metadata snapshot is used to record the latest metadata generated in the target storage segment; and the data in the target storage segment is cleared.

[0007] In a second aspect, an embodiment of the present disclosure provides a metadata garbage collection device, including:

[0008] A detection module, configured to detect, in response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace, and determine a target storage segment, wherein the metadata persistent storage area includes a storage segment storing metadata based on a sequential write method, the target storage segment is a storage segment to be garbage collected, and the metadata is used to realize data recovery of corresponding business data;

[0009] A generation module, used for generating a corresponding metadata snapshot for the target storage segment, and storing the metadata snapshot in the metadata persistent storage area, wherein the metadata snapshot is used for recording the latest metadata generated in the target storage segment;

[0010] The cleaning module is used to clear the data in the target storage segment.

[0011] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0012] The memory stores computer-executable instructions;

[0013] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the metadata garbage collection method as described in the first aspect and various possible designs of the first aspect.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the metadata garbage collection method described in the first aspect and various possible designs of the first aspect is implemented.

[0015] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the metadata garbage collection method as described in the first aspect and various possible designs of the first aspect.

[0016] The metadata garbage collection method, device, electronic device and storage medium provided in this embodiment detect the metadata persistent storage area in the storage device based on the partition namespace in response to the garbage collection instruction, determine the target storage segment, wherein the metadata persistent storage area includes a storage segment storing metadata based on sequential writing, the target storage segment is the storage segment to be garbage collected, and the metadata is used to realize data recovery of the corresponding business data; for the target storage segment, generate a corresponding metadata snapshot, and store the metadata snapshot in the metadata persistent storage area, the metadata snapshot is used to record the latest metadata generated in the target storage segment; clear the data in the target storage segment. By determining the target storage segment that needs to be garbage collected in the metadata persistent storage area, and triggering the snapshot generation event for the target storage segment, a data snapshot of the metadata in the target storage segment, i.e., a metadata snapshot, is generated, and then the metadata snapshot is stored in the metadata persistent storage area, and the data in the target storage segment is cleared to complete the garbage collection of the target storage segment. At the same time, by saving the metadata snapshot in the metadata persistent storage area, data recovery of the storage device can be realized, the utilization rate of storage space and the space recovery efficiency can be improved, and the operation stability of the storage device can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 An application scenario diagram of the metadata garbage collection method provided by the embodiment of the present disclosure;

[0019] Figure 2 Schematic diagram of the process of metadata garbage collection method provided by the embodiment of the present disclosure Figure 1 ;

[0020] Figure 3 A schematic diagram of a metadata persistent storage area provided in an embodiment of the present disclosure;

[0021] Figure 4 A schematic diagram of a process for determining a target storage segment provided by an embodiment of the present disclosure;

[0022] Figure 5 for Figure 2 A flowchart of a specific implementation method of step S102 in the illustrated embodiment;

[0023] Figure 6Schematic diagram of the process of metadata garbage collection method provided by the embodiment of the present disclosure Figure 2 ;

[0024] Figure 7 for Figure 6 A flowchart of a specific implementation method of step S200 in the illustrated embodiment;

[0025] Figure 8 A schematic diagram of a process for persisting metadata snapshots provided in an embodiment of the present disclosure;

[0026] Fig. 9 for Figure 6 A flowchart of a specific implementation method of step S203 in the illustrated embodiment;

[0027] Fig.10 A structural block diagram of a metadata garbage collection device provided by an embodiment of the present disclosure;

[0028] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0029] Fig.12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] The application scenarios of the embodiments of the present disclosure are explained below:

[0033] Figure 1A diagram of an application scenario of the metadata garbage collection method provided in the embodiment of the present disclosure. The metadata garbage collection method provided in the embodiment of the present disclosure can be applied to the application scenario of cloud storage, more specifically, it can be applied to the application scenario of distributed cloud storage. The execution subject of this embodiment can be a storage node in a distributed cloud storage system, such as a storage device or other electronic device with similar functions. Among them, in some embodiments, the storage device can implement the metadata garbage collection method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, the computer executable instructions can be program-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local application, that is, a program that needs to be installed in the operating system to run, or it can be a small program embedded in any APP, that is, a program running based on a browser environment. In summary, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form, and the specific implementation form can be configured as needed. Further, in the process of implementing the metadata garbage collection method provided in the embodiment of the present application, the storage device can execute the method by running a computer executable instruction or computer program set locally, or by calling a computer executable instruction or computer program set in an external server. In some embodiments, the above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communications, cloud databases, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0034] refer to Figure 1 As shown in , in a storage device based on a partition namespace, the storage space is abstracted into continuous storage segments through a specific interface. The storage segment is a mapping of the physical storage medium at the software layer, wherein most of the storage segments are used to store business data, i.e., business data storage areas, such as Segment_00 to Segment_95 shown in the figure; and another small part of the storage segments are used to store metadata corresponding to the business data, such as Segment_96 to Segment_99, i.e., metadata persistent storage areas. After the partition is formatted, the above-mentioned space division method is also determined, and the space size of the storage segments used to store business data and the storage segments used to store metadata will no longer change.

[0035] Due to the technical characteristics of the partitioned namespace (ZNS) technology, metadata, like business data, can only be written sequentially to the storage segment, usually by writing operation logs into the storage segment. After the storage segment is full, if you need to reuse the space in the storage segment, you need to migrate the valid data in the storage segment first, and then erase the storage segment before you can reuse it. However, since the storage space used to store metadata corresponding to business data is small, when business data is frequently modified, a large number of operation logs will be generated, causing the corresponding storage segment to be filled up and triggering the garbage collection process. The frequent garbage collection process will cause the logs to be moved repeatedly, thereby generating additional resource overhead, resulting in insufficient storage space for metadata and low space recovery efficiency.

[0036] The embodiment of the present disclosure provides a metadata garbage collection method to solve the above-mentioned problem.

[0037] refer to Figure 2 , Figure 2 Schematic diagram of the process of metadata garbage collection method provided by the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a storage device. The metadata garbage collection method includes:

[0038] Step S101: In response to a garbage collection instruction, detect a metadata persistent storage area in a storage device based on a partition namespace, and determine a target storage segment, wherein the metadata persistent storage area includes a storage segment for storing metadata based on a sequential write method, and the target storage segment is a storage segment to be garbage collected.

[0039] For example, reference Figure 1As shown in the schematic diagram of the application scenario, the execution subject of the method provided in this embodiment can be a storage device or a control unit in the storage device (the request of the control unit as the execution subject will be introduced later as an example), wherein one or more storage segments are provided in the storage device, and the storage device controls one or more storage segments through the control unit, thereby realizing the metadata garbage collection method provided in this embodiment. Specifically, the control unit detects the metadata persistent storage area in the storage device based on the partition name space (ZNS) in response to the garbage collection instruction, and the metadata persistent storage area is a collection of storage spaces in the storage device for storing metadata-related information. Among them, the metadata persistent storage area includes one or more storage segments (Segment), each storage segment is used to store corresponding metadata, and the metadata in the storage segment is stored based on a sequential write method. The control unit detects each storage segment in the metadata persistent storage area through a preset detection logic, and determines one or more storage segments as the target storage segments. Specifically, the control unit determines the target storage segment by detecting the storage load of each storage segment. In a possible implementation, the storage load represents the proportion of invalid data in the storage segment. In this case, one or more storage segments with the largest proportion of invalid data are determined as the target storage segment; wherein invalid data refers to expired metadata in the data segment, and the higher the proportion of invalid data, the more garbage collection is needed to release its storage space.

[0040] In one possible implementation, Figure 3 A schematic diagram of a metadata persistent storage area provided in an embodiment of the present disclosure, such as Figure 3As shown, the metadata persistent storage area includes a first metadata persistent storage area and a second metadata persistent storage area. The first metadata persistent storage area is used to store historical operation records for business data, wherein the operation record (journal log) records the creation, freezing, discarding and deletion of a single-machine storage engine file (chunk), as well as the opening, closing, erasing and other operation records for a storage segment. The single-machine storage engine file is a subunit in a storage segment for storing business data, which is a concept in the prior art and will not be repeated here. Through the operation record, the operation of business data can be reproduced, and then the data recovery of business data can be achieved. Therefore, the first metadata persistent storage area for storing historical operation records is also the operation log persistent storage area (journal segment). The second metadata persistent storage area is used to store snapshot (checkpoint) data for metadata, that is, metadata snapshots. Among them, generating a snapshot is a data persistence technology. Generating a snapshot of the above storage segment or stand-alone storage engine file will persist the relevant information of the metadata of the storage segment or stand-alone storage engine file, thereby realizing data recovery of business data. At the same time, the metadata snapshot has a better data recovery priority. Therefore, the metadata snapshot will invalidate the operation log of the previously generated storage segment or stand-alone storage engine file. The second metadata persistent storage area is used to store the metadata snapshot generated by the storage system, that is, the metadata snapshot persistent storage area (matasegment).

[0041] Accordingly, the specific implementation of step S101 includes: responding to the first garbage collection instruction, detecting the first metadata persistent storage area, determining the target storage segment in the first metadata persistent storage area, and / or responding to the second garbage collection instruction, detecting the second metadata persistent storage area, and determining the target storage segment in the second metadata persistent storage area. That is, when the garbage collection instruction responded by the control unit is the first garbage collection instruction, the first metadata persistent storage area is detected, and the target storage segment in the first metadata persistent storage area is determined, so that in the subsequent steps, the garbage collection of the first metadata persistent storage area is completed; when the garbage collection instruction responded by the control unit is the second garbage collection instruction, the second metadata persistent storage area is detected, and the target storage segment in the second metadata persistent storage area is determined, so that in the subsequent steps, the garbage collection of the second metadata persistent storage area is completed.

[0042] It can be understood that, in another possible implementation, the garbage collection instruction can also include the above-mentioned first garbage collection instruction and the second garbage collection instruction at the same time, that is, the control unit detects the target storage segment in the first metadata persistent storage area and the target storage segment in the second metadata persistent storage area respectively by responding to the garbage echo instruction, thereby completing the garbage collection of the first metadata persistent storage area and the second metadata persistent storage area. The specific settings can be made as needed and will not be repeated here.

[0043] Furthermore, in response to the first garbage collection instruction, in which garbage collection is performed on the first metadata persistent storage area, when the first data storage area is detected, the data write timestamps corresponding to each storage segment in the first metadata persistent storage area are obtained, and the storage segment corresponding to the earliest write timestamp is determined as the target storage segment; and in response to the second garbage collection instruction, in which garbage collection is performed on the second metadata persistent storage area, when the second data storage area is detected, the storage loads corresponding to each storage segment in the second metadata persistent storage area are obtained, and at least one storage segment with the largest storage load is determined as the target storage segment.

[0044] Figure 4 A schematic diagram of a process for determining a target storage segment provided by an embodiment of the present disclosure, such as Figure 4As shown, when the control unit detects the first data storage area in response to the first garbage collection instruction, since the first data storage area is used to store historical operation records for business data, the historical operation records have a time series attribute. Therefore, when using the historical operation records for data recovery, the playback must be strictly performed in chronological order. In view of this feature, when garbage collection is performed on the storage segments in the first data storage area (shown as journal_segment_1, journal_segment_2, etc. in the figure, and the storage loads are storage load = 30% and storage load = 76%, respectively), the earliest storage segment will be given priority, that is, the data write timestamps corresponding to each storage segment in the first metadata persistent storage area are obtained, and the storage segment corresponding to the earliest write timestamp is determined as the target storage segment, such as the storage segment journal_segment_1 shown in the figure. When the control unit detects the second data storage area in response to the second garbage collection instruction, since the second data storage area is used to store metadata snapshots for metadata, data recovery of business data corresponding to the metadata can be achieved through the metadata snapshot, which is not affected by log time. Therefore, the target storage segment can be determined directly through the storage load of each storage segment in the second data storage, that is, one or more storage segments with the largest storage load are determined as target storage segments. For example, as shown in the figure, the two storage segments meta_segment_1 (storage load = 60%) and meta_segment_2 (storage load = 75%) with the largest storage load are determined as target storage segments.

[0045] Furthermore, a specific implementation method of determining at least one storage segment with the largest storage load as the target storage segment includes:

[0046] Step S1011: Obtain a comprehensive load value of the second metadata persistent storage area, and determine a target quantity according to the comprehensive load value and preset load mapping information, wherein the preset load mapping information represents a mapping relationship between the comprehensive load value and the quantity of storage segments.

[0047] Step S1012: Determine the target number of storage segments with the largest storage load in the second metadata persistent storage area as target storage segments.

[0048] Exemplarily, before determining the target storage segment, the control unit first obtains the comprehensive load value of the second metadata persistent storage area, that is, the current usage of the storage segment used to store the metadata snapshot, and then determines the number of target storage segments from the second metadata persistent storage area based on the comprehensive load value, that is, the target number. Among them, the comprehensive load value is in direct proportion to the target number, that is, within a certain range, the larger the comprehensive load value, the greater the target number. For example, when the comprehensive load value is 60%, the target number is 1, that is, a storage segment with the largest storage load is selected and determined as the target storage segment; when the comprehensive load value is 80%, the target number is 3, that is, the three storage segments with the largest storage load are selected and determined as the target storage segments. Through the steps of this embodiment, when the comprehensive load value of the second metadata persistent storage area is large, the garbage cleaning amount can be increased, thereby quickly reducing the comprehensive load value and improving the efficiency of garbage cleaning.

[0049] Step S102: Generate a corresponding metadata snapshot for the target storage segment, and store the metadata snapshot in the metadata persistent storage area. The metadata snapshot is used to achieve data recovery of business data corresponding to the metadata by recording the latest metadata generated in the target storage segment.

[0050] Step S103: Clear the data in the target storage segment.

[0051] Exemplarily, after determining the target storage segment in the metadata persistent storage area, a corresponding metadata snapshot is generated for the target storage segment, wherein the implementation process of generating the metadata snapshot for the target storage segment can be implemented through the functional interface provided by the ZNS system, and the specific process of generating the metadata snapshot is not described in detail this time. Afterwards, the metadata snapshot is stored in the metadata persistent storage area to achieve persistence of the metadata snapshot. After the metadata snapshot is persisted, when the system restarts and business data needs to be restored, the corresponding metadata snapshot can be read from the above-mentioned metadata persistent storage area (persistent storage area), and based on the metadata snapshot, the business data can be restored to the state when the metadata snapshot was generated, thereby achieving recovery of the business data. After completing the above-mentioned metadata snapshot generation and persistence process, the data in the above-mentioned target storage segment can be cleared, or the target storage segment can be deleted, thereby achieving the purpose of garbage collection.

[0052] In one possible implementation, a storage device based on a partition namespace includes at least one stand-alone storage engine file, which is used to store business data corresponding to the metadata; before step S102, it also includes: obtaining validity information of the stand-alone storage engine file corresponding to the target storage segment, and the validity information is used to indicate the validity of the stand-alone storage engine file.

[0053] Correspondingly, the specific implementation of step S102 includes: according to the validity information of the stand-alone storage engine file, if the stand-alone storage engine file is in a validity state, generating a corresponding metadata snapshot for the target storage segment.

[0054] Exemplarily, in the steps of this embodiment, before generating a metadata snapshot, the control unit first obtains and reads the stand-alone storage engine file corresponding to the metadata in the target storage segment to detect the validity information of the stand-alone storage engine file. If the stand-alone storage engine file is deleted or moved out to other storage devices or media, it is determined that the stand-alone storage engine file is in an invalid state. In this case, the subsequent metadata snapshot generation steps may not be performed to save resource overhead. On the other hand, if the stand-alone storage engine file continues to exist normally, it is determined that the stand-alone storage engine file is in a valid state. In this case, the corresponding functional interface is called for the stand-alone storage engine file to generate the corresponding metadata snapshot.

[0055] Further, in a possible implementation, in a possible implementation, as Figure 5 As shown, the specific implementation of step S102 includes:

[0056] Step S1021: obtaining storage priorities of the first metadata persistent storage area and the second metadata persistent storage area;

[0057] Step S1022: determining a ratio value between a first metadata snapshot and a second metadata snapshot according to the storage priority, wherein the first metadata snapshot is a metadata snapshot generated based on the target storage segment in the first metadata persistent storage area; and the second metadata snapshot is a metadata snapshot generated based on the target storage segment in the second metadata persistent storage area;

[0058] Step S1023: executing metadata relocation requests based on the ratio values ​​through the pre-configured task queue, and generating a corresponding number of first metadata snapshots and second metadata snapshots within a unit time.

[0059] Step S1024: storing the first metadata snapshot and the second metadata snapshot in the metadata persistent storage area.

[0060] Exemplarily, the process of generating a metadata snapshot is equivalent to the process of generating a metadata snapshot, so for example, a certain amount of computing resources will be consumed. In the steps of this embodiment, in the process of generating a metadata snapshot, the storage priority of the first metadata persistent storage area and the second metadata persistent storage area is first determined, and then, based on the storage priority, it is converted into a corresponding ratio value, and then, through a pre-configured task queue, the metadata relocation request is respectively executed based on the ratio value, and a corresponding number of first metadata snapshots and second metadata snapshots are generated in a unit time. Specifically, the first metadata snapshot and the second metadata snapshot are the latest metadata information of the relocation chunk involved. This process is equivalent to generating events for the snapshots corresponding to the target storage segments in the first metadata persistent storage area and the second metadata persistent storage area based on the ratio value, and respectively allocating corresponding computing resources, thereby realizing dynamic adjustment of the garbage collection process and improving the overall garbage collection efficiency of the storage device.

[0061] Furthermore, the steps of this embodiment also include:

[0062] Step S1025: Obtain the comprehensive load values ​​corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0063] Step S1026: adjusting the storage priorities of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load values ​​corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0064] Exemplarily, based on the steps of the above embodiment, the control unit may further adjust the storage priority of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area. For example, when the comprehensive load value of the first metadata persistent storage area is greater than a preset load ratio, and / or the comprehensive load value of the first metadata persistent storage area is greater than a preset multiple of the comprehensive load value of the second metadata persistent storage area, the corresponding storage priority of the first metadata persistent storage area is increased. For another example, when the comprehensive load value of the second metadata persistent storage area is greater than a preset load ratio, and / or the comprehensive load value of the second metadata persistent storage area is greater than a preset multiple of the comprehensive load value of the first metadata persistent storage area, the corresponding storage priority of the second metadata persistent storage area is increased. The specific implementation method can be set according to the resource configuration and specific requirements of the storage device, which will not be repeated here.

[0065] In this embodiment, by responding to a garbage collection instruction, detecting a metadata persistent storage area in a storage device based on a partition namespace, determining a target storage segment, wherein the metadata persistent storage area includes a storage segment storing metadata based on a sequential write method, the target storage segment is a storage segment to be garbage collected, and the metadata is used to realize data recovery of corresponding business data; for the target storage segment, generating a corresponding metadata snapshot, and storing the metadata snapshot in the metadata persistent storage area, the metadata snapshot is used to record the latest metadata generated in the target storage segment; clearing the data in the target storage segment. By determining the target storage segment that needs to be garbage collected in the metadata persistent storage area, and triggering a snapshot generation event for the target storage segment, generating a data snapshot of the metadata in the target storage segment, i.e., a metadata snapshot, then storing the metadata snapshot in the metadata persistent storage area, and clearing the data in the target storage segment, completing garbage collection of the target storage segment, and by saving the metadata snapshot in the metadata persistent storage area, data recovery of the storage device can be realized, the utilization rate of storage space and the efficiency of space recovery can be improved, and the operation stability of the storage device can be improved.

[0066] refer to Figure 6 , Figure 6 Schematic diagram of the process of metadata garbage collection method provided by the embodiment of the present disclosure Figure 2 In this embodiment Figure 2 On the basis of the illustrated embodiment, steps S102-103 are further refined, and the metadata garbage collection method includes:

[0067] Step S200: Generate a garbage collection instruction, where the garbage collection instruction includes a first garbage collection instruction and / or a second garbage collection instruction.

[0068] Exemplarily, during operation of the storage device, a control unit generates garbage collection instructions using preset program logic, for example, based on preset time intervals to generate garbage collection instructions, thereby triggering a subsequent metadata garbage collection process; or, garbage collection instructions are generated by detecting indicators in the storage device.

[0069] In one possible implementation, Figure 7 As shown, the specific implementation of step S200 includes:

[0070] Step S2001: Obtain the comprehensive load value of the metadata persistent storage area;

[0071] Step S2002: determining a corresponding trigger recovery load interval according to the comprehensive load value, wherein the comprehensive load value is inversely proportional to the median of the trigger recovery load interval;

[0072] Step S2003: When the storage load of at least one storage segment in the metadata persistent storage area is located in the triggering collection load interval, a garbage collection instruction is generated.

[0073] Exemplarily, the control unit first obtains the comprehensive load value of the metadata persistent storage area, and the comprehensive load value may be the data storage amount, occupancy amount, occupancy rate, etc. of the metadata persistent storage area. Afterwards, the corresponding trigger recovery load interval is determined according to the comprehensive load value, wherein the comprehensive load value is inversely proportional to the median value of the interval of the trigger recovery load interval. Specifically, the higher the comprehensive load value, the lower the interval value of the corresponding trigger recovery load interval. The interval value of the trigger recovery load interval here may refer to the upper limit value, or the lower limit value, or the median value of the interval of the trigger recovery load interval, without limitation. For example, the interval value of the trigger recovery load interval is, for example, the lower limit value of the trigger recovery load interval. For example, when the comprehensive load value is 50%, the corresponding trigger recovery load interval is 70%, that is, the lower limit value of the trigger recovery load interval is 70%, and the upper limit value is fixed at 100%. Afterwards, based on the trigger recovery load interval and the storage load of each storage segment in the metadata persistent storage area, a garbage collection instruction is generated, that is, when the storage load of the storage segment is in the trigger recovery load interval (i.e., greater than 70%), a garbage collection instruction is generated; and when the comprehensive load value is 70%, the corresponding trigger recovery load interval is 50%, that is, the lower limit value of the trigger recovery load interval is 50%, and the upper limit value is fixed at 100%. Afterwards, based on the trigger recovery load interval and the storage load of each storage segment in the metadata persistent storage area, a garbage collection instruction is generated, that is, when the storage load of the storage segment is in the trigger recovery load interval (i.e., greater than 50%), a garbage collection instruction is generated.

[0074] In the steps of this embodiment, the corresponding trigger recovery load interval is dynamically determined according to the comprehensive load value of the metadata persistent storage area, thereby realizing dynamic adjustment of the garbage collection timing and frequency, avoiding performance loss caused by excessive garbage collection, and improving the overall read and write efficiency of the storage device.

[0075] Step S201: In response to a first garbage collection instruction, detect a first metadata persistent storage area and determine a target storage segment in the first metadata persistent storage area, and / or, in response to a second garbage collection instruction, detect a second metadata persistent storage area and determine a target storage segment in the second metadata persistent storage area.

[0076] Step S202: generating a first metadata snapshot for the target storage segment in the first metadata persistent storage area, and storing the first metadata snapshot in the second metadata persistent storage area.

[0077] Step S203: for the target storage segment in the second metadata persistent storage area, generate a second metadata snapshot in the second metadata persistent storage area.

[0078] Exemplarily, after generating a garbage collection instruction, according to the specific implementation content of the garbage collection instruction, that is, the first garbage collection instruction or the second garbage collection instruction, the control unit executes corresponding steps to determine the target storage segment in the first metadata persistent storage area and / or the target storage segment in the second metadata persistent storage area. Thereafter, in response to the first garbage collection instruction, a first metadata snapshot is generated for the target storage segment in the first metadata persistent storage area, and the first metadata snapshot is stored in the second metadata persistent storage area; and in response to the second garbage collection instruction, a second metadata snapshot is generated in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area. Figure 8 A schematic diagram of a process for persisting metadata snapshots provided in an embodiment of the present disclosure is shown in FIG. Figure 8 As shown, exemplarily, in response to a first garbage collection instruction, a first metadata snapshot is generated for a target storage segment in a first metadata persistent storage area (journal_segment), and then the first metadata snapshot is transferred to a second metadata persistent storage area (meta_segment), and in response to a second garbage collection instruction, a second metadata snapshot is generated directly in the second metadata persistent storage area (meta_segment) for the target storage segment in the second metadata persistent storage area (journal_segment).

[0079] Among them, in other possible implementations, step S202 and step S203 can be performed separately, that is, in one possible embodiment, by performing step S200, step S201, step S202, and step S204, garbage collection of the first metadata persistent storage area is completed; or, in another possible embodiment, by performing step S200, step S201, step S203, and step S204, garbage collection of the second metadata persistent storage area is completed. As described in the above embodiment, step S202 and step S203 are performed simultaneously, which is determined based on specific garbage collection instructions, and will not be repeated here.

[0080] Exemplarily, the target storage segment includes at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; for the target storage segment in the second metadata persistent storage area, when a second metadata snapshot is generated in the second metadata persistent storage area, as follows Fig. 9 As shown, the specific implementation of step S203 includes:

[0081] Step S203 - 1 : Generate a metadata snapshot of each target storage segment in the second metadata persistent storage area, and obtain the data heat corresponding to each target storage segment, where the data heat represents the update probability of the metadata corresponding to the target storage segment.

[0082] Step S203 - 2 : storing the metadata snapshot corresponding to the target storage segment whose data heat is greater than the heat threshold in the first sub-storage area.

[0083] Step S203 - 3 : storing the metadata snapshot corresponding to the target storage segment whose data heat is not greater than the heat threshold in the second sub-storage area.

[0084] Exemplarily, in one possible scenario, the second metadata persistent storage area includes multiple target storage segments, and a first sub-storage area and a first sub-storage area for storing metadata snapshots; in this case, when a snapshot generation event for a target storage segment is triggered, the data heat corresponding to each target storage segment is first determined, and the data heat represents the update probability of the metadata corresponding to the target storage segment. In short, for metadata (corresponding storage segments) that are no longer updated or have a low probability of being updated, the number and frequency of generating new metadata snapshots can be reduced by storing the metadata snapshots generated by them in the second sub-storage area, and by reducing the frequency of detecting the second sub-storage area, thereby saving computing resource overhead; and for metadata (corresponding storage segments) that are frequently updated or have a high probability of being updated, the metadata snapshots generated by them can be stored in the first sub-storage area, thereby ensuring the real-time nature of data recovery and improving data security.

[0085] Step S204: clear the data in the target storage segment.

[0086] In this embodiment, the implementation method of step S204 is the same as that of the present disclosure. Figure 2 The implementation method of step S103 in the illustrated embodiment is the same and will not be described in detail here.

[0087] Corresponding to the metadata garbage collection method of the above embodiment, Fig.10 A block diagram of the structure of a metadata garbage collection device provided in an embodiment of the present disclosure. The method described in the above embodiment can be executed by the metadata garbage collection device, which can be implemented by software and / or hardware, and can be integrated in an electronic device with certain data processing functions. The electronic device can include but is not limited to a mobile terminal with big data processing capabilities, and a fixed terminal with big data processing capabilities such as a desktop computer and a supercomputer.

[0088] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Fig.10, the metadata garbage collection device 3 includes:

[0089] A detection module 31 is used to detect, in response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace, and determine a target storage segment, wherein the metadata persistent storage area includes a storage segment storing metadata based on a sequential write method, the target storage segment is a storage segment to be garbage collected, and the metadata is used to realize data recovery of corresponding business data;

[0090] A generating module 32 is used to generate a corresponding metadata snapshot for the target storage segment, and store the metadata snapshot in a metadata persistent storage area, wherein the metadata snapshot is used to record the latest metadata generated in the target storage segment;

[0091] The cleaning module 33 is used to clear the data in the target storage segment.

[0092] According to one or more embodiments of the present disclosure, the metadata persistent storage area includes a first metadata persistent storage area and a second metadata persistent storage area, wherein the first metadata persistent storage area is used to store historical operation records for business data, and the second metadata persistent storage area is used to store metadata snapshots for metadata; the detection module 31 is specifically used to: respond to a first garbage collection instruction, detect the first metadata persistent storage area, determine the target storage segment in the first metadata persistent storage area, and / or, respond to a second garbage collection instruction, detect the second metadata persistent storage area, and determine the target storage segment in the second metadata persistent storage area.

[0093] According to one or more embodiments of the present disclosure, when the detection module 31 detects the first metadata persistent storage area in response to a first garbage collection instruction and determines the target storage segment in the first metadata persistent storage area, it is specifically used to: in response to the first garbage collection instruction, obtain the data write timestamps corresponding to each storage segment in the first metadata persistent storage area, and determine the storage segment corresponding to the earliest write timestamp as the target storage segment; when the detection module 31 detects the second metadata persistent storage area in response to a second garbage collection instruction and determines the target storage segment in the second metadata persistent storage area, it is specifically used to: in response to the second garbage collection instruction, obtain the storage loads corresponding to each storage segment in the second metadata persistent storage area, and determine at least one storage segment with the largest storage load as the target storage segment, wherein the storage load is used to characterize the proportion of invalid data in the storage segment.

[0094] According to one or more embodiments of the present disclosure, when the detection module 31 determines at least one storage segment with the largest storage load as the target storage segment, it is specifically used to: obtain the comprehensive load value of the second metadata persistent storage area, and determine the target number based on the comprehensive load value; determine the storage segments with the target number of the largest storage loads in the second metadata persistent storage area as the target storage segments.

[0095] According to one or more embodiments of the present disclosure, the generation module 32 is specifically used to: generate a first metadata snapshot for a target storage segment in a first metadata persistent storage area, and store the first metadata snapshot in a second metadata persistent storage area, and / or; generate a second metadata snapshot in the second metadata persistent storage area for a target storage segment in a second metadata persistent storage area.

[0096] According to one or more embodiments of the present disclosure, the target storage segment includes at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; when the generation module 32 generates a second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area, it is specifically used to: generate metadata snapshots of each target storage segment in the second metadata persistent storage area, and obtain the data heat corresponding to each target storage segment, the data heat characterizing the update probability of the metadata corresponding to the target storage segment; store the metadata snapshot corresponding to the target storage segment whose data heat is greater than the heat threshold in the first sub-storage area; store the metadata snapshot corresponding to the target storage segment whose data heat is not greater than the heat threshold in the second sub-storage area.

[0097] According to one or more embodiments of the present disclosure, the generation module 32 is specifically used to: obtain the storage priority of the first metadata persistent storage area and the second metadata persistent storage area; determine the ratio value of the single first metadata snapshot and the second metadata snapshot according to the storage priority, wherein the first metadata snapshot is a metadata snapshot generated based on the target storage segment in the first metadata persistent storage area; the second metadata snapshot is a metadata snapshot generated based on the target storage segment in the second metadata persistent storage area; through a pre-configured task queue, execute metadata relocation requests based on the ratio value respectively, and generate a corresponding number of first metadata snapshots and second metadata snapshots within a unit time.

[0098] According to one or more embodiments of the present disclosure, the generation module 32 is further used to: obtain a comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area; and adjust the storage priority of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0099] According to one or more embodiments of the present disclosure, in response to a garbage collection instruction, the metadata persistent storage area in the storage device based on the partition namespace is detected, and before the target storage segment is determined, the detection module 31 is also used to: obtain a comprehensive load value of the metadata persistent storage area; determine a corresponding trigger recovery load interval based on the comprehensive load value, wherein the comprehensive load value is inversely proportional to the median of the trigger recovery load interval; and generate a garbage collection instruction based on the trigger recovery load interval and the storage load of each storage segment in the metadata persistent storage area.

[0100] According to one or more embodiments of the present disclosure, a storage device based on a partition namespace includes at least one stand-alone storage engine file, and a generation module 32 is further used to obtain validity information of the stand-alone storage engine file corresponding to the target storage segment; when the generation module 32 generates a corresponding metadata snapshot for the target storage segment, it is specifically used to generate a metadata snapshot corresponding to the stand-alone storage engine file according to the validity information of the stand-alone storage engine file.

[0101] The detection module 31, the generation module 32 and the cleaning module 33 are connected in sequence. The metadata garbage collection device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0102] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Fig.11 As shown, the electronic device 4 includes:

[0103] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0104] The memory 42 stores computer executable instructions;

[0105] The processor 41 executes the computer execution instructions stored in the memory 42 to implement the following Figure 2-Figure 9 The metadata garbage collection method in the illustrated embodiment.

[0106] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0107] For related instructions, please refer to Figure 2-Figure 9 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.

[0108] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement the present invention when executed by a processor. Figure 2-Figure 9 The metadata garbage collection method provided by any one of the corresponding embodiments.

[0109] The present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the present invention is realized. Figure 2-Figure 9 The metadata garbage collection method provided by any one of the corresponding embodiments.

[0110] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0111] refer to Fig.12 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0112] like Fig.12 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0113] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0114] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0115] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0116] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0117] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0118] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit or module does not, in some cases, constitute a limitation on the unit itself.

[0121] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0122] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] In a first aspect, according to one or more embodiments of the present disclosure, a metadata garbage collection method is provided, comprising:

[0124] In response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace is detected, and a target storage segment is determined, wherein the metadata persistent storage area includes a storage segment for storing metadata based on a sequential write method, and the target storage segment is a storage segment to be garbage collected; for the target storage segment, a corresponding metadata snapshot is generated, and the metadata snapshot is stored in the metadata persistent storage area, and the metadata snapshot is used to achieve data recovery of business data corresponding to the metadata by recording the latest metadata generated in the target storage segment; and the data in the target storage segment is cleared.

[0125] According to one or more embodiments of the present disclosure, the metadata persistent storage area includes a first metadata persistent storage area and a second metadata persistent storage area, wherein the first metadata persistent storage area is used to store historical operation records for business data, and the second metadata persistent storage area is used to store metadata snapshots for metadata; the responding to a garbage collection instruction, detecting the metadata persistent storage area in a storage device based on a partition namespace, and determining a target storage segment includes: responding to a first garbage collection instruction, detecting the first metadata persistent storage area, determining the target storage segment in the first metadata persistent storage area, and / or, responding to a second garbage collection instruction, detecting the second metadata persistent storage area, and determining the target storage segment in the second metadata persistent storage area.

[0126] According to one or more embodiments of the present disclosure, the detecting the first metadata persistent storage area in response to a first garbage collection instruction and determining a target storage segment in the first metadata persistent storage area includes: obtaining data write timestamps corresponding to each storage segment in the first metadata persistent storage area in response to the first garbage collection instruction, and determining the storage segment corresponding to the earliest write timestamp as the target storage segment; the detecting the second metadata persistent storage area in response to a second garbage collection instruction and determining the target storage segment in the second metadata persistent storage area includes: obtaining storage loads corresponding to each storage segment in the second metadata persistent storage area in response to the second garbage collection instruction, and determining at least one storage segment with the largest storage load as the target storage segment, wherein the storage load is used to characterize the proportion of invalid data in a storage segment.

[0127] According to one or more embodiments of the present disclosure, determining at least one storage segment with the largest storage load as the target storage segment includes: obtaining a comprehensive load value of the second metadata persistent storage area, and determining a target number based on the comprehensive load value and preset load mapping information, wherein the preset load mapping information represents a mapping relationship between the comprehensive load value and the number of storage segments; determining the target number of storage segments with the largest storage load in the second metadata persistent storage area as the target storage segments.

[0128] According to one or more embodiments of the present disclosure, generating a corresponding metadata snapshot for the target storage segment and storing the metadata snapshot in the metadata persistent storage area includes: generating a first metadata snapshot for the target storage segment in the first metadata persistent storage area and storing the first metadata snapshot in the second metadata persistent storage area, and / or generating a second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area.

[0129] According to one or more embodiments of the present disclosure, the target storage segment includes at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; and generating a second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area includes:

[0130] Generate a metadata snapshot of each of the target storage segments in the second metadata persistent storage area, and obtain the data heat corresponding to each of the target storage segments, wherein the data heat represents the update probability of the metadata corresponding to the target storage segment; store the metadata snapshot corresponding to the target storage segment whose data heat is greater than a heat threshold in the first sub-storage area; store the metadata snapshot corresponding to the target storage segment whose data heat is not greater than the heat threshold in the second sub-storage area.

[0131] According to one or more embodiments of the present disclosure, generating a corresponding metadata snapshot for the target storage segment includes: obtaining storage priorities of the first metadata persistent storage area and the second metadata persistent storage area; determining a ratio value of a single first metadata snapshot and a second metadata snapshot according to the storage priorities, wherein the first metadata snapshot is a metadata snapshot generated based on the target storage segment in the first metadata persistent storage area; the second metadata snapshot is a metadata snapshot generated based on the target storage segment in the second metadata persistent storage area; and executing metadata relocation requests based on the ratio value respectively through a pre-configured task queue to generate a corresponding number of first metadata snapshots and second metadata snapshots within a unit time.

[0132] According to one or more embodiments of the present disclosure, the method also includes: obtaining a comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area; and adjusting the storage priority of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0133] According to one or more embodiments of the present disclosure, before detecting the metadata persistent storage area in the storage device based on the partition namespace and determining the target storage segment in response to the garbage collection instruction, the method also includes: obtaining a comprehensive load value of the metadata persistent storage area; determining a corresponding trigger recovery load interval based on the comprehensive load value, wherein the comprehensive load value is inversely proportional to the median of the trigger recovery load interval; and generating the garbage collection instruction based on the trigger recovery load interval and the storage load of each storage segment in the metadata persistent storage area.

[0134] According to one or more embodiments of the present disclosure, the storage device based on the partition namespace includes at least one stand-alone storage engine file, and the stand-alone storage engine file is used to store business data corresponding to the metadata; the method also includes: obtaining validity information of the stand-alone storage engine file corresponding to the target storage segment; generating a corresponding metadata snapshot for the target storage segment, including: based on the validity information of the stand-alone storage engine file, if the stand-alone storage engine file is in a validity state, generating a corresponding metadata snapshot for the target storage segment.

[0135] In a second aspect, according to one or more embodiments of the present disclosure, a metadata garbage collection device is provided, comprising:

[0136] A detection module, configured to detect, in response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace, and determine a target storage segment, wherein the metadata persistent storage area includes a storage segment storing metadata based on a sequential write method, the target storage segment is a storage segment to be garbage collected, and the metadata is used to realize data recovery of corresponding business data;

[0137] A generation module, used for generating a corresponding metadata snapshot for the target storage segment, and storing the metadata snapshot in the metadata persistent storage area, wherein the metadata snapshot is used for recording the latest metadata generated in the target storage segment;

[0138] The cleaning module is used to clear the data in the target storage segment.

[0139] According to one or more embodiments of the present disclosure, the metadata persistent storage area includes a first metadata persistent storage area and a second metadata persistent storage area, wherein the first metadata persistent storage area is used to store historical operation records for business data, and the second metadata persistent storage area is used to store metadata snapshots for metadata; the detection module is specifically used to: respond to a first garbage collection instruction, detect the first metadata persistent storage area, determine a target storage segment in the first metadata persistent storage area, and / or, respond to a second garbage collection instruction, detect the second metadata persistent storage area, and determine a target storage segment in the second metadata persistent storage area.

[0140] According to one or more embodiments of the present disclosure, when the detection module detects the first metadata persistent storage area in response to a first garbage collection instruction and determines the target storage segment in the first metadata persistent storage area, the detection module is specifically used to: in response to the first garbage collection instruction, obtain the data write timestamps corresponding to each storage segment in the first metadata persistent storage area, and determine the storage segment corresponding to the earliest write timestamp as the target storage segment; when the detection module detects the second metadata persistent storage area in response to a second garbage collection instruction and determines the target storage segment in the second metadata persistent storage area, the detection module is specifically used to: in response to the second garbage collection instruction, obtain the storage loads corresponding to each storage segment in the second metadata persistent storage area, and determine at least one storage segment with the largest storage load as the target storage segment, wherein the storage load is used to characterize the proportion of invalid data in the storage segment.

[0141] According to one or more embodiments of the present disclosure, when the detection module determines at least one storage segment with the largest storage load as the target storage segment, it is specifically used to: obtain a comprehensive load value of the second metadata persistent storage area, and determine a target number based on the comprehensive load value and preset load mapping information, wherein the preset load mapping information represents a mapping relationship between the comprehensive load value and the number of storage segments; and determine the target number of storage segments with the largest storage load in the second metadata persistent storage area as the target storage segments.

[0142] According to one or more embodiments of the present disclosure, the generation module is specifically used to: generate a first metadata snapshot for a target storage segment in the first metadata persistent storage area, and store the first metadata snapshot in the second metadata persistent storage area, and / or; generate a second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area.

[0143] According to one or more embodiments of the present disclosure, the target storage segments include at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; when the generation module generates a second metadata snapshot in the second metadata persistent storage area for the target storage segments in the second metadata persistent storage area, the generation module is specifically used to: generate metadata snapshots of each of the target storage segments in the second metadata persistent storage area, and obtain data heat corresponding to each of the target storage segments, wherein the data heat represents the update probability of the metadata corresponding to the target storage segment; store the metadata snapshot corresponding to the target storage segment whose data heat is greater than a heat threshold in the first sub-storage area; store the metadata snapshot corresponding to the target storage segment whose data heat is not greater than the heat threshold in the second sub-storage area.

[0144] According to one or more embodiments of the present disclosure, the generation module is specifically used to: obtain the storage priorities of the first metadata persistent storage area and the second metadata persistent storage area; determine the ratio value of a single first metadata snapshot and a second metadata snapshot according to the storage priorities, wherein the first metadata snapshot is a metadata snapshot generated based on a target storage segment in the first metadata persistent storage area; the second metadata snapshot is a metadata snapshot generated based on a target storage segment in the second metadata persistent storage area; and execute metadata relocation requests based on the ratio value respectively through a pre-configured task queue to generate a corresponding number of first metadata snapshots and second metadata snapshots within a unit time.

[0145] According to one or more embodiments of the present disclosure, the generation module is further used to: obtain the comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area; and adjust the storage priority of the first metadata persistent storage area and the second metadata persistent storage area according to the comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area.

[0146] According to one or more embodiments of the present disclosure, before the detection module detects the metadata persistent storage area in the storage device based on the partition namespace and determines the target storage segment in response to the garbage collection instruction, the detection module is also used to: obtain the comprehensive load value of the metadata persistent storage area; determine the corresponding trigger recovery load interval based on the comprehensive load value, wherein the comprehensive load value is inversely proportional to the median of the trigger recovery load interval; and generate the garbage collection instruction based on the trigger recovery load interval and the storage load of each storage segment in the metadata persistent storage area.

[0147] According to one or more embodiments of the present disclosure, the storage device based on the partition namespace includes at least one stand-alone storage engine file, and the generation module is further used to: obtain validity information of the stand-alone storage engine file corresponding to the target storage segment; when the generation module generates a corresponding metadata snapshot for the target storage segment, it is specifically used to: generate a metadata snapshot corresponding to the stand-alone storage engine file according to the validity information of the stand-alone storage engine file.

[0148] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0149] The memory stores computer-executable instructions;

[0150] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the metadata garbage collection method described in the first aspect and various possible designs of the first aspect.

[0151] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the metadata garbage collection method described in the first aspect and various possible designs of the first aspect is implemented.

[0152] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the metadata garbage collection method as described in the first aspect and various possible designs of the first aspect.

[0153] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0154] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0155] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A metadata garbage collection method, characterized in that: include: In response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace is detected to determine a target storage segment, wherein the metadata persistent storage area includes a storage segment for storing metadata based on a sequential write method, and the target storage segment is a storage segment to be garbage collected; Generate a corresponding metadata snapshot for the target storage segment, and store the metadata snapshot in the metadata persistent storage area, wherein the metadata snapshot is used to achieve data recovery of the business data corresponding to the metadata by recording the latest metadata generated in the target storage segment; Clear the data in the target storage segment.

2. The method according to claim 1, characterized in that The metadata persistent storage area includes a first metadata persistent storage area and a second metadata persistent storage area, wherein the first metadata persistent storage area is used to store historical operation records for business data, and the second metadata persistent storage area is used to store metadata snapshots for metadata; The method of detecting, in response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace and determining a target storage segment comprises: In response to a first garbage collection instruction, the first metadata persistent storage area is detected and a target storage segment in the first metadata persistent storage area is determined, or, in response to a second garbage collection instruction, the second metadata persistent storage area is detected and a target storage segment in the second metadata persistent storage area is determined.

3. The method according to claim 2, characterized in that The step of detecting the first metadata persistent storage area in response to the first garbage collection instruction and determining a target storage segment in the first metadata persistent storage area includes: In response to the first garbage collection instruction, obtaining data write timestamps corresponding to each storage segment in the first metadata persistent storage area, and determining the storage segment corresponding to the earliest write timestamp as the target storage segment; The step of detecting the second metadata persistent storage area in response to the second garbage collection instruction and determining a target storage segment in the second metadata persistent storage area includes: In response to the second garbage collection instruction, the storage load corresponding to each storage segment in the second metadata persistent storage area is obtained, and at least one storage segment with the largest storage load is determined as the target storage segment, wherein the storage load is used to characterize the proportion of invalid data in the storage segment.

4. The method according to claim 3, characterized in that The step of determining at least one storage segment with the largest storage load as the target storage segment includes: Obtaining a comprehensive load value of the second metadata persistent storage area, and determining a target quantity according to the comprehensive load value; The target number of storage segments with the largest storage load in the second metadata persistent storage area are determined as the target storage segments.

5. The method according to claim 2, characterized in that: The step of generating a corresponding metadata snapshot for the target storage segment and storing the metadata snapshot in the metadata persistent storage area includes: generating a first metadata snapshot for the target storage segment in the first metadata persistent storage area, and storing the first metadata snapshot in the second metadata persistent storage area, and / or; For the target storage segment in the second metadata persistent storage area, a second metadata snapshot is generated in the second metadata persistent storage area.

6. The method according to claim 5, characterized in that The target storage segment includes at least two, and the second metadata persistent storage area includes a first sub-storage area and a second sub-storage area; The step of generating a second metadata snapshot in the second metadata persistent storage area for the target storage segment in the second metadata persistent storage area includes: Generate a metadata snapshot of each of the target storage segments in the second metadata persistent storage area, and obtain data heat corresponding to each of the target storage segments, where the data heat represents an update probability of the metadata corresponding to the target storage segment; The target storage segment whose data heat is greater than the heat threshold corresponds to the metadata snapshot and is stored in the first sub-storage area; The target storage segment whose data heat is not greater than the heat threshold corresponds to the metadata snapshot and is stored in the second sub-storage area.

7. The method according to claim 2, characterized in that The step of generating a corresponding metadata snapshot for the target storage segment includes: Obtaining storage priorities of the first metadata persistent storage area and the second metadata persistent storage area; Determine, according to the storage priority, a ratio value of a single first metadata snapshot and a second metadata snapshot, wherein the first metadata snapshot is a metadata snapshot generated based on a target storage segment in the first metadata persistent storage area; and the second metadata snapshot is a metadata snapshot generated based on a target storage segment in the second metadata persistent storage area; Through the pre-configured task queue, metadata relocation requests are respectively executed based on the ratio value, and a corresponding number of first metadata snapshots and second metadata snapshots are generated within a unit time.

8. The method according to claim 7, characterized in that The method further comprises: Obtaining a comprehensive load value corresponding to the first metadata persistent storage area and the second metadata persistent storage area; According to the comprehensive load values ​​corresponding to the first metadata persistent storage area and the second metadata persistent storage area, the storage priorities of the first metadata persistent storage area and the second metadata persistent storage area are adjusted.

9. The method according to claim 1, characterized in that: Before detecting the metadata persistent storage area in the storage device based on the partition namespace in response to the garbage collection instruction and determining the target storage segment, the method further includes: Obtaining a comprehensive load value of the metadata persistent storage area; Determine a corresponding trigger recovery load interval according to the comprehensive load value, wherein the comprehensive load value is inversely proportional to the median of the trigger recovery load interval; When the storage load of at least one storage segment in the metadata persistent storage area is located in the triggering collection load interval, the garbage collection instruction is generated.

10. The method according to claim 1, characterized in that The partition namespace-based storage device includes at least one stand-alone storage engine file, and the stand-alone storage engine file is used to store business data corresponding to the metadata; The method further comprises: Acquire validity information of a stand-alone storage engine file corresponding to the target storage segment, where the validity information is used to indicate the validity of the stand-alone storage engine file; The step of generating a corresponding metadata snapshot for the target storage segment includes: According to the validity information of the stand-alone storage engine file, if the stand-alone storage engine file is in a validity state, a corresponding metadata snapshot is generated for the target storage segment.

11. A metadata garbage collection device, characterized in that: include: A detection module, configured to detect, in response to a garbage collection instruction, a metadata persistent storage area in a storage device based on a partition namespace, and determine a target storage segment, wherein the metadata persistent storage area includes a storage segment storing metadata based on a sequential write method, the target storage segment is a storage segment to be garbage collected, and the metadata is used to realize data recovery of corresponding business data; A generation module, used for generating a corresponding metadata snapshot for the target storage segment, and storing the metadata snapshot in the metadata persistent storage area, wherein the metadata snapshot is used for recording the latest metadata generated in the target storage segment; The cleaning module is used to clear the data in the target storage segment.

12. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the metadata garbage collection method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the metadata garbage collection method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the metadata garbage collection method according to any one of claims 1 to 10 is implemented.

Citation Information

Patent Citations

  • Memory data persistence method and device, storage medium and computer equipment

    CN112685360A

  • Data persistence method and device for terminal application

    CN113741796A

  • Garbage collection in data storage system

    CN114556310A

  • Metadata snapshot rollback method, system and device and storage medium

    CN117093542A

  • Data storage method and equipment

    CN117724996A

Cited By

  • Data writing method, recovery method, and reading method, and corresponding apparatus

    US12619790B2