Method and device for managing hot data cache by adding write in distributed storage system and medium

Abstract

The invention relates to the technical field of solid-state storage, and provides a method and device for managing hot data cache in a distributed storage system and a storage medium. The core of the method is that a Bloom filter is utilized to count popularity by taking an object as a unit, and hot data objects are distinguished. When the data is written, the hot data object is additionally written into the NVMe disk cache after being written into the HDD, so that sequential writing is realized to improve the performance. And during reading, if the cache is hit, preferentially reading the cache. And when the business object ID is deleted or the hot data is overwritten, checking and deleting the invalid hot data object to recycle the space. And identifying and deleting the corresponding hotspot data during the main and standby switching of the collating group. When the OSD is powered on, the hot data cache is defaulted to be invalid, the hot data is cleared up in the power-on process, and the hot data cache function is recovered after the hot data cache function is cleared up. According to the method, the overall performance of the storage system is remarkably improved, and efficient operation of the storage system is ensured.

Description

Technical Field

[0001] The present application relates to the field of solid-state storage technology, and more particularly to a method, device and medium for managing hot data cache by appending writes in a distributed storage system. Background Art

[0002] At present, with the rapid development of information technology, storage technology is also constantly improving. Solid-state storage devices, such as NVMe (Non-Volatile Memory Express) and SSD (Solid State Drive), as a new generation of storage devices, have gradually entered and changed the storage market with their high-speed read and write performance and low latency characteristics. However, due to the high cost, solid-state storage devices cannot completely replace traditional mechanical storage devices (HDD, Hard Disk Drive). Therefore, in practical applications, solid-state storage devices are often used as small-capacity cache devices in combination with traditional HDD devices to form a hybrid storage architecture.

[0003] In the current storage server market, storage servers can be roughly divided into three categories according to the type of storage devices: all-flash servers (all using high-speed devices such as SSD, NVMe, etc.), hybrid flash servers (mixed with high-speed devices such as SSD or NVMe and low-speed devices such as HDD), and all-HDD servers (all using HDD devices). Among them, hybrid flash servers have been widely used in distributed storage systems because of their advantages in both performance and cost.

[0004] In the distributed storage system of hybrid flash servers, in order to achieve a balance between cost-effectiveness and system security and reliability, a "data tiered storage" strategy is usually adopted. The core of this strategy is to use high-speed devices (such as SSD, NVMe) to store hot data that users access frequently to reduce user reading latency; and use low-performance HDD devices to store data that users access less frequently to fully utilize the advantages of HDD disks with large capacity and low price.

[0005] For the flash device management of hybrid flash servers, mainstream manufacturers currently use three main solutions. The first solution is to use a dual storage pool solution, that is, use flash disks to create a high-speed cache pool and HDD disks to create a slow pool, which are managed by different object storage resource processes. However, this solution increases the length of the IO path and reduces storage performance. The second solution is not to divide the storage pool separately, but to divide the partitions on the NVMe disk and create an XFS file system to manage hot data objects in the form of files. However, this method will also increase the length of the IO path, and the stability of the XFS file system faces challenges. The third solution does not require dual storage pools and XFS file systems, and the NVMe partitions are matched with the HDD disks on the storage nodes, and are managed by the same object storage resource process. This method can greatly improve the read performance, but because the data writing is random, when the cached data volume is large enough, the NVMe disk will trigger the garbage collection (GC) mechanism because it is written very scattered, resulting in performance degradation.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The embodiments of the present disclosure provide a method, device and medium for managing hot data cache by appending writes in a distributed storage system. By appending writes to cached data, sequential writes to NVMe disks are implemented, thereby reducing the impact of NVMe disk GC on storage system performance, improving the storage performance and stability of hybrid flash servers, and meeting the growing storage needs.

[0009] In some embodiments, the method comprises:

[0010] Take the object as the unit and use Bloom filter to realize heat statistics to determine whether the object is a hot data object;

[0011] When writing data, if the object is determined to be a hot data object, the data is written to the NVMe disk cache after the IO is written to the HDD disk; when reading the object, if the cache hits, the cache is read first;

[0012] When a business object id is deleted or hot data is overwritten, determine whether the data segments of all business object ids on the hot data object have expired. If they have, delete the hot data object to reclaim the hot data space.

[0013] When a primary-backup switch occurs in a placement group, the hot data objects corresponding to the placement group where the primary-backup switch occurs are identified, added to the background hot data deletion queue, and the hot data is deleted; when the OSD is powered on, all previously retained hot data caches are invalid by default, and the hot data is cleaned up during the power-on process. After all hot data is cleaned up, the hot data cache function of the placement group takes effect.

[0014] Preferably, whether the object is a hot data object is determined in the following manner:

[0015] A hit_set_map is maintained according to the set time period to record the objects accessed in the current period and retain the statistical information of the last five periods. Whether the object is a hot data object is determined based on the historical heat record hits. If the currently read object is hit three times within the five historical heat record periods, it is considered hot data.

[0016] Preferably, the specific method of writing data is as follows:

[0017] Apply for a hot data object based on the hot data that needs to be written, apply for storage space on the NVMe disk, write the data, update the object metadata, and record the hot data offset, length, hot data object, and actual storage location mapping relationship.

[0018] Preferably, the specific method of reading the object is as follows:

[0019] Determine whether the read data fully or partially hits the hot data cache. If all hits, read the data from the hot data object according to the forward relationship and return the data to the client; if part of the cache hits, read the hit part according to the forward relationship, read the missed data from the HDD, and return the completed data to the client after data splicing.

[0020] Preferably, when the cache space is insufficient, the colder hot data is eliminated according to the LRU mechanism to release the cache space, and the elimination speed is controlled according to the gradient of the used capacity ratio of the cache space;

[0021] Preferably, the criterion for judging whether the cache space is insufficient is as follows: if the cache space utilization rate is greater than or equal to 80%, the cache space is insufficient; if the cache space utilization rate is less than 80%, the cache space is sufficient.

[0022] Preferably, the gradient-controlled elimination speed is controlled according to the used capacity of the cache space. When the cache space is greater than or equal to 80% but less than 90%, the hotspot object deletion speed is 5 / s; when the cache space is greater than or equal to 90% but less than 95%, the hotspot object deletion speed is 10 / s; when the cache space is greater than or equal to 95, the hotspot object deletion speed is 20 / s.

[0023] Preferably, hot data space recovery also includes a space integration recovery mechanism for hotspot objects, as follows:

[0024] Scan hot data objects forward, add hot data objects whose effective space is lower than the set threshold to the garbage collection queue, reaggregate the data in the queue into storage space equal to the storage space requested on the NVMe disk, write new hot data cache objects, update the mapping relationship of business objects, and delete the original hot data objects to release hot data space.

[0025] In some embodiments, the device includes a processor and a memory storing program instructions, and the processor is configured to execute the new grid-connected new energy transient reactive support strategy and setting method when running the program instructions.

[0026] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the novel grid-building type renewable energy transient reactive support strategy and setting method.

[0027] The method, device and medium for managing hot data cache by appending write in a distributed storage system provided by the embodiments of the present disclosure can achieve the following technical effects:

[0028] Beneficial Effects

[0029] The present invention implements sequential writing to NVMe disks by implementing additional writing to hot data cached by NVMe disks. This method effectively reduces the large amount of GC (Garbage Collection) data movement generated by small random writes to NVMe disks, thereby significantly improving the performance of the storage system.

[0030] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0032] Figure 1 It is a schematic flow chart of the method of the present invention;

[0033] Figure 2 is a schematic diagram of a hotspot data identification strategy provided by an embodiment of the present disclosure;

[0034] Figure 3 It is a schematic diagram of a method for appending hotspot data to an NVMe disk provided by an embodiment of the present disclosure;

[0035] Figure 4 It is a schematic diagram of data flow of the present invention;

[0036] Figure 5 It is a schematic diagram of a device provided by an embodiment of the present disclosure.

[0037] In the figure: 300: device for air conditioner; 301: memory; 302: communication interface; 303: bus; 304: processor. DETAILED DESCRIPTION

[0038] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0039] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0040] Unless otherwise stated, the term "plurality" means two or more.

[0041] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0042] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0043] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0044] In the disclosed embodiments, the terminal device refers to an electronic device with a wireless connection function. The terminal device can communicate with the above-mentioned smart home appliances by connecting to the Internet, or can communicate with the above-mentioned smart home appliances directly through Bluetooth, WiFi, etc. In some embodiments, the terminal device is, for example, a mobile device, a computer, or a vehicle-mounted device built into a hover car, or any combination thereof. Mobile devices may include, for example, mobile phones, smart home devices, wearable devices, smart mobile devices, virtual reality devices, etc., or any combination thereof, wherein wearable devices include, for example: smart watches, smart bracelets, pedometers, etc.

[0045] Example 1

[0046] Combination Figure 1-Figure 4 , a method, device and medium for managing hot data cache by appending writes in a distributed storage system, mainly by appending writes to the hot data cached in the NVMe disk to achieve sequential writes to the NVMe disk, thereby reducing the large amount of GC data movement caused by small block random writes on the NVMe disk and improving the performance of the storage system. This solution is applicable to distributed storage systems using hybrid flash servers, and is mainly divided into four modules: hot data identification, hot data management, hot data space recovery and fault handling. After the hot data is written to the HDD disk, it is written to the NVMe Cache in the form of an append write. When the front-end business reads the data, it will first determine whether it hits the NVMe cache. If it hits, it can be read directly from the cache disk to improve performance. In some embodiments, the method includes:

[0047] S1: Hot data identification: Taking objects as units, heat statistics are implemented through Bloom filters to determine whether the object is a hot data object.

[0048] S2: Hot data management; when writing data, if the object is determined to be a hot data object, the data is written to the NVMe disk cache after the IO is written to the HDD disk; when reading the object, if the cache hits, the cache is read first.

[0049] S3: Reclaiming hot data space; when a business object id is deleted or hot data is overwritten, determine whether the data segments of all business object ids on the hot data object are invalid. If they are invalid, delete the hot data object to reclaim the hot data space.

[0050] S4: Fault handling; when a primary-slave switch occurs in a placement group, the hot data object corresponding to the placement group where the primary-slave switch occurs is identified, added to the background hot data deletion queue, and the hot data is deleted; when the OSD is powered on, all previously retained hot data caches are invalid by default, and the hot data is cleaned up during the power-on process. After all hot data is cleaned up, the hot data cache function of the placement group takes effect.

[0051] As a refinement of the above embodiment, whether the object is a hot data object is determined in the following specific manner:

[0052] A hit_set_map is maintained according to the set time period to record the objects accessed in the current period and retain the statistical information of the last five periods. Whether the object is a hot data object is determined based on the historical heat record hits. If the currently read object is hit three times within the five historical heat record periods, it is considered hot data.

[0053] Specifically, the set time period can be configured as 1min, and each period maintains a hit_set_map to record the objects accessed in the current period. After the current period ends, a hit_set_map corresponding to the new period is created to record the objects accessed. And the number of cycles retained can be configured according to actual conditions. In this embodiment, if the currently read object is hit 3 times within the 5 historical heat recording periods, it is considered to be hot data. The object reference is input, and the output is whether the object is a hot data object, the result: true or false.

[0054] As a refinement of the above embodiment, the specific method of writing data is as follows:

[0055] Apply for a hot data object based on the hot data that needs to be written, apply for storage space on the NVMe disk, write the data, update the object metadata, and record the hot data offset, length, hot data object, and actual storage location mapping relationship.

[0056] Specifically, considering the limited cache space, hot data is saved in the form of a single copy, and only the main PG processes the corresponding hot data reading and writing. When oid_1 needs to write hot data with a length of 8k, it will first apply for a hot_oid (hot data object), and apply for a 4M storage space for hot_oid1 from the NVMe disk to write the data to the hot data with offset_len of [0,8k]. After the writing is completed, the metadata of oid_1 is updated to record the offset, length, hot_oid1 of the hot data, and the actual storage location on hot_oid1, that is, the forward mapping relationship from the actual oid to the hot_oid. No space will be applied for subsequent hot data writing until the hot data written exceeds 4M, then hot_oid2 will be generated and 4M space will continue to be applied for from the NVMe disk. The 4M space applied for by different hot_oids is continuous on the physical disk, which ensures that each hot data is written continuously, and the additional writing to the NVMe disk is realized.

[0057] As a refinement of the above embodiment, the specific method of reading the object is as follows:

[0058] Determine whether the read data fully or partially hits the hot data cache. If all hits, read the data from the hot data object according to the forward relationship and return the data to the client; if part of the cache hits, read the hit part according to the forward relationship, read the missed data from the HDD, and return the completed data to the client after data splicing.

[0059] As a refinement of the above embodiment, the OID of the hot data object is managed by LRU.

[0060] When the cache space is insufficient, the LRU mechanism is used to eliminate the colder hot data to release the cache space, and the elimination speed is controlled according to the gradient of the used capacity ratio of the cache space;

[0061] Specifically, in this embodiment, the judgment criteria for insufficient cache space are as follows: if the cache space utilization rate is greater than or equal to 80%, the cache space is insufficient; if the cache space utilization rate is less than 80%, the cache space is sufficient.

[0062] The gradient control elimination speed is controlled according to the used capacity of the cache space. When the cache space is greater than or equal to 80% but less than 90%, the hotspot object deletion speed is 5 / s; when the cache space is greater than or equal to 90% but less than 95%, the hotspot object deletion speed is 10 / s; when the cache space is greater than or equal to 95, the hotspot object deletion speed is 20 / s.

[0063] As a refinement of the above embodiment, hot data caching is implemented through additional writing, which means that a hot_oid will actually contain multiple business oids. When the business oid is deleted or hot data is overwritten (hot data overwriting also means that the previous hot data cache is invalid, which is actually equivalent to data deletion), because hot_oid may still have hot data references of other objects, hot_oid cannot be deleted. Only when the data segments of all business oids on hot_oid are invalid, can hot_oid be deleted and the space be reclaimed. These reclaimed NVMe disk spaces can be used by new hot data objects when subsequent hot objects are appended. Hot data space recovery also requires a space integration recovery mechanism for hot objects.

[0064] Specifically, hot data space recovery also includes a space consolidation and recovery mechanism for hotspot objects, as follows:

[0065] Scan hot data objects forward, add hot data objects whose effective space is lower than the set threshold to the garbage collection queue, reaggregate the data in the queue into storage space equal to the storage space requested on the NVMe disk, write new hot data cache objects, update the mapping relationship of business objects, and delete the original hot data objects to release hot data space.

[0066] Example 2

[0067] A hot data cache system managed by additional writing in a distributed storage system includes: a hot data identification module, a hot data management module, a hot data space recovery module and a fault processing module.

[0068] Hot data identification module: Hot data identification is based on objects, and heat statistics are implemented through Bloom filters. The main function is to determine whether the object is a hot data object. According to the set time period (default 1min, configurable), a hit_set_map is maintained in each cycle to record the objects accessed in the current cycle. After the current cycle ends, a hit_set_map corresponding to the new cycle is created to record the accessed objects. The heat information retains the statistics of the last 5 cycles (the number of retained cycles is configurable). The heat decision can be regarded as a black box for the business process and is determined based on the hit situation of historical heat records. If the object currently being read is hit 3 times within the historical 5 heat record cycles, it is considered to be hot data. The object reference is input and the output is whether the object is a hot data object. The result is: true or false.

[0069] Hot data management module: The main function is that when data is written, if the object is determined to be a hot data object, the data will be written to the NVMe disk cache after the IO is written to the HDD disk; when reading the object, if the cache hits, the cache is read first. Considering the limited cache space, hot data is saved in the form of a single copy, and only the main PG processes the corresponding hot data reading and writing. When oid_1 needs to write hot data with a length of 8k, it will first apply for a hot_oid (hot data object), and apply for a 4M storage space for hot_oid1 from the NVMe disk to write the data to the hot data with offset_len of [0,8k]. After the writing is completed, the metadata of oid_1 is updated to record the offset, length, hot_oid1 of the hot data, and the actual storage location on hot_oid1, that is, the forward mapping relationship between the actual oid and hot_oid. No space will be applied for subsequent hot data writing until the hot data written exceeds 4M, then hot_oid2 is generated and 4M space is continued to be applied to the NVMe disk. The 4M space requested by different hot_oids is continuous on the physical disk, which ensures that each hot data is written continuously and implements additional writing to the NVMe disk.

[0070] When reading data, it is necessary to determine whether the data read this time is fully hit or partially hit in the hot data cache. If all hits, read the data from hot_oid according to the forward relationship, and then return the data to the client; if part of the cache hits, first read the hit part according to the forward relationship, and then read the missed data from the HDD. After data splicing, return the completed data to the client.

[0071] Hot data space recovery module: Hot data cache is implemented through appending writes, which means that a hot_oid actually contains multiple business oids. When the business oid is deleted or hot data is overwritten (hot data overwriting also means that the previous hot data cache is invalid, which is actually equivalent to data deletion), because the hot_oid may still have hot data references from other objects, the hot_oid cannot be deleted. Only when the data segments of all business oids on the hot_oid are invalid, can the hot_oid be deleted and the space be reclaimed. These reclaimed NVMe disk spaces can be used by new hot data objects when subsequent hot objects are appended. Hot data space recovery also requires a space integration recovery mechanism for hot objects. First, it is necessary to forward scan the hot_oid, add the hot_oid with effective space below the threshold (such as less than 256k) to the garbage collection queue, re-aggregate the data in the queue into a size of 4M, write it to the new hot data cache object, and then update the mapping relationship of the business object. Finally, delete the original hot_oid to release the space.

[0072] The oid of hot data objects must be managed through LRU. When the cache space is insufficient (the cache space has reached 80%), it is necessary to eliminate relatively cold hot data according to the LRU mechanism to release the cache space; the cache space is released according to the percentage of the used capacity of the cache space, and the elimination speed is controlled by gradient. The gradient control elimination speed can be controlled according to the used capacity of the cache space. When the cache space is at a low water level (greater than 80%, but less than 90%), the hot object deletion speed is 5 / s; when the cache space is at a medium water level (greater than 90%, but less than 95%), the hot object deletion speed is 10 / s; when the cache space is at a high water level (greater than 95), the hot object deletion speed is 20 / s.

[0073] Fault handling module: When the PG switches between active and standby modes, the hot_oid corresponding to the PG where the switch occurs is identified, and the corresponding hot_oid is added to the background hot data deletion queue to delete the hot data. When the OSD is powered on, the previously retained hot data cache is invalid by default, and the hot data is cleaned up during the power-on process. After all the hot data of the corresponding PG is cleaned up, the hot data cache function of the PG can take effect.

[0074] Glossary

[0075] OSD (Object Storage Device): object storage resources.

[0076] Object: Object is the smallest unit of data storage, which is distributed in each PG. A PG stores a large number of objects.

[0077] Oid: Object id, object id.

[0078] PG (Placement Group): Placement group, used to place the carrier of the object.

[0079] Cache Pool: Cache pool, composed of solid-state storage devices such as NVMe.

[0080] StoragePool: Data storage pool, composed of low-speed storage devices such as HDD disks.

[0081] GC (Garbage Collection): Garbage collection.

[0082] Example 3

[0083] Combination Figure 5 As shown, an embodiment of the present disclosure provides a device 300 for managing hot data cache by appending writes in a distributed storage system, including a processor (processor) 304 and a memory (memory) 301. Optionally, the device may also include a communication interface (Communication Interface) 302 and a bus 303. Among them, the processor 304, the communication interface 302, and the memory 301 can communicate with each other through the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can call the logical instructions in the memory 301 to execute the method for managing hot data cache by appending writes in the distributed storage system of the above embodiment.

[0084] In addition, the logic instructions in the memory 301 described above can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0085] The memory 301 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 304 executes functional applications and data processing by running the program instructions / modules stored in the memory 301, that is, the method for managing hot data cache by appending writes in the distributed storage system in the above embodiment is implemented.

[0086] The memory 301 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 301 may include a high-speed random access memory and may also include a non-volatile memory.

[0087] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the method for managing hot data cache by appending writes in the above-mentioned distributed storage system.

[0088] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0089] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0090] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations listed in association with one or more. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0091] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0092] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0093] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for managing hot data cache by appending writes in a distributed storage system, characterized in that: The following steps are involved: Take the object as the unit and use Bloom filter to realize heat statistics to determine whether the object is a hot data object; When writing data, if the object is determined to be a hot data object, the data is written to the NVMe disk cache after the IO is written to the HDD disk; When reading an object, if the cache hits, the cache is read first; When a business object id is deleted or hot data is overwritten, determine whether the data segments of all business object ids on the hot data object have expired. If they have, delete the hot data object to reclaim the hot data space. When a primary-backup switch occurs in a placement group, the hot data objects corresponding to the placement group where the primary-backup switch occurs are identified, added to the background hot data deletion queue, and the hot data is deleted; when the OSD is powered on, all previously retained hot data caches are invalid by default, and the hot data is cleaned up during the power-on process. After all hot data is cleaned up, the hot data cache function of the placement group takes effect.

2. The method for managing hot data cache by appending writes in a distributed storage system according to claim 1, characterized in that: Determine whether the object is a hot data object. The specific method is as follows: A hit_set_map is maintained according to the set time period to record the objects accessed in the current period and retain the statistical information of the last five periods. Whether the object is a hot data object is determined based on the historical heat record hits. If the currently read object is hit three times within the five historical heat record periods, it is considered hot data.

3. The method for managing hot data cache by appending writes in a distributed storage system according to claim 1, characterized in that: The specific method of writing data is as follows: Apply for a hot data object based on the hot data that needs to be written, apply for storage space on the NVMe disk, write the data, update the object metadata, and record the hot data offset, length, hot data object, and actual storage location mapping relationship.

4. The method for managing hot data cache by appending writes in a distributed storage system according to claim 3, characterized in that: The specific method of reading objects is as follows: Determine whether all or part of the read data hits the hot data cache. If all hits, read the data from the hot data object according to the forward relationship and return the data to the client. If part of the data hits the cache, the hit part is read according to the forward relationship, and the missed data is read from the HDD. After the data is spliced, the completed data is returned to the client.

5. The method for managing hot data cache by appending writes in a distributed storage system according to claim 1, characterized in that: When the cache space is insufficient, the LRU mechanism is used to eliminate the colder hot data to release the cache space, and the elimination speed is controlled according to the gradient of the cache space used capacity ratio.

6. The method for managing hot data cache by appending writes in a distributed storage system according to claim 5, characterized in that: The criteria for judging insufficient cache space are as follows: if the cache space utilization is greater than or equal to 80%, the cache space is insufficient; if the cache space is less than 80%, the cache space is sufficient.

7. The method for managing hot data cache by appending writes in a distributed storage system according to claim 6, characterized in that: The gradient control elimination speed is controlled according to the used capacity of the cache space. When the cache space is greater than or equal to 80% but less than 90%, the hotspot object deletion speed is 5 / s; when the cache space is greater than or equal to 90% but less than 95%, the hotspot object deletion speed is 10 / s; When the cache space is greater than or equal to 95, the hotspot object deletion speed is 20 / s.

8. The method for managing hot data cache by appending writes in a distributed storage system according to claim 3, characterized in that: Hot data space reclamation also includes the space consolidation and reclamation mechanism for hotspot objects, as follows: Scan hot data objects forward, add hot data objects whose effective space is lower than the set threshold to the garbage collection queue, reaggregate the data in the queue into storage space equal to the storage space requested on the NVMe disk, write new hot data cache objects, update the mapping relationship of business objects, and delete the original hot data objects to release hot data space.

9. A novel grid-building renewable energy transient reactive support strategy and setting device, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the new grid-connected new energy transient reactive support strategy and setting method as described in any one of claims 1-8 when running the program instructions.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, which, when executed by a processor, implements a novel grid-connected new energy transient reactive support strategy and setting method as described in any one of claims 1-8 above.

Images