Memory garbage collection method, device, equipment, storage medium and program product

Abstract

The application relates to a memory garbage collection method, device, equipment, storage medium and program product. The method comprises the following steps: according to update frequency information corresponding to a target data object to be written into a persistent memory, selecting a target log segment from a cold log segment and a hot log segment contained in the persistent memory to store the target data object; if the target data object is update data, recording garbage collection auxiliary information of an old data object corresponding to the target data object in the persistent memory; wherein the garbage collection auxiliary information of the old data object is used for a computer device to perform garbage collection processing on a log segment to which the old data object belongs according to the garbage collection auxiliary information of the old data object. The method can improve the efficiency of memory garbage collection.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a memory garbage collection method, apparatus, device, storage medium, and program product. Background Technology

[0002] Persistent memory (PM) is a novel storage technology. PM offers memory-level access and byte-level addressing capabilities, while also boasting advantages such as data retention under power loss, large capacity, and low cost. Current PM-based storage systems employ a log structure design to fully leverage PM's performance advantages. Specifically, the log-structured key-value store sequentially appends newly written key-value objects to the end of the log and maintains a global index for indexing these objects. The sequential write characteristic of the log structure avoids random writes, and memory management based on the log structure avoids the internal fragmentation problems of traditional memory allocators, achieving high space utilization.

[0003] However, in a log-structured key-value store system, as some key-value objects are updated or deleted, some key-value objects in the log segment become invalid garbage. The log-structured key-value store system reclaims the space occupied by the old key-value objects by merging the log segments. However, traditional garbage collection methods are inefficient. Summary of the Invention

[0004] Therefore, it is necessary to provide a memory garbage collection method, apparatus, device, storage medium, and program product that can improve the efficiency of memory garbage collection in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a memory garbage collection method. The method includes: selecting a target log segment from cold and hot log segments contained in persistent memory to store the target data object based on the update frequency information corresponding to the target data object to be written to persistent memory; if the target data object is updated data, recording garbage collection auxiliary information of the old data object corresponding to the target data object in persistent memory; wherein, the garbage collection auxiliary information of the old data object is used by a computer device to perform garbage collection processing on the log segment to which the old data object belongs based on the garbage collection auxiliary information of the old data object.

[0006] In one embodiment, based on the update frequency information corresponding to the target data object to be written to persistent memory, a target log segment is selected from the cold log segments and hot log segments contained in persistent memory to store the target data object, including: if the update frequency information indicates frequent updates, then the hot log segment is used as the target log segment to store the target data object; if the update frequency information indicates infrequent updates, then the cold log segment is used as the target log segment to store the target data object.

[0007] In one embodiment, the method further includes: calculating a target hash value for the keys included in the target data object; querying a hash table based on the target hash value, wherein the hash table stores hash values ​​of keys whose update frequency satisfies the frequent update condition; if the target hash value is found in the hash table, the update frequency information is determined to indicate frequent updates; if the target hash value is not found in the hash table, the update frequency information is determined to indicate infrequent updates.

[0008] In one embodiment, the method further includes: upon detecting that a data object is stored in persistent memory, recording the key of the stored data object; and updating the hash table based on the key of the stored data object.

[0009] In one embodiment, the target data object and the old data object contain the same key, and both are target keys. The garbage collection auxiliary information of the old data object includes the size of the old data object, the garbage identifier of the old data object, and the index address of the target index entry. The target index entry includes a target key and a pointer. The garbage identifier of the old data object is used to indicate that the old data object is garbage that needs to be collected.

[0010] In one embodiment, recording garbage collection auxiliary information of the old data object corresponding to the target data object in persistent memory includes: writing the size of the old data object into the pointer of the target index; writing the garbage identifier into the volatile memory corresponding to the storage address of the old data object; and writing the address of the target index into the log segment to which the target data object belongs.

[0011] In one embodiment, the method further includes: updating the target index entry according to the storage address of the target data object to obtain the updated target index entry, wherein the pointer in the updated target index entry contains the storage address of the target data object.

[0012] In one embodiment, the method further includes: obtaining multiple candidate log segments that meet the garbage collection conditions from multiple log segments included in persistent memory, and obtaining a garbage collection auxiliary information set corresponding to each candidate log segment, wherein the garbage collection auxiliary information set includes garbage collection auxiliary information of data objects in the corresponding candidate log segment; determining the amount of garbage in each candidate log segment based on the garbage collection auxiliary information set corresponding to each candidate log segment; determining the merging cost-effectiveness of each candidate log segment based on the amount of garbage in each candidate log segment, the size of each candidate log segment, and the existence time of each candidate log segment; determining a target candidate log segment from each candidate log segment based on the merging cost-effectiveness of each candidate log segment, and performing garbage collection processing on the target candidate log segment.

[0013] In one embodiment, garbage collection processing of the target candidate log segment includes: determining non-garbage data objects from the target candidate log segment based on the garbage collection auxiliary information set of the target candidate log segment; and copying the non-garbage data objects to a new log segment.

[0014] In one embodiment, copying a non-garbage data object to a new log segment includes: copying the non-garbage data object to a buffer in volatile memory; using the NTSTORE instruction to copy the non-garbage data object from the buffer in volatile memory to a new log segment, while recording the address of the non-garbage data object in the new log segment.

[0015] In one embodiment, the method further includes updating the index entry of the non-garbage data object based on the garbage collection auxiliary information set of the target candidate log segment and the address of the non-garbage data object in the new log segment.

[0016] In one embodiment, updating the index entries of non-garbage data objects based on the garbage collection auxiliary information set of the target candidate log segment and the addresses of non-garbage data objects in the new log segment includes: prefetching the index entries of non-garbage data objects into the processor cache using cache line prefetch instructions based on the index entry addresses in the garbage collection auxiliary information set of the target candidate log segment; updating the index entries based on the recorded addresses of non-garbage data objects in the new log segment; recording the index entry addresses of the modified non-garbage data object index entries; and when the index entry addresses of the modified non-garbage data object index entries accumulate to a preset threshold, executing concurrent cache line flush instructions to persist the index entries of non-garbage data objects.

[0017] Secondly, this application also provides a memory garbage collection device. The device includes:

[0018] The selection module is used to select the target log segment from the cold log segments and hot log segments contained in the persistent memory to store the target data object based on the update frequency information corresponding to the target data object to be written to the persistent memory.

[0019] The recording module is used to record garbage collection auxiliary information of the old data object corresponding to the target data object in persistent memory if the target data object is updated data; wherein, the garbage collection auxiliary information of the old data object is used by the computer device to perform garbage collection processing on the log segment to which the old data object belongs based on the garbage collection auxiliary information of the old data object.

[0020] In one embodiment, the selection module is specifically used to store the target data object as the target log segment if the update frequency information indicates frequent updates, and to store the target data object as the target log segment if the update frequency information indicates infrequent updates.

[0021] In one embodiment, the apparatus further includes a determining module, configured to calculate a target hash value for the keys included in the target data object; query a hash table based on the target hash value, wherein the hash table stores hash values ​​of keys whose update frequency satisfies the frequent update condition; if the target hash value is found in the hash table, the update frequency information is determined to indicate frequent updates; if the target hash value is not found in the hash table, the update frequency information is determined to indicate infrequent updates.

[0022] In one embodiment, the apparatus further includes a first update module, which is configured to record the key of the stored data object when it is detected that the data object is stored in persistent memory, and update the hash table according to the key of the stored data object.

[0023] In one embodiment, the target data object and the old data object contain the same key, and both are target keys. The garbage collection auxiliary information of the old data object includes the size of the old data object, the garbage identifier of the old data object, and the index address of the target index entry. The target index entry includes a target key and a pointer. The garbage identifier of the old data object is used to indicate that the old data object is garbage that needs to be collected.

[0024] In one embodiment, the recording module is specifically used to write the size of the old data object into the pointer of the target index entry; write the garbage identifier into the volatile memory corresponding to the storage address of the old data object; and write the address of the target index entry into the log segment to which the target data object belongs.

[0025] In one embodiment, the device further includes a second update module, which is used to update the target index entry according to the storage address of the target data object to obtain an updated target index entry, wherein the pointer in the updated target index entry contains the storage address of the target data object.

[0026] In one embodiment, the device further includes a log segment recycling module, which is configured to: obtain multiple candidate log segments that meet the garbage collection conditions from multiple log segments included in persistent memory; obtain a garbage collection auxiliary information set corresponding to each candidate log segment, the garbage collection auxiliary information set including garbage collection auxiliary information of data objects in the corresponding candidate log segment; determine the amount of garbage in each candidate log segment based on the garbage collection auxiliary information set corresponding to each candidate log segment; determine the merging cost-effectiveness of each candidate log segment based on the amount of garbage in each candidate log segment, the size of each candidate log segment, and the existence time of each candidate log segment; and determine a target candidate log segment from the candidate log segments based on the merging cost-effectiveness of each candidate log segment, and perform garbage collection processing on the target candidate log segment.

[0027] In one embodiment, the log segment recycling module is specifically used to determine non-garbage data objects from the target candidate log segment based on the garbage collection auxiliary information set of the target candidate log segment; and copy the non-garbage data objects to a new log segment.

[0028] In one embodiment, the log segment recycling module is specifically used to copy non-garbage data objects to a buffer in volatile memory; the NTSTORE instruction is used to copy the non-garbage data objects in the buffer in volatile memory to a new log segment, while recording the address of the non-garbage data objects in the new log segment.

[0029] In one embodiment, the apparatus further includes a third update module, which is used to update the index entries of non-garbage data objects based on the garbage collection auxiliary information set of the target candidate log segment and the addresses of non-garbage data objects in the new log segment.

[0030] In one embodiment, the third update module is specifically used to prefetch the index entries of non-garbage data objects into the processor cache through cache line prefetch instructions based on the index entry addresses in the garbage collection auxiliary information set of the target candidate log segment; update the index entries according to the recorded addresses of the non-garbage data objects in the new log segment; record the index entry addresses of the modified non-garbage data object index entries; and when the index entry addresses of the modified non-garbage data object index entries accumulate to a preset threshold, execute concurrent cache line flush instructions to persist the index entries of the non-garbage data objects.

[0031] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the memory garbage collection method as described in any of the first aspects above.

[0032] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the memory garbage collection method as described in any of the first aspects above.

[0033] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the memory garbage collection method as described in any of the first aspects above.

[0034] The aforementioned memory garbage collection method, apparatus, device, storage medium, and program products select a target log segment from the cold and hot log segments contained in persistent memory to store the target data object based on the update frequency information corresponding to the target data object to be written to persistent memory. Then, if the target data object is updated data, garbage collection auxiliary information of the old data object corresponding to the target data object is recorded in persistent memory. This garbage collection auxiliary information of the old data object is used by the computer device to perform garbage collection processing on the log segment to which the old data object belongs. Through this method, when writing data objects, they are written to different cold and hot log segments according to their update frequency. Simultaneously, garbage collection auxiliary information of the old data object is recorded when the data object is updated. Thus, during garbage collection, data objects with low update frequency and data objects with high update frequency reside in different log segments. Data objects with low update frequency do not need to undergo frequent garbage collection log segment merging operations with data objects with high update frequency in the same log segment. Furthermore, garbage collection is performed based on the recorded garbage collection auxiliary information, resulting in higher garbage collection efficiency. Attached Figure Description

[0035] Figure 1 This is a flowchart of a memory garbage collection method in one embodiment;

[0036] Figure 2 This is a flowchart illustrating a memory garbage collection method in another embodiment;

[0037] Figure 3 This is a schematic diagram of the index structure of a memory garbage collection method in another embodiment;

[0038] Figure 4 This is a flowchart illustrating a memory garbage collection method in another embodiment;

[0039] Figure 5 This is a flowchart illustrating a memory garbage collection method in another embodiment;

[0040] Figure 6 This is a flowchart illustrating a memory garbage collection method in another embodiment;

[0041] Figure 7 This is a flowchart illustrating a memory garbage collection method in another embodiment;

[0042] Figure 8 This is a schematic diagram of the log copying process of a memory garbage collection method in another embodiment;

[0043] Figure 9 This is a flowchart illustrating a memory garbage collection method in another embodiment;

[0044] Figure 10This is a flowchart illustrating a memory garbage collection method in another embodiment;

[0045] Figure 11 This is a flowchart illustrating a memory garbage collection method in another embodiment;

[0046] Figure 12 A flowchart for updating index entries in a memory garbage collection method in another embodiment;

[0047] Figure 13 This is a flowchart illustrating a memory garbage collection method in another embodiment;

[0048] Figure 14 This is a structural block diagram of a memory garbage collection device in another embodiment;

[0049] Figure 15 This is a structural block diagram of a memory garbage collection device in another embodiment;

[0050] Figure 16 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] Persistent memory (PM) is a novel storage technology. PM offers memory-level access and byte-level addressing capabilities, while also boasting advantages such as data retention under power loss, large capacity, and low cost. Current PM-based storage systems employ a log structure design to fully leverage PM's performance advantages. Specifically, the log-structured key-value store sequentially appends newly written key-value objects to the end of the log and maintains a global index for indexing these objects. The sequential write characteristic of the log structure avoids random writes, and memory management based on the log structure avoids the internal fragmentation problems of traditional memory allocators, achieving high space utilization.

[0053] However, in a log-structured key-value store system, as some key-value objects are updated or deleted, some key-value objects in the log segment become invalid garbage. The log-structured key-value store system reclaims the space occupied by the old key-value objects by merging the log segments. However, traditional garbage collection methods are inefficient.

[0054] In view of this, embodiments of this application provide a memory garbage collection method, which includes multiple foreground threads and multiple background threads.

[0055] like Figure 1 As shown, each foreground thread includes a write phase and an update phase. The persistent memory log is divided into several log segments of 4 megabytes in size. Each foreground thread simultaneously holds two private log segments, one for storing cold key-value objects and the other for storing hot key-value objects. When a foreground thread performs a write operation, it writes the key-value object to the corresponding log segment based on its cold or hot status. When a log segment is full and cannot accommodate new key-value objects, the foreground thread marks the log segment as mergeable and obtains a new empty log segment from the global empty log segment pool. When a foreground thread performs an update operation, it records garbage collection auxiliary information for the old data corresponding to the key-value object while updating the written key-value object.

[0056] Each background thread includes a log merging phase and an index update phase. The log merging phase performs a merging operation on log segments when they are in a mergeable state and meet garbage collection criteria. Based on garbage collection auxiliary information recorded by the foreground thread, it identifies all non-garbage valid key-value objects in the old log segment to be reclaimed and copies all non-garbage valid key-value objects to the new log segment. The index update phase, after the valid key-value objects have been copied, updates the index entries based on the addresses of the valid key-value objects in the new log segment. Based on the garbage collection auxiliary information recorded by the foreground thread, it uses concurrent instructions to concurrently persist the index entries of the valid key-value objects.

[0057] In one embodiment, such as Figure 2 As shown, a memory garbage collection method is provided. Taking its application to a terminal as an example, it can be understood that this method can also be applied to servers, and to systems including both terminals and servers, and implemented through interaction between the terminal and the server. The method includes the following steps:

[0058] Step 201: The terminal selects the target log segment from the cold log segments and hot log segments contained in the persistent memory to store the target data object based on the update frequency information corresponding to the target data object to be written to the persistent memory.

[0059] The target data object can be a key-value object to be written to persistent memory, and the target log segment can be either a selected cold or hot log segment used to store the target data object. The persistent memory log structure key-value storage system appends newly written key-value objects sequentially to the end of the log and maintains a global index for indexing key-value objects. For example... Figure 3 As shown, the global index includes multiple index entries. The key of the index entry is the key of the key-value object 1 that is written. The value of the index entry is an 8-byte composite pointer, where the high 16 bits are the size of key-value object 1 and the low 48 bits are the address of key-value object 1 in persistent memory.

[0060] Persistent memory comprises multiple foreground threads, each holding two private log segments to store cold and hot key-value objects, respectively. When a foreground thread performs a write operation, it selects the appropriate log segment based on the key-value object's temperature (hot or cold) and writes the object to that segment. The temperature of a key-value object is determined by its update frequency. Figure 3 As shown, the foreground thread also records the address of the index item in the log segment where the key-value object is located.

[0061] Step 202: If the target data object is updated data, record the garbage collection auxiliary information of the old data object corresponding to the target data object in persistent memory.

[0062] The garbage collection auxiliary information for old data objects is used by computer equipment to perform garbage collection on the log segment to which the old data objects belong. When the key of the target data object is the same as the key of an index entry in the global index, it indicates that the target data object is updated. In this case, the old data object before the update becomes invalid garbage data, and the space occupied by the old data object can be reclaimed by performing garbage collection on the log segment to which the old data object belongs. In another case, when the key of the target data object is the same as the key of an index in the global index, and the value corresponding to the key of the target data object is empty, it indicates that a deletion operation is performed on the target data object. In this case, the deleted old data object becomes invalid garbage data. Optionally, the log segment to which the old data object belongs and the log segment to which the target data object is written can be the same or different.

[0063] In the aforementioned memory garbage collection method, based on the update frequency information of the target data object to be written to persistent memory, a target log segment is selected from the cold and hot log segments contained in persistent memory to store the target data object. Then, if the target data object is updated data, garbage collection auxiliary information of the old data object corresponding to the target data object in persistent memory is recorded. This garbage collection auxiliary information of the old data object is used by the computer device to perform garbage collection processing on the log segment to which the old data object belongs. Through this method, when writing data objects, they are written to different cold and hot log segments according to their update frequency. Simultaneously, garbage collection auxiliary information of the old data object is recorded when the data object is updated. Thus, during garbage collection, data objects with low update frequency and data objects with high update frequency reside in different log segments. Data objects with low update frequency do not need to be frequently merged with data objects with high update frequency in the same log segment, and garbage collection is performed based on the recorded garbage collection auxiliary information, resulting in higher garbage collection efficiency.

[0064] In one embodiment, whether the target data object to be written is written to a cold log segment or a hot log segment is determined based on the update frequency information of the target data object to be written, specifically in the following two cases:

[0065] In the first case, when the update frequency information indicates frequent updates, the hot log segment is used as the target log segment to store the target data object.

[0066] The update frequency information refers to the frequency of updates to the target data object. This application determines the update frequency information of the target data object through a global static read-only hash table used to determine the "hotness" or "coldness" of the target data object. For example... Figure 4 As shown, the specific methods include:

[0067] Step 401: The terminal calculates the target hash value of the keys included in the target data object.

[0068] Calculate the hash value of the key of the target data object to be written to as the target hash value.

[0069] Step 402: The terminal queries the hash table based on the target hash value.

[0070] The hash table stores the hash values ​​of keys whose update frequency meets the frequent update condition. The query checks whether the calculated target hash value exists in the hash table.

[0071] Step 403: If the target hash value is found in the hash table, the update frequency information is determined to be frequently updated.

[0072] If the target hash value exists in the hash table, it indicates that the target data object is frequently updated. In this case, the hot log segment is used as the target log segment to store the log data object.

[0073] Step 404: If the target hash value is not found in the hash table, the update frequency information is determined to be infrequent.

[0074] Conversely, if the target hash value does not exist in the hash table, it means that the update frequency information of the target data object is not frequently updated.

[0075] In the second scenario, when the update frequency information indicates infrequent updates, the cold log segment is used as the target log segment to store the target data object.

[0076] In the above embodiments, by using the cold and hot separation technology, data objects with different update frequencies are stored in different log segments. Infrequently updated data objects generate less garbage and do not need to be stored in the same log segment as frequently updated data objects for frequent garbage collection, thus improving the efficiency of garbage collection.

[0077] In one embodiment, such as Figure 5As shown, the aforementioned global static read-only hash table can be modified based on the update frequency of the keys of the written data objects. This enables real-time detection of hotspot migrations, replacing the static read-only hash table by calculating a new set of hotkeys. Specific methods include:

[0078] Step 501: When the terminal detects that a data object has been stored in persistent memory, it records the key of the stored data object.

[0079] Optionally, when writing to a data object, the foreground thread records the hash value of the data object's key in a local buffer.

[0080] Step 502: The terminal updates the hash table based on the key of the stored data object.

[0081] A background thread scans the buffer containing the hash values ​​of keys recorded by the foreground thread, counting the frequency of key updates for each written data object. Then, using a TopK algorithm with a min-heap, it obtains the K most frequently updated keys for each written data object, thus creating a new set of hot keys. This generates a new static read-only hash table containing the hash values ​​of the K most frequently updated keys. Finally, through an RCU (Read-Copy Update) mechanism, the old static read-only hash table is safely replaced once all foreground threads no longer access it.

[0082] In one embodiment, the target data object and the old data object contain the same key, and both are target keys. The garbage collection auxiliary information of the old data object includes the size of the old data object, the garbage identifier of the old data object, and the index address of the target index entry. The target index entry includes the target key and a pointer. The garbage identifier of the old data object is used to indicate that the old data object is garbage that needs to be reclaimed.

[0083] When the target data object is updated data, that is, when there is an old data object and the target data object with the same key, the old data object becomes garbage data. The background thread can perform garbage collection processing on the log segment to which the old data object belongs based on the garbage collection auxiliary information of the old data object.

[0084] The garbage collection auxiliary information for old data objects includes the size of the old data object, and the amount of garbage in the corresponding log segment is increased based on this size. The garbage collection auxiliary information also includes a garbage identifier for the old data object, which indicates that the old data object is garbage that needs to be reclaimed. Finally, the garbage collection auxiliary information also includes the address of the target index entry, which includes the target key and a pointer.

[0085] In one embodiment, the foreground thread records garbage collection auxiliary information of the old data object corresponding to the target data object in persistent memory, such as... Figure 6 As shown, it specifically includes:

[0086] Step 601: The terminal writes the size of the old data object into the pointer of the target index item.

[0087] When the foreground thread updates an index entry, it increases the amount of garbage in the log segment based on the size of the data object in the high 16 bits of the index entry pointer.

[0088] Step 602: The terminal writes the garbage identifier to the volatile memory corresponding to the storage address of the old data object.

[0089] The foreground thread calculates the corresponding log segment based on the lower 48 bits of the index pointer of the old data object corresponding to the target data object. Garbage identification records for the log segment are stored in volatile memory. These records are implemented using a bitmap located in volatile memory. To locate the garbage identifier of a mutable data object based on its address, this application optionally defaults to a minimum data object size of 32 bytes, reserving one bit for every 32 bytes in the bitmap. When marking a garbage identifier, the offset of the data object's address relative to its corresponding log segment is divided by 32 to obtain the bit position of the data object in the bitmap. The foreground thread sets the bit corresponding to the garbage identifier record to 1 to indicate that the data object is no longer valid.

[0090] Step 603: The terminal writes the address of the target index item to the log segment to which the target data object belongs.

[0091] The target index entry includes the target key and a pointer. The address of the target index entry is first cached in a volatile memory buffer. When the log segment can no longer hold the new key-value object and index entry address, the index entry address is then written and persisted to the log segment. Persistence is a mechanism for converting data between a persistent and transient state; that is, transient data, such as data in memory that cannot be permanently stored, is persisted to persistent memory for permanent storage. Cacheing the target index entry address in a volatile memory buffer before writing and persisting it to the log segment reduces persistence overhead.

[0092] In one embodiment of this application, after the foreground thread updates the target data object, it is also necessary to update the index of the target data object. The target index item is updated according to the storage address of the target data object to obtain the updated target index item.

[0093] The updated target index entry contains the storage address of the target data object. After the foreground thread updates the target data object, it needs to update the target data object's index, changing the object address in the pointer of the index entry to the target object's storage address.

[0094] In one embodiment of this application, such as Figure 7 As shown, the method also includes:

[0095] Step 701: The terminal obtains multiple candidate log segments that meet the garbage collection conditions from the multiple log segments included in persistent memory, and obtains the garbage collection auxiliary information set corresponding to each candidate log segment.

[0096] When a log segment is full and cannot accommodate new key-value objects, the foreground thread marks the log segment as mergeable, making it a candidate log segment that meets the garbage collection criteria. The background thread retrieves the garbage collection auxiliary information set corresponding to each candidate log segment. This set includes garbage collection auxiliary information for the data objects within the corresponding candidate log segments.

[0097] Step 702: The terminal determines the amount of garbage in each candidate log segment based on the garbage collection auxiliary information set corresponding to each candidate log segment.

[0098] The terminal obtains the amount of garbage based on the object size in the garbage collection auxiliary information of each data object in the candidate log segment, and sums up the garbage amounts of all garbage data objects in the log segment to obtain the total amount of garbage in the entire candidate log segment.

[0099] Step 703: The terminal determines the cost-effectiveness of merging each candidate log segment based on the amount of garbage in each candidate log segment, the size of each candidate log segment, and the existence time of each candidate log segment.

[0100] The cost-benefit calculation method for merging log segments is as follows:

[0101]

[0102]

[0103] Where score represents cost-effectiveness, u represents the utilization rate of the log segment, and age represents the time from when the log segment is full until the current moment.

[0104] Step 704: The terminal determines the target candidate log segment from the candidate log segments based on the cost-effectiveness of merging each candidate log segment, and performs garbage collection on the target candidate log segment.

[0105] Once a foreground thread's log segment is full, it assigns the log segment number modulo the number of background threads to a background thread and adds it to that background thread's merge queue. The background thread then selects the log segment from its merge queue that offers the highest cost-effectiveness for garbage collection.

[0106] In the above embodiments, by calculating the merging cost and benefit of each candidate log segment, the candidate log segment with the largest merging cost and benefit is selected as the target log segment for garbage collection, which improves the efficiency of garbage collection.

[0107] After identifying the target candidate log segment, the steps for garbage collection of the target candidate log segment are as follows: Figure 8 As shown, it includes:

[0108] Step 801: The terminal determines non-garbage data objects from the target candidate log segment based on the garbage collection auxiliary information set of the target candidate log segment.

[0109] The terminal determines the non-garbage data in the target candidate log segment based on the garbage identifier of each data object in the garbage collection auxiliary information set of the target candidate log segment. When the garbage identifier corresponding to the data object is 1, it indicates that the data object is garbage data. When the garbage identifier corresponding to the data object is 0, it indicates that the data object is non-garbage data.

[0110] Step 802: The terminal copies the non-garbage data objects to a new log segment.

[0111] The background thread copies non-garbage data objects to a new log segment, as follows: Figure 9 The following are included:

[0112] Step 901: The terminal copies non-garbage data objects to a buffer in volatile memory.

[0113] Step 902: The terminal uses the NTSTORE instruction to copy the non-garbage data objects in the volatile memory buffer to a new log segment, and records the address of the non-garbage data objects in the new log segment.

[0114] For example, such as Figure 10 As shown, the log segment to be merged includes objects 1 to 4. The garbage identifier is represented in binary as shown in the figure. It can be seen that the garbage identifier corresponding to objects 1 and 3 is 1, that is, objects 1 and 3 are garbage data, and objects 2 and 4 are non-garbage data. First, objects 2 and 4 are copied to the buffer of volatile memory. Then, the NTSTORE instruction is used to copy the non-garbage data objects in the buffer of volatile memory to the new log segment, and at the same time, the address of the non-garbage data objects in the new log segment is recorded.

[0115] In the above embodiments, during the copying phase of non-garbage data objects, the background thread uses the NTSTORE instruction, a copying method with higher bandwidth, which reduces the overhead of persistence instructions.

[0116] After the log segments are merged, the index entries for the non-garbage data objects need to be updated based on the garbage collection auxiliary information set of the target candidate log segment and the addresses of the non-garbage data objects in the new log segment. This ensures that the non-garbage data objects in the new log segment can be indexed through the index entries. Figure 11 As shown, the steps for updating an index item include:

[0117] Step 1101: The terminal prefetches the index entries of non-garbage data objects into the processor cache using a cache line prefetch instruction based on the index entry addresses in the garbage collection auxiliary information set of the target candidate log segment, and updates the index entries according to the addresses of the recorded non-garbage data objects in the new log segment.

[0118] Optionally, the background thread can quickly locate the index entry of the non-garbage data that needs to be modified based on the index entry address recorded in the garbage collection auxiliary information set of the target candidate log segment. It then uses the cache line prefetch instruction Prefetch to prefetch the index entry of the non-garbage data object into the processor cache based on the index entry address, and modifies the address of the index entry pointer in the index entry to the address of the non-garbage data object in the new log segment.

[0119] Step 1102: The terminal records the index address of the modified non-garbage data object's index entry. When the index address of the modified non-garbage data object's index entry accumulates to a preset threshold, concurrent cache line flushing instructions are executed to persist the index entry of the non-garbage data object.

[0120] Optionally, after an index entry is modified, the background thread does not immediately persist the index entry, but records the address of the modified index entry. When the accumulated addresses of the modified non-garbage data object index entries reach a preset threshold, such as a preset threshold of 32, the concurrent cache line flush instruction CLWB is executed to persist the index entry of the non-garbage data object. The concurrent cache line flush instruction can be executed concurrently in the processor pipeline, which can reduce the latency of sequential cache line flushing in traditional methods.

[0121] The specific update process steps, such as Figure 12 As shown.

[0122] (1) Prefetch the index of the next non-garbage data object based on the index address of the record.

[0123] (2) Update the index entries of the current data object.

[0124] (3) If the number of updated index items reaches the preset threshold, proceed to step (4) if the number of updated index items reaches the preset threshold, otherwise return to step (1).

[0125] (4) Perform concurrent cache line persistence instructions on all non-persistent index entries.

[0126] (5) Check if all non-garbage data object index entries have been updated. If they have, end the process. If they have not, return to step (1).

[0127] In the embodiments of this application, please refer to Figure 13 The document illustrates a flowchart of a memory garbage collection method provided in an embodiment of this application. The memory garbage collection method includes the following steps:

[0128] Step 1301: The terminal selects the target log segment from the cold log segments and hot log segments contained in the persistent memory to store the target data object based on the update frequency information corresponding to the target data object to be written to the persistent memory.

[0129] Step 1302: If the target data object is updated data, record the garbage collection auxiliary information of the old data object corresponding to the target data object in persistent memory.

[0130] Step 1303: The terminal obtains multiple candidate log segments that meet the garbage collection conditions from the multiple log segments included in persistent memory, and obtains the garbage collection auxiliary information set corresponding to each candidate log segment.

[0131] Step 1304: The terminal determines the amount of garbage in each candidate log segment based on the garbage collection auxiliary information set corresponding to each candidate log segment.

[0132] Step 1305: The terminal determines the cost-effectiveness of merging each candidate log segment based on the amount of garbage in each candidate log segment, the size of each candidate log segment, and the existence time of each candidate log segment.

[0133] Step 1306: The terminal determines the target candidate log segment from the candidate log segments based on the cost-effectiveness of merging each candidate log segment, and performs garbage collection on the target candidate log segment.

[0134] Step 1307: The terminal determines non-garbage data objects from the target candidate log segment based on the garbage collection auxiliary information set of the target candidate log segment.

[0135] Step 1308: The terminal copies non-garbage data objects to a buffer in volatile memory.

[0136] Step 1309: The terminal uses the NTSTORE instruction to copy the non-garbage data objects in the volatile memory buffer to a new log segment, and records the address of the non-garbage data objects in the new log segment.

[0137] Step 1310: The terminal prefetches the index entries of non-garbage data objects into the processor cache using a cache line prefetch instruction based on the index entry addresses in the garbage collection auxiliary information set of the target candidate log segment, and updates the index entries based on the addresses of the recorded non-garbage data objects in the new log segment.

[0138] Step 1311: The terminal records the index address of the modified non-garbage data object's index entry. When the index address of the modified non-garbage data object's index entry accumulates to a preset threshold, concurrent cache line flushing instructions are executed to persist the non-garbage data object's index entry.

[0139] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0140] Based on the same inventive concept, this application also provides a memory garbage collection apparatus for implementing the memory garbage collection method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more memory garbage collection apparatus embodiments provided below can be found in the limitations of the memory garbage collection method described above, and will not be repeated here.

[0141] In one embodiment, such as Figure 14 As shown, a memory garbage collection device 1400 is provided, including: a selection module 1401 and a recording module 1402.

[0142] The selection module 1401 is used to select a target log segment from the cold log segments and hot log segments contained in the persistent memory to store the target data object based on the update frequency information corresponding to the target data object to be written to the persistent memory.

[0143] The recording module 1402 is used to record garbage collection auxiliary information of the old data object corresponding to the target data object in persistent memory if the target data object is updated data; wherein, the garbage collection auxiliary information of the old data object is used by the computer device to perform garbage collection processing on the log segment to which the old data object belongs based on the garbage collection auxiliary information of the old data object.

[0144] In one embodiment, the selection module 1401 is specifically used to store the target data object as the target log segment if the update frequency information indicates frequent updates; and to store the target data object as the target log segment if the update frequency information indicates infrequent updates.

[0145] In one embodiment, the apparatus further includes a determining module 1501, which is configured to calculate a target hash value of the keys included in the target data object; query a hash table based on the target hash value, wherein the hash table stores hash values ​​of keys whose update frequency satisfies the frequent update condition; if the target hash value is found in the hash table, the update frequency information is determined to indicate frequent updates; if the target hash value is not found in the hash table, the update frequency information is determined to indicate infrequent updates.

[0146] In one embodiment, the device further includes a first update module 1502, which is used to record the key of the stored data object when it is detected that the data object is stored in persistent memory; and update the hash table according to the key of the stored data object.

[0147] In one embodiment, the target data object and the old data object contain the same key, and both are target keys. The garbage collection auxiliary information of the old data object includes the size of the old data object, the garbage identifier of the old data object, and the index address of the target index entry. The target index entry includes a target key and a pointer. The garbage identifier of the old data object is used to indicate that the old data object is garbage that needs to be collected.

[0148] In one embodiment, the recording module 1402 is specifically used to write the size of the old data object into the pointer of the target index entry; write the garbage identifier into the volatile memory corresponding to the storage address of the old data object; and write the address of the target index entry into the log segment to which the target data object belongs.

[0149] In one embodiment, the device further includes a second update module 1503, which is used to update the target index entry according to the storage address of the target data object to obtain the updated target index entry, wherein the pointer in the updated target index entry contains the storage address of the target data object.

[0150] In one embodiment, the device further includes a log segment recycling module 1504, which is configured to: obtain multiple candidate log segments that meet the garbage collection conditions from multiple log segments included in persistent memory; obtain a garbage collection auxiliary information set corresponding to each candidate log segment, the garbage collection auxiliary information set including garbage collection auxiliary information of data objects in the corresponding candidate log segment; determine the amount of garbage in each candidate log segment based on the garbage collection auxiliary information set corresponding to each candidate log segment; determine the merging cost-effectiveness of each candidate log segment based on the amount of garbage in each candidate log segment, the size of each candidate log segment, and the existence time of each candidate log segment; and determine a target candidate log segment from each candidate log segment based on the merging cost-effectiveness of each candidate log segment, and perform garbage collection processing on the target candidate log segment.

[0151] In one embodiment, the log segment recycling module 1504 is specifically used to determine non-garbage data objects from the target candidate log segment based on the garbage collection auxiliary information set of the target candidate log segment; and copy the non-garbage data objects to a new log segment.

[0152] In one embodiment, the log segment recycling module 1504 is specifically used to copy non-garbage data objects to a buffer in volatile memory; the NTSTORE instruction is used to copy the non-garbage data objects in the buffer in volatile memory to a new log segment, and the address of the non-garbage data objects in the new log segment is recorded at the same time.

[0153] In one embodiment, the device further includes a third update module 1505, which is used to update the index entries of non-garbage data objects based on the garbage collection auxiliary information set of the target candidate log segment and the addresses of non-garbage data objects in the new log segment.

[0154] In one embodiment, the third update module 1505 is specifically used to prefetch the index entries of non-garbage data objects into the processor cache through cache line prefetch instructions based on the index entry addresses in the garbage collection auxiliary information set of the target candidate log segment; update the index entries according to the recorded addresses of the non-garbage data objects in the new log segment; record the index entry addresses of the modified non-garbage data object index entries; and when the index entry addresses of the modified non-garbage data object index entries accumulate to a preset threshold, execute concurrent cache line flush instructions to persist the index entries of the non-garbage data objects.

[0155] The modules in the aforementioned memory garbage collection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0156] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 16 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a memory garbage collection method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0157] Those skilled in the art will understand that Figure 16 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0158] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the memory garbage collection methods provided in the above-described method embodiments.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the memory garbage collection methods provided in the above-described method embodiments.

[0160] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the memory garbage collection methods provided in the above-described method embodiments.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0162] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A memory garbage collection method, characterized in that, The method includes: Based on the update frequency information corresponding to the target data object to be written to persistent memory, a target log segment is selected from the cold log segments and hot log segments contained in the persistent memory to store the target data object; If the target data object is updated data, then record the garbage collection auxiliary information of the old data object corresponding to the target data object in the persistent memory; The garbage collection auxiliary information of the old data object is used to enable computer equipment to perform garbage collection processing on the log segment to which the old data object belongs based on the garbage collection auxiliary information of the old data object; The target data object and the old data object contain the same key, and both are target keys. The garbage collection auxiliary information of the old data object includes the size of the old data object, the garbage identifier of the old data object, and the index address of the target index item. The target index item includes the target key and a pointer. The garbage identifier of the old data object is used to indicate that the old data object is garbage that needs to be collected. The method of recording garbage collection auxiliary information for the old data object corresponding to the target data object in persistent memory includes: Write the size of the old data object into the pointer of the target index item; Write the garbage identifier to the volatile memory corresponding to the storage address of the old data object; Write the address of the target index entry to the log segment to which the target data object belongs; The method further includes: Multiple candidate log segments that meet the garbage collection conditions are obtained from the multiple log segments included in the persistent memory, and a garbage collection auxiliary information set corresponding to each candidate log segment is obtained. The garbage collection auxiliary information set includes the garbage collection auxiliary information of the data objects in the corresponding candidate log segment. The amount of garbage in each candidate log segment is determined based on the garbage collection auxiliary information set corresponding to each candidate log segment; The cost-effectiveness of merging each candidate log segment is determined based on the amount of garbage in each candidate log segment, the size of each candidate log segment, and the existence time of each candidate log segment. Based on the cost-effectiveness of merging each candidate log segment, a target candidate log segment is determined from the candidate log segments, and the target candidate log segment is subjected to garbage collection.

2. The method according to claim 1, characterized in that, The method further includes: Calculate the target hash value of the keys included in the target data object; The target hash value is used to query the hash table, which stores the hash values ​​of keys whose update frequency satisfies the frequent update condition; If the target hash value is found in the hash table, the update frequency information indicates frequent updates, and the hot log segment is used as the target log segment to store the target data object. If the target hash value is not found in the hash table, the update frequency information is determined to be an infrequent update, and the cold log segment is used as the target log segment to store the target data object.

3. The method according to claim 2, characterized in that, The method further includes: When a data object is detected to be stored in the persistent memory, the key of the stored data object is recorded; The hash table is updated based on the key of the stored data object.

4. The method according to claim 1, characterized in that, The garbage collection process for the target candidate log segment includes: Based on the garbage collection auxiliary information set of the target candidate log segment, non-garbage data objects are determined from the target candidate log segment; Copy the non-garbage data object to a new log segment.

5. The method according to claim 4, characterized in that, The step of copying the non-garbage data object to the new log segment includes: Copy the non-garbage data object to a buffer in volatile memory; The NTSTORE instruction is used to copy the non-garbage data object from the buffer of the volatile memory to the new log segment, and the address of the non-garbage data object in the new log segment is recorded at the same time.

6. The method according to claim 5, characterized in that, The method further includes: Based on the garbage collection auxiliary information set of the target candidate log segment and the address of the non-garbage data object in the new log segment, update the index entry of the non-garbage data object.

7. The method according to claim 6, characterized in that, The step of updating the index entry of the non-garbage data object based on the garbage collection auxiliary information set of the target candidate log segment and the address of the non-garbage data object in the new log segment includes: Based on the index address in the garbage collection auxiliary information set of the target candidate log segment, the index of the non-garbage data object is prefetched into the processor cache through the cache line prefetch instruction, and the index is updated according to the recorded address of the non-garbage data object in the new log segment; Record the index address of the modified index entry of the non-garbage data object. When the index address of the modified index entry of the non-garbage data object accumulates to a preset threshold, execute concurrent cache line flushing instructions to persist the index entry of the non-garbage data object.

8. A memory garbage collection device, characterized in that, The device includes: The selection module is used to select a target log segment from the cold log segments and hot log segments contained in the persistent memory to store the target data object based on the update frequency information corresponding to the target data object to be written to the persistent memory. The recording module is used to record garbage collection auxiliary information of the old data object corresponding to the target data object in the persistent memory if the target data object is updated data; wherein, the garbage collection auxiliary information of the old data object is used for the computer device to perform garbage collection processing on the log segment to which the old data object belongs based on the garbage collection auxiliary information of the old data object; The target data object and the old data object contain the same key, and both are target keys. The garbage collection auxiliary information of the old data object includes the size of the old data object, the garbage identifier of the old data object, and the index address of the target index item. The target index item includes the target key and a pointer. The garbage identifier of the old data object is used to indicate that the old data object is garbage that needs to be collected. The recording module is specifically used to write the size of the old data object into the pointer of the target index entry; write the garbage identifier into the volatile memory corresponding to the storage address of the old data object; and write the address of the target index entry into the log segment to which the target data object belongs. The log segment reclamation module is configured to: obtain multiple candidate log segments that meet garbage collection conditions from the multiple log segments included in the persistent memory; obtain a garbage collection auxiliary information set corresponding to each candidate log segment, wherein the garbage collection auxiliary information set includes garbage collection auxiliary information of data objects in the corresponding candidate log segment; determine the amount of garbage in each candidate log segment based on the garbage collection auxiliary information set corresponding to each candidate log segment; determine the merging cost-effectiveness of each candidate log segment based on the amount of garbage in each candidate log segment, the size of each candidate log segment, and the existence time of each candidate log segment; determine a target candidate log segment from the candidate log segments based on the merging cost-effectiveness of each candidate log segment, and perform garbage collection processing on the target candidate log segment.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images