Data storage method, device, medium and computer program product

By finding suitable pages in the data storage for data writing and GC writing, combining arrays and heap structure to manage available space within the page, the problem of insufficient available space management in the existing technology is solved, and disk space utilization and data security are improved sex.

CN118502682BActive Publication Date: 2024-11-19NANJING PENGYUN NETWORK TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410977247.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-19
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

In the management of available space in the field of data storage, there are shortcomings in disk space utilization, memory share and data security, making it difficult to efficiently manage disk space and ensure data security.

Method used

Using data storage method, search for pages with suitable tail-free space in persistent storage space and write data in them; perform GC writes to optimize space utilization in the absence of suitable pages; use arrays and heaps Structure manages available space within the page, records metadata and performs regular persistence to ensure data security.

Benefits of technology

Improve disk space utilization and memory usage efficiency, ensure data security and reliability, avoid data loss, and reduce memory usage and resource consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118502682B_ABST
    Figure CN118502682B_ABST
Patent Text Reader

Abstract

The present invention relates to a data storage method, device, medium and computer program product. The method comprises: recording a first array composed of tail free space metadata and total free space metadata of each page in a persistent storage space, a bit array simulation heap of the corresponding relationship between the tail free space metadata of each page and the sequence number of each page; receiving a data write request, searching for a page with tail free space from the bit array simulation heap, wherein the tail free space is not less than the size of the data requested to be written; when the page with tail free space is found, writing the data to the tail of the found page; when the page with tail free space is not found, searching for a page with total free space from the first array, performing GC on the page with total free space and writing the data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data storage method, device, medium and computer program product. Background Art

[0002] In the field of data storage, free space management is an indispensable function. Before data is actually stored on the hard disk, the write location and sector size need to be allocated, which is an inevitable problem in the bare disk write scenario. Its performance and availability will seriously affect the performance and availability of the entire storage system. Therefore, efficient and reliable management of available space is very necessary and indispensable. However, the available space management solutions in the existing technology are still insufficient in terms of disk space utilization, memory occupancy and data security. Summary of the invention

[0003] The purpose of the present invention is to provide a data storage method, device, medium and computer program product to solve the technical problem of how to improve disk space utilization, memory occupancy and enhance data security in available space management.

[0004] A first embodiment of the present invention discloses a data storage method for an electronic device, the method comprising:

[0005] Receive a data write request, and search a persistent storage space including a plurality of pages for a page having a suitable tail available space among the plurality of pages, wherein the suitable tail available space is a tail available space not less than the data requested to be written;

[0006] When the page with suitable tail available space is found, the tail of the data is written into the found page;

[0007] If the page with the appropriate tail free space is not found, the data GC is written to a page with the appropriate total free space among the multiple pages.

[0008] Optionally, it also includes:

[0009] A bit array simulating a heap that records a first array consisting of the tail free space metadata and the total free space metadata of each of the pages, and a correspondence between the tail free space metadata of each of the pages and the sequence number of each of the pages;

[0010] Searching for a page with suitable tail available space among the plurality of pages comprises: searching for the page with suitable tail available space from the bit array simulation heap;

[0011] Writing the GC data into a page with a suitable total available space among the multiple pages includes: searching the first array for the page with a suitable total available space and writing the GC data into the page.

[0012] Optionally, it also includes:

[0013] After writing the data, a write-before log is recorded, wherein the write-before log includes the tail available space metadata and the total available space metadata of each page after the data is written;

[0014] The tail free space metadata and the total free space metadata in the first array and the bit array simulation heap are updated.

[0015] Optionally, it also includes:

[0016] At predetermined intervals, a copy of the first array is persistently stored, and the write-ahead log before the persistent storage is deleted.

[0017] Optionally, it also includes:

[0018] In the event that the first array and the bit array simulation heap are lost, the first array and the bit array simulation heap are restored based on a copy of the first array in persistent storage and based on all the non-deleted write-ahead logs in chronological order.

[0019] Optionally, it also includes:

[0020] After the data is written, the index of the corresponding data is recorded, where the index is the correspondence between the key of the data and the serial number of the page.

[0021] Optionally, the page includes a plurality of sectors, and the sector stores its own sector metadata and metadata of the stored data, and the metadata of the data includes a key of the data and a length of the data;

[0022] When reading data, the page where the data requested to be read is located is searched according to the index, and the data requested to be read is parsed in the page according to the sector metadata and the metadata of the data.

[0023] Optionally, it also includes:

[0024] In response to data deletion, the index is deleted and the tail free space metadata and the total free space metadata in the first array and the bit array simulation heap are updated.

[0025] Optionally, it also includes:

[0026] When the GC is writing, it is determined whether the index exists for the data to be sorted according to the key of the data to be sorted. If the index exists, the sector where the data to be sorted is located is not overwritten.

[0027] Optionally, the sector metadata includes a version number of the sector metadata, a sector serial number of a previous version, and a sector serial number of a latest version; and the method further includes:

[0028] In the case where there are two versions of sector metadata in the page, the sector order is determined according to the latest version sector sequence number of the current version of the sector metadata and the latest version sector sequence number of the previous version of the sector metadata to parse data.

[0029] Optionally, the data is Blob data, and each sector stores one or more Blob data.

[0030] Optionally, it also includes:

[0031] Adding a write lock to the page to which data is being written;

[0032] After updating the tail available space metadata and the total available space metadata in the first array and the bit array simulation heap, the write lock is released.

[0033] Optionally, it also includes:

[0034] Using one or more second arrays to record the number of corresponding pages having corresponding total available space;

[0035] When the GC writes, it is determined whether there is a page with a suitable total available space according to the second array.

[0036] A second embodiment of the present invention discloses an electronic device, which includes a memory storing computer executable instructions and a processor. When the instructions are executed by the processor, the electronic device implements the data storage method according to the first embodiment of the present invention.

[0037] The third embodiment of the present invention discloses a computer storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer is enabled to execute the data storage method according to the first embodiment of the present invention.

[0038] A fourth embodiment of the present invention discloses a computer program product, comprising computer executable instructions, wherein the instructions are executed by a processor to implement the data storage method according to the first embodiment of the present invention.

[0039] Compared with the prior art, the main differences and effects of the embodiments of the present invention are:

[0040] The present invention is highly efficient: the most suitable page is searched through the heap, the comparison times are controllable and efficient, and the allocation efficiency is high; it is lightweight: the allocation efficiency and resource occupation ratio are balanced, the memory occupation is equivalent to the bitmap management method, the memory space occupied is small, and the persistent metadata space is much smaller than the bitmap management method; it is safe: the available space metadata information and pre-write log are regularly persisted, and these data are restored after the metadata is lost or the system is restarted, ensuring the persistence and reliability of the writable space metadata. The in-page metadata layout and the unique in-page GC method ensure the maximum utilization of the disk space, and will not lose data under any abnormal circumstances, thus ensuring data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flow chart of a data storage method according to an embodiment of the present application is shown.

[0042] Figure 2 A block diagram of a storage system according to an embodiment of the present application is shown.

[0043] Figure 3A A schematic diagram showing a first array as an example according to an embodiment of the present application is shown.

[0044] Figure 3B A schematic diagram showing a bit array simulation heap as an example according to an embodiment of the present application.

[0045] Figure 4 A schematic diagram of a storage system according to an embodiment of the present application is shown.

[0046] Figure 5 A schematic diagram showing a layout example of a page according to an embodiment of the present application.

[0047] Figure 6 A schematic diagram showing a page before normal GC processing and a page after GC processing according to an embodiment of the present application.

[0048] Figure 7 A schematic diagram showing a page before and after GC processing in which a fault occurs according to an embodiment of the present application is shown.

[0049] Fig. 8A A schematic diagram showing a page before normal GC processing and a page after GC processing according to an embodiment of the present application.

[0050] Figure 8B A schematic diagram showing a page before and after GC processing in which a fault occurs according to an embodiment of the present application is shown.

[0051] Fig. 9 A hardware structure block diagram of an electronic device implementing data storage according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0053] Considering the deficiencies of available space management solutions in the prior art in terms of disk space utilization, memory occupancy and data security. The embodiment of the present application provides a data storage method for an electronic device, Figure 1 A flow chart of the method is shown, which includes:

[0054] S102, receiving a data write request, searching a persistent storage space including multiple pages for a page with suitable tail available space among the multiple pages, where the suitable tail available space is no less than the tail available space of the data requested to be written.

[0055] In this application, the disk is divided into multiple containers, which is a continuous disk area. Each container is further divided into multiple pages, which is a continuous sector area. A container is an integer multiple of its internal pages, and a page is an integer multiple of its internal sectors. According to the user's data write request, the data written by the user will be sliced ​​according to the page size in the upper system. Each data slice is called a Blob in this application, and the data of each Blob must be smaller than the page size.

[0056] In this application, two values ​​are used to describe the available space of a page, namely the tail free space and the total free space. All pages are numbered, that is, serial numbers are added, and then a bit array can be used to describe the available space of all pages in a container. Another bit array is used to simulate the heap structure to describe the corresponding relationship between the tail free space and the page serial number.

[0057] As an example, Figure 2The storage system 20 is shown, including a page free space allocator management 21 and a disk 22. The page free space allocator management 21 includes two metadata areas, one of which is an area that needs to be persisted: a persistent free space metadata memory 211, which is a first array composed of the tail free space metadata of each page in the disk 22 and the total free space metadata; and a tail free space metadata sort heap 212, which is a bit array simulation heap of the correspondence between the tail free space metadata of each page in the disk 22 and the sequence number of the page. The above two metadata areas can be in the memory.

[0058] In this application, the number of sectors is used to represent the size of the tail available space and the total available space. n bits can represent a combination of the number of tail or total available sectors in a page from 0 to (2^n - 1). Specifically, Figure 3A The first array 301 is shown as an example, and the size of the entire first array area is (2 * n * number of pages / 8) bytes. Similarly, in the bit array simulation heap, n bits can also be used to represent the tail available space part, and the page number is represented by m bits. Figure 3B A bit array simulation stack 302 is shown as an example.

[0059] As an example, searching for a page with suitable tail free space among multiple pages specifically includes: in response to a user's data write request, first searching for a suitable tail free space in a container that is not less than the data requested to be written from the tail free space metadata sort heap 212 .

[0060] S104, determining whether a page with suitable tail free space is found.

[0061] S106, when a page with suitable tail available space is found, the tail of the data is written into the found page.

[0062] For example, if a suitable tail available space that is not less than the data requested to be written is found from the tail available space metadata sorting heap 212, the serial number of the page corresponding to the suitable tail available space is detected, and then the tail of the data is written to the corresponding page in the disk 22. After the writing is successful, the persistent available space metadata memory 211 and the tail available space metadata sorting heap 212 are updated. The tail writing here is to write the sector offset directly according to the tail available space of the page. In this way, the sequential tail writing of the page is guaranteed first, with extremely high disk writing efficiency and page allocation efficiency.

[0063] S108, when no page with suitable tail free space is found, write the data GC (GarbageCollection) to a page with suitable total free space among multiple pages. For example, once there is no page with suitable tail free space in the above container, the entire persistent free space metadata memory 211 is traversed from the beginning to find a page with suitable total free space in the container, and then the data GC is written to the found page. GC writing is triggered by a GC trigger in the page, that is, the page needs to be read first, then the data in the page is sorted and compressed, and then the data is written.

[0064] GC writes will affect the performance of this write, and since this scenario will inevitably lead to GC writes, another corresponding relationship heap of total available space and page numbers is not set up. Instead, it is chosen to directly traverse the first array to save the resource usage of this small scenario optimization. However, in order to avoid the worst case scenario, one or more second arrays int[n] are used to record the number of corresponding pages with corresponding total available space. For example, int

[32] represents the number of pages with a remaining size of 32 sectors, int

[31] represents the number of pages with a remaining size of 31 sectors, and so on. With this array int[n], it is possible to quickly determine whether there are pages of corresponding sizes in the container, avoiding the situation where, after traversal, it is found that there are no pages with suitable total available space.

[0065] Regarding memory usage, the following examples can illustrate that the memory usage of the method of the present invention is equivalent to the memory usage of the bitmap algorithm. Assuming that the container size is 32GB and the sector size is 4KB, the memory space occupied by the bitmap algorithm is 32GB / 4KB / 8 = 1MB. Assuming that the page size is 128KB, the number of pages of a container in the present invention is 262144, so that the page number can be represented by 18 bits, the number of sectors of a page is 32, and the tail available space and total available space of each page are represented by 6 bits respectively. The space usage of the persistent available space metadata memory 211 is 2 * 6 * 262144 / 8≈384KB, and the space usage of the tail available space metadata sorting heap 212 is (6 + 18) * 262144 / 8≈768KB. An int

[33] array occupies 4 * 33 = 132B, and the three memory areas add up to about 1MB, which is equivalent to the memory space of the bitmap algorithm. Furthermore, in the bitmap mode, 1MB must be persisted to the disk, whereas in the above example of the present invention, only 384KB needs to be persisted.

[0066] The present invention introduces an array and a heap structure, effectively distinguishes the writable space and the used space in a page, and optimizes the efficiency of disk space allocation and release.

[0067] According to some embodiments of the present application, the method also includes: recording a write-before log after writing the data, the write-before log including the tail available space metadata and the total available space metadata of each of the pages after the data is written; updating the tail available space metadata and the total available space metadata in the first array and the bit array simulation heap.

[0068] According to some embodiments of the present application, the method further includes: persistently storing a copy of the first array at predetermined time intervals, and deleting the write-before log before the persistent storage.

[0069] According to some embodiments of the present application, the method also includes: in the event that the first array and the bit array simulation heap are lost, restoring the first array and the bit array simulation heap based on a copy of the first array stored persistently and in chronological order based on all undeleted write-ahead logs.

[0070] According to some embodiments of the present application, the method further includes: recording an index of the corresponding data after writing the data, where the index is a correspondence between a key of the data and a serial number of a page.

[0071] According to some embodiments of the present application, a page includes a plurality of sectors, and a sector stores its own sector metadata and metadata of stored data, and the metadata of the data includes a key of the data and a length of the data;

[0072] When reading data, the page where the data requested to be read is located is searched according to the index, and the data requested to be read is parsed in the page according to the sector metadata and the metadata of the data.

[0073] According to some embodiments of the present application, the method further includes: in response to data deletion, deleting the index and updating the tail free space metadata and the total free space metadata in the first array and the bit array simulation heap.

[0074] According to some embodiments of the present application, the method further includes: when GC is written, determining whether there is an index for the data to be sorted according to the key of the data to be sorted, and if there is an index, not overwriting the sector where the data to be sorted is located.

[0075] According to some embodiments of the present application, the sector metadata includes a version number of the sector metadata, a sector sequence number of a previous version, and a sector sequence number of a latest version; the method also includes: when there are two versions of sector metadata in a page, determining the sector order to parse data based on the latest version sector sequence number of the current version of the sector metadata and the latest version sector sequence number of the previous version of the sector metadata.

[0076] According to some embodiments of the present application, the data is Blob data, and each sector stores one or more Blob data.

[0077] According to some embodiments of the present application, the method further includes: adding a write lock to the page to which data is being written; and releasing the write lock after updating the tail free space metadata and the total free space metadata in the first array and the bit array simulation heap.

[0078] As an example, Figure 4 A storage system 40 is shown, comprising Figure 2 The following modules are added to the page free space allocator 21 and disk 22 of the storage system 20 shown: a metadata timing persister (free space metadatatiming persister) 41, a page metadata parser (page metadata parser) 42, a page GC trigger (pageGC trigger) 43, a write-before log system 44 and a data index system 45.

[0079] Based on the storage system 40, the data storage method of the present invention further includes:

[0080] S108 , when no page with suitable tail free space is found, write the data GC (Garbage Collection) into a page with suitable total free space among the multiple pages.

[0081] When a page with suitable tail free space is found, the serial number of the page with suitable tail free space is first detected through the page free space allocator 21, and then the write-ahead log system 44 records the write-ahead log. After the record is successful, the tail free space metadata and the total free space metadata corresponding to the serial number of this page in the page free space allocator 21 are modified, as well as the position of the tail free space metadata sorting heap 212. Then, the data is written to the underlying disk 22. After the write is successful, the data index is recorded through the data index system 45, and the data write is completed.

[0082] The write-before log system 44 records a modified value in the write-before log every time the storage system 40 modifies the tail available space metadata and the total available space metadata corresponding to the page, ensuring that all available space-related metadata can be restored after restart. The data index system 45 mainly stores user data indexes in the form of keys and values, and records here the correspondence between a Blob data key, the container serial number, and the page serial number. The remaining modules are introduced below.

[0083] 1) Metadata timed persistence device 41

[0084] In the present application, there is an area on the disk 22 that completely corresponds to the persistent available space metadata memory 211 area in the memory, and the persistent available space metadata memory 211 in the memory will be flushed to the disk once every predetermined time (for example, a few minutes). In order to prevent the downtime during the flush interval and cause the loss of available space metadata during the interval, the present invention relies on the write-before log system 44 to ensure data security. Before modifying the metadata of the available space of the page in the page available space allocator 21 each time, it is necessary to write the write-before log through the write-before log system 44, and wait for the write-before log to be persisted successfully before modifying the memory value in the page available space allocator 21. The loss of metadata of available space will cause serious impact, because the tail write is directly offset written in the page according to the tail available space metadata. For example, in a page with 2 sectors of tail available space, a 4096KB Blob is written, and the page offset plus the tail sector offset are directly used to write 4096KB. Therefore, once the modification of the tail available space metadata is lost, the subsequent tail write will inevitably cause data loss. Each time the metadata flush of the available space succeeds, the write-ahead log system 44 is notified to eliminate all write-ahead logs before the flush.

[0085] When the metadata of the available space is lost or the system is restarted, the persistent available space metadata memory 211 in the disk 22 (or the backup) is read first to restore the persistent available space metadata memory 211 area in the memory, and then all the write-ahead logs that have not been eliminated are redone in chronological order. After all the redo is completed, the int array and the tail available space metadata sort heap 212 can be constructed. This operation can ensure that the metadata data of the available space in the memory after the restart is completely consistent with the data before the loss or restart.

[0086] 2) Page metadata parser 42

[0087] The above description mentioned that the data index system only saves the correspondence between the Blob key, container number and page number. In order to facilitate the identification and parsing of these data when there are multiple Blobs in a page, in this application, each sector stores several bytes of sector metadata, and each Blob also stores certain Blob metadata, which includes the index key of the Blob, the length of the Blob and other information. After each page is read, the Blob data will be parsed based on this metadata information. Figure 5 A sample layout 501 of a page is shown.

[0088] Since the deletion of a Blob in response to a request for data deletion only deletes the index of the Blob and modifies the corresponding tail free space metadata and total free space metadata, the deleted sector will only be overwritten when GC is written next time, so it is common for dirty data to exist in the page. In order to ensure that when reading data, the deleted Blob is not returned to the upstream subsystem in the read data, in this application, an index key of the Blob is stored in the Blob metadata mentioned above. In this way, in scenarios where dirty reads are unacceptable (for example, in-page GC), after reading the data, the corresponding data index can be checked by the index key to quickly determine whether the Blob has been deleted.

[0089] 3) GC in page trigger 43 (GC in page)

[0090] Normally, when writing data, only the tail free space of the page can be considered, because a new data block can be directly appended to the page without reading it first. However, with the occurrence of continuous deletion, a scenario may occur in which the blob at the head of the page is deleted, while the tail blob has not been deleted, which will cause the total free space in the page to be greater than the tail free space. After this situation occurs in large quantities, it will be impossible to find a page with enough suitable tail free space in the container, and it is necessary to switch to using the total free space. In this case, the in-page GC trigger 43 will be triggered to perform GC processing to sort the page once to free up space, which will introduce the following series of problems.

[0091] If all blobs are the same size, it means that the size of the sectors that have been released in the front of the page must be an integer multiple of the size of the blobs, and these released sectors can be used directly to store blobs. However, this is a very ideal situation. In real scenarios, blobs can be any size smaller than a page. Therefore, it is necessary to organize the fragmented sectors in the page to form sectors with continuous available space before new data can be written. This is a typical GC process that does not consider power failure recovery at all. Such GC processing in a page is as follows: Figure 6 As shown, a page 601 before GC processing and a page 602 after GC processing are shown.

[0092] Now consider the case of a power failure. Only single-sector writes to the disk can guarantee atomicity, so in the GC processing scenario, all writes are entire page writes, which span multiple sectors. If the system crashes before successfully writing the entire page, the data in the page may be corrupted. What's worse, the existence of the underlying IO scheduling algorithm does not guarantee that writes to different sectors may be performed in order. Figure 7In the example shown, a failure occurred during GC processing. Originally, we hoped to perform defragmentation and compression, but Blob2 was damaged and could not be recovered (although we have multiple copies in the distributed system, we cannot rely on them because they also face the same problem), resulting in page 601 becoming page 701, in which Blob3 is lost.

[0093] It can be seen that in this example, the main problem is that Blob 3 may be overwritten by Blob 4. Therefore, in order to avoid data loss, the first principle of GC in page is obtained, that is, any sector that has data and has not failed cannot be overwritten. At the same time, the sector order before GC cannot be lost and the sector order cannot be adjusted arbitrarily. Because in the case where a Blob spans multiple sectors, once the following sectors are moved forward, the data of the Blob spanning multiple sectors will be lost, and the correct order and data can no longer be restored. Therefore, the second principle of GC in page is obtained, that is, the sector order before GC cannot be lost.

[0094] In order to solve the above problems and meet the two principles of GC in page. In this application, three pieces of information are recorded in the sector metadata, including the version number of the sector metadata, the sector serial number of the previous version (previous round), and the latest (current) version sector serial number. And the key in the Blob metadata is constantly used to reversely check the data index to confirm whether the Blob is invalid to avoid overwriting sectors with valid data. In the case where there are two versions of sector metadata in the page, the sector order is determined according to the latest (current) version sector serial number of the current version of the sector metadata and the latest (current) version sector serial number of the previous version (previous round) of the sector metadata to parse the data. Fig. 8A The normal GC process is shown, in which page 801 is changed into page 802 after GC process. Figure 8B A situation where a GC process fails is shown, where page 801 is transformed into page 803 after GC process. Fig. 8A and Figure 8B The first digit in the brackets in the middle sector indicates the version number of the sector metadata, the second digit indicates the sector serial number of the previous version (previous round), and the third digit indicates the sector serial number of the latest (current) version. As can be seen from the above example, Figure 8B As shown, even if the GC process fails, no data is lost, and once two versions of the sector metadata are found, the sequence of the latest (current) version sector sequence number of the current version of the sector metadata and the latest (current) version sector sequence number of the previous version (previous round) of the sector metadata can be directly trusted to correctly parse all data.

[0095] In this application, when processing concurrent write requests, if the data of multiple blobs arrive at the same time, these blobs are compactly stored in the page, that is, the space of the same sector can store multiple blobs. This avoids the problem that even the smallest blobs will occupy the space of a sector, causing a relatively large waste of space. The above processing is performed regardless of normal tail write or in-page GC process.

[0096] According to some embodiments of the present application, a write lock is first added to the page each time data is written. The heap position of the detected page can be directly moved out of the simulated heap to add a write lock to the page. The io scheduling algorithm used for the underlying disk 22 is uncertain, and all reads of the system are read as a whole page. In order to ensure that reading or writing will not be affected by other write requests of the page, the page needs to be locked after allocation. Each page uses 1 bit to represent the lock. After the data is written and the pre-write log is successfully recorded, the tail available space metadata and the total available space metadata corresponding to this page in the page available space allocator 21 can be modified. The three memory areas of the persistent available space metadata memory 211, the tail available space metadata sorting heap 212 and the int array must be modified synchronously each time they are modified. After modifying the tail available space metadata and the total available space metadata of the page available space allocator 21, the corresponding relationship between the modified tail available space metadata and the sequence number of the page needs to be adjusted in the position in the tail available space metadata sorting heap 212 and the number in the int array. After completing these operations, the page can be unlocked.

[0097] Fig. 9 It is a hardware structure block diagram of an electronic device for implementing data storage according to an embodiment of the present application.

[0098] like Fig. 9 As shown, the electronic device 900 may include one or more processors 902, a system motherboard 908 connected to at least one of the processors 902, a system memory 904 connected to the system motherboard 908, a non-volatile memory (NVM) 906 connected to the system motherboard 908, and a network interface 910 connected to the system motherboard 908.

[0099] The processor 902 may include one or more single-core or multi-core processors. The processor 902 may include any combination of a general-purpose processor and a dedicated processor (e.g., a graphics processor, an application processor, a baseband processor, etc.). In an embodiment of the present invention, the processor 902 may be configured to execute a data storage method according to an embodiment of the present application.

[0100] In some embodiments, system board 908 may include any suitable interface controller to provide any suitable interface to at least one of processors 902 and / or any suitable device or component in communication with system board 908 .

[0101] In some embodiments, the system board 908 may include one or more memory controllers to provide an interface to the system memory 904. The system memory 904 may be used to load and store data and / or instructions. In some embodiments, the system memory 904 of the electronic device 900 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0102] NVM 906 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM 906 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (CompactDisc) drive, and a DVD (Digital Versatile Disc) drive.

[0103] NVM 906 may include a portion of storage resources installed on a device of electronic device 900, or it may be accessible by the device but not necessarily a portion of the device. For example, NVM 906 may be accessed over a network via network interface 910.

[0104] In particular, the system memory 904 and the NVM 906 may include: a temporary copy and a permanent copy of the instruction 920, respectively. The instruction 920 may include: an instruction that causes the electronic device 900 to implement the data storage method according to the embodiment of the present application when executed by at least one of the processors 902. In some embodiments, the instruction 920, hardware, firmware and / or software components thereof may be additionally / alternatively placed in the system motherboard 908, the network interface 910 and / or the processor 902.

[0105] The network interface 910 may include a transceiver for providing a radio interface for the electronic device 900, thereby communicating with any other suitable device (e.g., a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 910 may be integrated with other components of the electronic device 900. For example, the network interface 910 may be integrated with at least one of the processor 902, the system memory 904, the NVM 906, and a firmware device (not shown) having instructions, and when at least one of the processors 902 executes the instructions, the electronic device 900 implements the data storage method according to the embodiment of the present application.

[0106] The network interface 910 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 910 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0107] In one embodiment, at least one of the processors 902 may be packaged together with one or more controllers for the system motherboard 908 to form a system in package (SiP). In one embodiment, at least one of the processors 902 may be integrated on the same die with one or more controllers for the system motherboard 908 to form a system on chip (SoC).

[0108] The electronic device 900 may further include an input / output (I / O) device 912 connected to the system motherboard 908. The I / O device 912 may include a user interface to enable a user to interact with the electronic device 900; the design of the peripheral component interface enables the peripheral components to interact with the electronic device 900. In some embodiments, the electronic device 900 further includes a sensor for determining at least one of an environmental condition and location information related to the electronic device 900.

[0109] In some embodiments, I / O device 912 may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), speakers, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0110] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0111] In some embodiments, the sensors may include, but are not limited to, gyroscope sensors, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning unit may also be part of or interact with the network interface 910 to communicate with components of a positioning network (e.g., global positioning system (GPS) satellites).

[0112] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 900. In other embodiments of the present application, the electronic device 900 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0113] Program code may be applied to input instructions to perform the functions described in the present invention and generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this application, a system for processing instructions including processor 902 includes any system having a processor such as a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0114] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in the present invention is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0115] One or more aspects of at least one embodiment may be implemented by instructions stored on a computer-readable storage medium, which, when read and executed by a processor, enables an electronic device to implement the method of the embodiment described in the present invention.

[0116] According to some embodiments of the present application, a computer storage medium is disclosed, on which instructions are stored. When the instructions are executed on a computer, the computer executes the data storage method according to the embodiments of the present application.

[0117] The method embodiment of the present application corresponds to this embodiment, and this embodiment can be implemented in conjunction with the method embodiment of the present application. The relevant technical details mentioned in the method embodiment of the present application are still valid in the present embodiment, and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment of the present application.

[0118] According to some embodiments of the present application, a computer program product is disclosed, including computer executable instructions, where the instructions are executed by a processor to implement the data storage method according to the embodiments of the present application.

[0119] The method embodiment of the present application corresponds to this embodiment, and this embodiment can be implemented in conjunction with the method embodiment of the present application. The relevant technical details mentioned in the method embodiment of the present application are still valid in the present embodiment, and are not repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment of the present application.

[0120] It is to be understood that the specific embodiments described herein are only for explaining the present application, rather than limiting the present application. In addition, for ease of description, only some but not all structures or processes related to the present application are shown in the accompanying drawings. It should be noted that in this specification, similar reference numerals and letters represent similar items in the accompanying drawings.

[0121] It should be understood that although the terms "first", "second", etc. may be used herein to describe various features, these features should not be limited by these terms. These terms are used only to distinguish and should not be understood as indicating or implying relative importance. For example, a first feature may be referred to as a second feature, and similarly a second feature may be referred to as a first feature without departing from the scope of the exemplary embodiments.

[0122] In the description of this application, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in this embodiment can be understood according to specific circumstances.

[0123] Illustrative embodiments of the present application include, but are not limited to, data storage methods, apparatus, media, and computer program products.

[0124] The various aspects of the illustrative embodiments will be described using terms commonly used by those skilled in the art to convey the essence of their work to other persons skilled in the art. However, it is apparent to those skilled in the art that some alternative embodiments may be implemented using the features described in part. For purposes of explanation, specific numbers and configurations are set forth to provide a more thorough understanding of the illustrative embodiments. However, it is apparent to those skilled in the art that alternative embodiments may be implemented without the specific details. In some other cases, some well-known features are omitted or simplified herein to avoid obscuring the illustrative embodiments of the present application.

[0125] In addition, various operations will be described as multiple separate operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be interpreted as implying that these operations must rely on the order of description, and many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can also be rearranged. When the described operations are completed, the process can be terminated, but can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0126] References in the specification to "one embodiment," "an embodiment," "an illustrative embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or property, but each embodiment may or may not necessarily include the particular feature, structure, or property. Moreover, these phrases are not necessarily referring to the same embodiment. In addition, when particular features are described in conjunction with a specific embodiment, the knowledge of those skilled in the art can affect the combination of these features with other embodiments, whether or not these embodiments are explicitly described.

[0127] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A and / or B" means "(A), (B), or (A and B)."

[0128] As used herein, the term "module" may refer to, be a part of, or include: a memory (shared, dedicated, or group) for running one or more software or firmware programs, an application-specific integrated circuit (ASIC), an electronic circuit and / or processor (shared, dedicated, or group), a combinational logic circuit, and / or other suitable components that provide the described functionality.

[0129] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not necessary. Instead, in some embodiments, these features may be described in a manner and / or order different from that shown in the illustrative drawings. In addition, the structural or method features included in a specific figure do not mean that all embodiments need to include such features. In some embodiments, these features may not be included or may be combined with other features.

[0130] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented in the form of instructions or programs carried or stored on one or more transient or non-transient machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors, etc. When the instructions or programs are executed by a machine, the machine may perform the various methods described above. For example, the instructions may be distributed via a network or other computer-readable medium. Therefore, machine-readable media may include, but are not limited to, any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, such as floppy disks, optical disks, compact disk read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electronically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, or flash memory or tangible machine-readable memories for transmitting network information via electrical, optical, acoustic, or other forms of signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, machine-readable media include any form of machine-readable media suitable for storing or transmitting electronic instructions or machine (e.g., computer) readable information.

[0131] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the use of the technical solution of the present application is not limited to the various applications mentioned in the embodiments of the present application. Various structures and modifications can be easily implemented with reference to the technical solution of the present application to achieve the various beneficial effects mentioned herein. Various changes made within the knowledge of ordinary technicians in the field without departing from the purpose of the present application should all fall within the scope of the patent coverage of the present application.

Claims

1. A data storage method for an electronic device, characterized in that: The method comprises: A bit array simulating a heap that records a first array consisting of the tail free space metadata and the total free space metadata of each page in the persistent storage space, and a correspondence between the tail free space metadata of each page and the sequence number of each page; receiving a data write request, searching the bit array simulation heap for a page with tail free space, wherein the tail free space is not less than the size of the data requested to be written; When the page with the tail free space is found, writing the data to the tail of the found page; In the case that the page with the tail free space is not found, a page with the total free space is searched from the first array, and data is written after GC is performed on the page with the total free space.

2. The method according to claim 1, characterized in that: Also includes: After writing the data, a write-before log is recorded, wherein the write-before log includes the tail available space metadata and the total available space metadata of each page after the data is written; The tail free space metadata and the total free space metadata in the first array and the bit array simulation heap are updated.

3. The method according to claim 2, characterized in that Also includes: At predetermined intervals, a copy of the first array is persistently stored, and the write-ahead log before the persistent storage is deleted.

4. The method according to claim 2, characterized in that: Also includes: In the event that the first array and the bit array simulation heap are lost, the first array and the bit array simulation heap are restored based on a copy of the first array in persistent storage and based on all the non-deleted write-ahead logs in chronological order.

5. The method according to claim 1, characterized in that Also includes: After the data is written, the index of the corresponding data is recorded, where the index is the correspondence between the key of the data and the serial number of the page.

6. The method according to claim 5, characterized in that The page includes a plurality of sectors, and the sector stores its own sector metadata and metadata of the stored data, and the metadata of the data includes a key of the data and a length of the data; When reading data, the page where the data requested to be read is located is searched according to the index, and the data requested to be read is parsed in the page according to the sector metadata and the metadata of the data.

7. The method according to claim 6, characterized in that Also includes: In response to data deletion, the index is deleted and the tail free space metadata and the total free space metadata in the first array and the bit array simulation heap are updated.

8. The method according to claim 7, characterized in that Also includes: When the GC is writing, it is determined whether the index exists for the data to be sorted according to the key of the data to be sorted. If the index exists, the sector where the data to be sorted is located is not overwritten.

9. The method according to claim 6, characterized in that The sector metadata includes a version number of the sector metadata, a sector serial number of a previous version, and a sector serial number of a latest version; the method further includes: In the case where there are two versions of sector metadata in the page, the sector order is determined according to the latest version sector sequence number of the current version of the sector metadata and the latest version sector sequence number of the previous version of the sector metadata to parse data.

10. The method according to claim 6, characterized in that The data is Blob data, and each sector stores one or more Blob data.

11. The method according to claim 2, characterized in that Also includes: Adding a write lock to the page to which data is being written; After updating the tail available space metadata and the total available space metadata in the first array and the bit array simulation heap, the write lock is released.

12. The method according to claim 1, characterized in that Also includes: Using one or more second arrays to record the number of corresponding pages having corresponding total available space; When the GC writes, it is determined whether the page with total available space exists according to the second array.

13. An electronic device, characterized in that: The electronic device comprises a memory storing computer executable instructions and a processor. When the instructions are executed by the processor, the electronic device implements the data storage method according to any one of claims 1 to 12.

14. A computer storage medium, characterized in that: Instructions are stored on the computer storage medium. When the instructions are executed on a computer, the computer is enabled to execute the data storage method according to any one of claims 1 to 12.

15. A computer program product, characterized in that The method comprises computer executable instructions, wherein the instructions are executed by a processor to implement the data storage method according to any one of claims 1 to 12.

Citation Information

Patent Citations

  • Data consistency and wear leveling-based memory file system management method

    CN106250320A

  • Data persistence processing method and system and computer equipment

    CN114385423A

  • Metadata management in non-volatile memory devices using in-memory logs

    CN115705152A

  • Index storage method and system and storage medium

    CN116186032A

  • Durable memory file system wear sensing space allocation method based on multi-level bitmap

    CN116360698A