Stand-alone storage engine and data processing method and device

Abstract

The embodiment of the invention provides a stand-alone storage engine and a data processing method and device.The processing method is applied to the stand-alone storage engine and comprises the steps that a bare disk is divided into a plurality of data blocks according to the preset byte number; a storage space is allocated for each data block according to object data to be stored, each piece of object data is written into the corresponding storage space, and each data block adopts a bitmap to record the storage space; and constructing a dictionary tree according to the name information of each object data and the position information corresponding to the data block bitmap, and querying and / or reading the object data according to the dictionary tree. According to the embodiment of the invention, the data is written into the bare disk, the data writing efficiency is improved, and meanwhile, full-memory indexing is realized, so that the query and traversal capabilities can be enhanced, and the data reading efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a stand-alone storage engine and a data processing method and device. Background Art

[0002] In the current single-machine storage engine, metadata is generally stored in a kv database (Key-Value Database), and data is generally stored in a local file system. When faced with high-performance storage requirements, the kv database will become the bottleneck of the entire engine, and the local file system will also generate multiple disk IOs (disk input and output) due to writing metadata and wal (Write-Ahead Logging), resulting in performance waste. Summary of the invention

[0003] In view of the above problems, a single-machine storage engine and a data processing method and device are proposed to overcome the above problems or at least partially solve the above problems, including:

[0004] A stand-alone storage engine, comprising a raw disk space management allocation engine and a metadata engine, wherein:

[0005] The raw disk space management and allocation engine is used to divide the raw disk into multiple data blocks according to a preset number of bytes, allocate storage space to each data block according to the object data to be stored, and write each object data into the corresponding storage space, wherein each data block uses a bitmap to record the storage space;

[0006] The metadata engine is used to construct a dictionary tree in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree.

[0007] Optionally, when the raw disk space management allocation engine is used to allocate storage space for each data block according to the object data to be stored, it is specifically used to:

[0008] Each data block is divided into a plurality of write units, and a plurality of cursors are used to allocate storage space for the object data to be stored on the bitmap of the data block, wherein each cursor allocates a preset number of write units according to the data amount of the object data to be stored.

[0009] Optionally, when the metadata engine is used to construct a dictionary tree in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, it is specifically used to:

[0010] The metadata engine is used to split the name information of each object data according to characters, and construct a dictionary tree in sequence according to the characters obtained by the split, and store the corresponding character values ​​in the non-leaf nodes of the dictionary tree, and store the corresponding character values ​​and the position information in the leaf nodes of the dictionary tree.

[0011] Optionally, the location information includes a data block identifier, an object data length, and an offset of the object data in the data block.

[0012] Optionally, the metadata engine is used to obtain target name information of the object data to be read, perform traversal query in the dictionary tree according to the target name information, and determine the target location information corresponding to the target name information;

[0013] The bare disk space management allocation engine is used to determine a target data block among multiple data blocks based on a data block identifier in the target position information, determine a starting reading position of the target object data based on an offset in the target position information, and read the target object data from the starting reading position according to the object data length in the target position information.

[0014] Optionally, the stand-alone storage engine further includes a snapshot module, which is used to store snapshot data generated based on the dictionary tree and bitmap at the current moment in the target disk after the snapshot is enabled, and record the snapshot time.

[0015] Optionally, the snapshot module is used to generate an operation log based on operations during data storage and dictionary tree construction, and after the snapshot is completed, delete the operation log before the snapshot time.

[0016] Optionally, when the stand-alone storage engine is restarted, the snapshot module loads the snapshot data in the target disk into the memory and plays back the operation log.

[0017] A data processing method, applied to a single-machine storage engine, comprises:

[0018] Split the raw disk into multiple data blocks according to the preset number of bytes;

[0019] Allocate storage space for each data block according to the object data to be stored, and write each object data into the corresponding storage space, wherein each data block uses a bitmap to record the storage space;

[0020] A dictionary tree is constructed in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree.

[0021] A data processing device, applied to a single-machine storage engine, comprising:

[0022] A hard disk segmentation module is used to segment a raw disk into multiple data blocks according to a preset number of bytes;

[0023] The object data writing module is used to allocate storage space for each data block according to the object data to be stored, and write each object data into the corresponding storage space, wherein each data block uses a bitmap to record the storage space;

[0024] The dictionary tree construction module is used to use the metadata engine to construct a dictionary tree in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree.

[0025] The embodiments of the present invention have the following advantages:

[0026] In an embodiment of the present invention, a stand-alone storage engine can divide a raw disk into multiple data blocks according to a preset number of bytes, and then allocate storage space for each data block according to the object data to be stored, and write each object data into the corresponding storage space, so that data can be directly written to the raw disk without going through the file system, thereby improving data writing efficiency. At the same time, each data block can use a bitmap to record the storage space; and then a dictionary tree can be constructed in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree, and realize full memory indexing, thereby enhancing the query and traversal capabilities and improving data reading efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0028] Figure 1a It is a structural diagram of a single-machine storage engine provided by an embodiment of the present invention;

[0029] Figure 1b This is a schematic diagram of a bare disk partition provided by an embodiment of the present invention;

[0030] Figure 1c It is a diagram of a dictionary tree provided by an embodiment of the present invention;

[0031] Figure 1d is a schematic diagram of a bitmap of a data block provided by an embodiment of the present invention;

[0032] Figure 1eIt is a schematic diagram of dual cursor storage space allocation provided by an embodiment of the present invention;

[0033] Figure 1f It is a schematic diagram of constructing a dictionary tree provided by an embodiment of the present invention;

[0034] Figure 1g It is a schematic diagram of a storage structure of a leaf node provided by an embodiment of the present invention;

[0035] Figure 2a It is a schematic diagram of the structure of another stand-alone storage engine provided by an embodiment of the present invention;

[0036] Figure 2b This is a schematic diagram of an operation log provided by an embodiment of the present invention;

[0037] Figure 3 is a flowchart of a data processing method provided by an embodiment of the present invention;

[0038] Figure 4 It is a schematic diagram of the structure of a data device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Reference Figure 1a , shows a schematic diagram of the structure of a stand-alone storage engine provided by an embodiment of the present invention, the stand-alone storage engine 100 includes a raw disk space management allocation engine 101 and a metadata engine 102. The stand-alone storage engine 100 is a storage system for receiving, storing, retrieving, and reading data. The stand-alone storage engine in the embodiment of the present invention may be an object storage engine for object data, and the object storage engine may organize data in a key-value store (Key-Value Store) manner, wherein each object has a unique key (Key), and the corresponding object data (Value) may be quickly retrieved through this key.

[0041] In an embodiment of the present invention, the raw disk space management and allocation engine 101 can be used to divide the raw disk into multiple data blocks according to a preset number of bytes, allocate storage space for each data block according to the object data to be stored, and write each object data into the corresponding storage space, wherein each data block uses a bitmap to record the storage space, and the bitmap can be used to record, manage, and allocate a segment of byte data in the space, wherein each bit represents a segment of space, and the preset number of bytes can be set according to the actual scenario requirements, and there are no excessive restrictions on this in the embodiment of the present invention.

[0042] A raw disk refers to a physical disk or logical volume that has not been formatted with a file system. A raw disk directly exposes the underlying storage device and does not contain a file system or other high-level abstraction layers. A raw disk can be used in application scenarios that require direct access to and control of the underlying storage, such as databases, virtualization, containerization, and high-performance storage systems. A raw disk has the following advantages:

[0043] (1) Direct access: Raw disks allow applications to directly access and operate the underlying storage device, bypassing the file system layer. This direct access method can improve performance, especially in scenarios that require high-performance I / O operations.

[0044] (2) No file system overhead: Since there is no file system, raw disks can avoid the overhead brought by the file system, such as metadata management, file system checking and repair, etc. This makes raw disks more advantageous in high-performance applications.

[0045] (3) Flexibility: Raw disks can be used for various advanced storage management technologies, such as LVM (Logical Volume Management), RAID (Disk Array), and storage virtualization. Applications can flexibly manage and organize storage space as needed.

[0046] (4) Security: Bare disks can be used for encrypted storage, using hardware or software encryption technology to protect data security. Since there is no file system, bare disks can avoid security risks caused by file system vulnerabilities.

[0047] In an embodiment of the present invention, the raw disk space management allocation engine 101 can store object data directly into the raw disk without passing through the file system. The writing process only needs to be written once. Compared with the file system that requires multiple disk IOs, the data writing path is short and the writing process is convenient, thereby improving the disk IO performance.

[0048] Reference Figure 1b, shows a schematic diagram of raw disk segmentation in an embodiment of the present invention; hard disk A, hard disk B, hard disk C and hard disk D are segmented according to a preset number of bytes of 64GiB (GiB is a unit of measurement for computer storage capacity, representing 2 to the 30th power of bytes), and hard disk A, hard disk B, hard disk C and hard disk D are raw disks, wherein hard disk A is segmented to obtain multiple data blocks: chunk0, chunk1, chunk2... The size of each data block is 64GiB.

[0049] In an embodiment of the present invention, when writing data into a data block, the remaining storage space of each data block may be determined, and then a data block with a large remaining storage space may be selected for storing the object data.

[0050] In an embodiment of the present invention, the metadata engine 102 can be used to construct a dictionary tree in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree. The correspondence between the name information and the position information of the object data can be stored in the dictionary tree, that is, the dictionary tree is used to manage metadata, and each node of the dictionary tree can store the node content (the node content can be generated based on the name information and the position information) and the pointer to the child node. In practical applications, the path formed from the root node in the dictionary tree to each leaf node can be used to represent the correspondence between the name information and the position information of the object data. Therefore, after the dictionary tree is constructed, when the user searches or reads a certain object data, the user can traverse and query the constructed dictionary tree based on the name information of the object data, and then determine the position information corresponding to the data object.

[0051] Among them, the name information can be composed of multiple characters, and the name information is the unique identification information of the object data. The location information is used to indicate the location where the object data is stored. Specifically, the location information may include the data block identifier, the object data length, and the offset of the object data in the data block.

[0052] The data block identifier is a unique identifier used to represent a data block. The object data length is the number of bytes or bits occupied by the object in memory. This length can be used to measure the size of the object. The object data length can be used to determine the object's memory usage, thereby performing memory management and optimization. The offset of the object data in the data block refers to the distance from the starting position of the object data in the data block to the starting position of the data block. The offset can be calculated in bytes.

[0053] Reference Figure 1c, shows a schematic diagram of a dictionary tree in an embodiment of the present invention, in which any path in the dictionary tree represents the corresponding relationship between the name information and the location information of an object data. The dictionary tree is used to store the location information of five object data whose object data names are abceg, abdfh, abdfi, tuvwy, tuvwz, and tuvxz.

[0054] In the embodiment of the present invention, the dictionary tree is stored in the memory (that is, the metadata is stored in the memory) to implement full memory indexing. Full memory indexing (In-Memory Indexing) is a technology that stores index data completely in the memory. Compared with the disk-based KV (Key-Value) database storage in the prior art, it has the following advantages:

[0055] (1) High performance:

[0056] Low latency: Memory access is much faster than disk access, and full memory indexing can significantly reduce the latency of data access. Memory access is usually at the nanosecond level, while disk access is usually at the millisecond level.

[0057] High throughput: Fully in-memory indexes can handle higher concurrent requests because memory bandwidth and processing power are much higher than disk.

[0058] (2) Real-time:

[0059] Real-time data access: Full memory indexing can provide real-time data access and query capabilities, suitable for application scenarios that require fast response, such as real-time analysis, real-time monitoring, and real-time trading systems.

[0060] Real-time update: Data in memory can be updated quickly, which is suitable for application scenarios that require frequent updates, such as real-time recommendation systems, real-time advertising, etc.

[0061] (3) Simplify the architecture

[0062] Reduce disk I / O: Full memory indexing avoids disk I / O operations, simplifies system architecture, and reduces the complexity and potential failure points caused by disk reading and writing.

[0063] Simplified data management: Data management in memory is relatively simple and does not require complex file system and disk management mechanisms.

[0064] (4) Flexibility:

[0065] Dynamic adjustment: In-memory indexes can be dynamically adjusted and optimized to adapt to different query patterns and load requirements.

[0066] Fast iteration: Data in memory can be quickly iterated and updated, which is suitable for application scenarios that require rapid development and deployment.

[0067] (5) High availability:

[0068] Memory redundancy: Through memory redundancy and replication technology, full memory indexing can provide high availability and fault tolerance, ensuring that data is still available when some nodes fail.

[0069] Fast recovery: Data in memory can be quickly restored, reducing system failure recovery time.

[0070] (6)Support complex queries:

[0071] Complex index structure: Full memory index can support complex index structures, such as B+ tree, hash table, skip table, etc., which is suitable for complex query and analysis needs.

[0072] Multidimensional index: The in-memory index can support multidimensional index, which is suitable for multidimensional data analysis and query.

[0073] (7) Applicable to specific scenarios:

[0074] Cache system: Full memory index is suitable for cache systems such as Redis, Memcached, etc., providing high-performance cache services.

[0075] Real-time analysis: Full memory indexing is suitable for real-time analysis systems such as Apache Druid and ClickHouse, providing real-time data analysis and query capabilities.

[0076] In summary, the full memory index in the embodiment of the present invention can improve disk IO performance.

[0077] In one embodiment of the present invention, when a bare disk space management allocation engine is used to allocate storage space for each data block according to object data to be stored, it is specifically used to: divide each data block into multiple write units, wherein a write unit is the smallest data storage unit in a data block; after the write units are divided, multiple cursors can be used to allocate storage space for the object data to be stored on a bitmap of the data block, wherein each cursor allocates a preset number of write units according to the amount of data of the object data to be stored, and the preset number of write units allocated by each cursor at a time can be the same or different.

[0078] When the data amounts of the object data to be stored differ by less than a preset value (e.g., the size of a write unit), the preset number of write units allocated by multiple cursors at a time is the same. When the data amounts of the object data to be stored differ by more than or equal to a preset value (e.g., the size of a write unit), the object data to be stored can be classified according to the data amounts of the object data to be stored, the data amounts of any two object data to be stored in the same object data class differ by less than a preset value, and the data amounts of any two object data to be stored in different object data classes differ by less than a preset value, and after the object data are classified, the preset number of object data of each class is set according to the data amount of the object data of that class.

[0079] In an embodiment of the present invention, multiple cursors can be used in the storage space allocation process. On the one hand, multiple cursors can improve the allocation efficiency of storage space, and on the other hand, storage space can be allocated at different scales. In addition, allocating object data according to data volume can make object data with similar data volume in the data block as close as possible to be clustered and allocated in one area, which is convenient for storage space management.

[0080] In the embodiment of the present invention, two cursors may be selected to simultaneously distribute data from the beginning and the end of the data block.

[0081] like Figure 1d The bitmap diagram of a data block in an embodiment of the present invention is shown. In the data block, the write unit is the smallest data storage unit, and each write unit is 4k. 0 and 1 displayed in the bitmap represent the allocation status of the write unit, 0 means that the write unit is not allocated, and 1 means that the write unit is allocated.

[0082] like Figure 1e The figure shows a schematic diagram of dual cursor storage space allocation in an embodiment of the present invention, wherein allocation pointer 1 and allocation pointer 2 are allocation pointers of two cursors, allocation pointer 1 allocates storage space from the head of the data block, and the storage space allocated each time is 3 write units. Allocation pointer 2 allocates storage space from the tail of the data block, and the storage space allocated each time is 3 write units.

[0083] In one embodiment of the present invention, when the metadata engine 102 is used to construct a dictionary tree in the memory according to the name information of each object data and the location information corresponding to the data block bitmap, it is specifically used to:

[0084] The metadata engine 102 splits the name information of each object data according to characters. For example, when the name of the object data 1 is abcd, it can be split into four characters: a, b, c, and d.

[0085] After completing the character splitting, the metadata engine 102 can construct a dictionary tree in sequence according to the split characters, such as root node→node corresponding to character a→node corresponding to character b→node corresponding to character c→node corresponding to character d.

[0086] In the embodiment of the present invention, when the dictionary tree inserts characters corresponding to new object data, the metadata engine 102 can merge or split the character values ​​to update the dictionary tree.

[0087] Specifically, characters with the same content can be merged to obtain new character nodes and the target dictionary tree.

[0088] For example, the name of object data 1 is abcd, and the name of object data 2 is abce, wherein the initial dictionary tree of object data 1 is: root node→node corresponding to character a→node corresponding to character b→node corresponding to character c→node corresponding to character d; the initial dictionary tree of object data 2 is: root node→node corresponding to character a→node corresponding to character b→node corresponding to character c→node corresponding to character e, wherein the three character nodes a, b, and c are the same in object data 1 and object data 2, thus, in the two initial dictionary trees of object data 1 and object data 2, the three character nodes a, b, and c are merged into nodes corresponding to character abc, thus, the generated target field tree contains two paths: root node→node corresponding to character abc→node corresponding to character d (this path represents object data 1), and root node→node corresponding to character abc→node corresponding to character e (this path represents object data 2).

[0089] In an embodiment of the present invention, when the newly inserted object data is split according to the characters in the original dictionary tree, if there is a split character that partially overlaps with an original node, the overlapping characters are determined, and after the original node is split based on the overlapping characters, the new object data is inserted.

[0090] Reference Figure 1f , showing a schematic diagram of constructing a dictionary tree in an embodiment of the present invention. The dictionary tree on the left contains the character node "tuv". After the object data tutwg is added, the character node "tuv" can be split again into the character node "tu" and the child node "v", and then the new object data is inserted based on the updated character node.

[0091] In the embodiment of the present invention, the metadata engine 102 can store data for each node on the dictionary tree after setting each node. In actual applications, the corresponding character value (i.e., key value) can be stored in the non-leaf node of the dictionary tree, and the corresponding character value (key value) and position information can be stored in the leaf node of the dictionary tree. This achieves that the metadata of each node can be managed with minimal resource consumption.

[0092] Reference Figure 1g, showing a schematic diagram of the storage structure of a leaf node; the leaf node stores internal information including 8 bytes, wherein the leaf node stores key information, the chunk (data block), the offset within the chunk, the object data length and other information, the key information (i.e., the character value) occupies 6 bits, and the maximum number is 64, that is, the key information can represent an integer between 0 and 63; the chunk identifier is 12 bits, and the maximum is 4096, that is, the chunk identifier can represent an integer between 0 and 4095; the object data length is 12 bits, and the maximum can represent 419304, that is, the object data length can represent an integer between 0 and 419304, the offset within the chunk is 24 bits, and the maximum offset is 64GiB, that is, the offset within the chunk can represent an integer between 0 and 64GiB.

[0093] In an embodiment of the present invention, the metadata engine 102 may be used to obtain target name information of the object data to be read, perform traversal query in the dictionary tree according to the target name information, and determine the target location information corresponding to the target name information.

[0094] In actual applications, the user can input the target name information of the object data to be read through the stand-alone storage engine. In the stand-alone storage engine, the metadata engine can obtain the target name information and then perform a traversal query according to the dictionary tree. Specifically, the traversal can be started from the first child node pointed to by the root node of the dictionary tree to determine whether the character value stored in the first child node is a character in the target name information (judged from the first character of the target name information). When the target first child node stores the first character value of the target name information, one or more candidate second child nodes connected to the target first child node are determined to determine whether the candidate second child node stores the character value connected to the first character value in the target name information. When the target second child node stores the second character value of the target name information (the second character value is the character connected to the first character value in the target name information), one or more candidate third child nodes connected to the target second child node are determined, and the above node traversal query is performed until the last character of the target name information is traversed. If the node corresponding to the last character is a non-leaf node, it is determined that the target position information corresponding to the target data node is not stored in the metadata engine. If the node corresponding to the last character is a leaf node, the target position information corresponding to the target name information is obtained from the leaf node.

[0095] It should be noted that if before the last character, no non-leaf node matching the target object data is traversed in the dictionary tree, or the traversed node is a leaf node, it is determined that the target location information corresponding to the target data node is not stored in the metadata engine.

[0096] The bare disk space management allocation engine 101 can be used to determine the target data block among multiple data blocks based on the data block identifier in the target position information, determine the starting reading position of the target object data based on the offset in the target position information, and read the target object data from the starting reading position according to the object data length in the target position information.

[0097] In actual applications, the raw disk space management allocation engine 101 can read data in the raw disk according to the target location information traversed by the metadata engine 102. Specifically, the target data block is located according to the data block identifier, and the starting reading position in the target data block is located according to the offset in the target location information, and then the data of the corresponding length is read according to the object data length to realize the target object data reading.

[0098] In an embodiment of the present invention, a stand-alone storage engine is provided, which may include a raw disk space management allocation engine and a metadata engine, thereby realizing object data storage on a raw disk and metadata storage in a memory through the raw disk space management allocation engine and the metadata engine. The stand-alone storage engine has an extremely short IO read and write path, improves the read and write performance, and realizes full memory indexing, thereby enhancing query and traversal capabilities.

[0099] Reference Figure 2a , shows a schematic diagram of the structure of a stand-alone storage engine provided by an embodiment of the present invention, the stand-alone storage engine 100 includes a raw disk space management allocation engine 101, a metadata engine 102 and a snapshot module 103. The snapshot module 103 can be used to store snapshot data generated based on the dictionary tree and bitmap at the current moment in the target disk after the snapshot is enabled, and record the snapshot time.

[0100] In computer science and data storage, a snapshot is a complete record or copy of the state of data at a specific point in time. Snapshots can be used for backup, recovery, data consistency checks, and other scenarios.

[0101] In one embodiment of the present invention, the snapshot module is used to generate operation logs based on operations in the data storage and dictionary tree construction process, and after the snapshot is completed, the operation logs before the snapshot time are deleted.

[0102] In actual applications, each operation performed by a user in a stand-alone storage engine can generate an operation log, which is used to record the user's operation. The operation log may be composed of operation time information and direct operation information corresponding to the operation time information.

[0103] When a large amount of data needs to be read and written, the corresponding operation log data volume is huge, and as the single-machine storage engine is used over time, a large number of operation logs will also accumulate, which will occupy a large amount of memory, and may affect the read and write efficiency. Every time a snapshot is generated, the dictionary tree and bitmap at the snapshot time are recorded, so the storage volume of the operation log can be reduced by deleting the operation log before the snapshot time. In addition, restoring the dictionary tree and bitmap based on snapshots and operation logs is more convenient than restoring data completely through operation logs.

[0104] Reference Figure 2b , shows a schematic diagram of an operation log in the implementation of the present invention, which may specifically include the following operation records:

[0105] "time1insert key1:chunk,offset,length

[0106] time2insert key2:chunk,offset,length

[0107] time3insert key3:chunk,offset,length

[0108] time4insert key4:chunk,offset,length

[0109] time5insert key5:chunk,offset,length

[0110] time6insert key6:chunk,offset,length

[0111] time7insert key7:chunk,offset,length

[0112] time8delete key1:chunk,offset,length

[0113] time9delete key3:chunk,offset,length”

[0114] That is, at time 1, key1 is inserted, the data block is stored, and the offset and length are set; at time 2, key2 is inserted, the data block is stored, and the offset and length are set; at time 3, key3 is inserted, the data block is stored, and the offset and length are set; at time 4, key4 is inserted, the data block is stored, and the offset and length are set; at time 5, key5 is inserted, the data block is stored, and the offset and length are set; at time 6, key6 is inserted, the data block is stored, and the offset and length are set; at time 7, key7 is inserted, the data block is stored, and the offset and length are set; at time 8, the data block, offset and length of key1 are deleted; at time 9, the data block, offset and length of key3 are deleted.

[0115] When the snapshot start time is time5, the operation records corresponding to time1 to time4 can be deleted.

[0116] In one embodiment of the present invention, when the stand-alone storage engine is restarted, the snapshot module loads the snapshot data in the target disk into the memory and plays back the operation log.

[0117] In actual applications, when a stand-alone storage engine is restarted, the snapshot data in the target disk is first loaded into the memory, and then the log records are replayed. Since the log records record object operations, there are only two states for the key of an object: existence or non-existence, which can ensure idempotence. Therefore, even if the key already exists when the record is restarted and replayed, it will not affect the execution again.

[0118] In an embodiment of the present invention, a stand-alone storage engine is provided, which can not only improve data reading and writing performance through a raw disk space management allocation engine and a metadata engine, but also avoid data loss through a snapshot module set in the stand-alone storage engine, thereby enhancing the stability of the stand-alone storage engine.

[0119] Reference Figure 3 , shows a flow chart of the steps of a data processing method provided by an embodiment of the present invention, which is applied to a single-machine storage engine and may specifically include the following steps:

[0120] Step S301, dividing the raw disk into multiple data blocks according to a preset number of bytes;

[0121] In actual applications, a single-machine storage engine has the function of managing raw disk data. In the embodiment of the present invention, in order to improve data writing and reading performance, a raw disk can be used to directly store data objects. Before writing to a raw disk, the raw disk used for storage can be segmented to obtain multiple data blocks with a preset number of bytes. The preset number of bytes can be set according to actual scene requirements, and there is no excessive restriction on this in the embodiment of the present invention.

[0122] A single-machine storage engine may include a raw disk space management allocation engine, which may be used to implement raw disk space allocation, and further, the single-machine storage engine may use the raw disk space management allocation engine to divide the raw disk into multiple data blocks according to a preset number of bytes.

[0123] Step S302, allocating storage space for each data block according to the object data to be stored, and writing each object data into the corresponding storage space, wherein each data block uses a bitmap to record the storage space;

[0124] After the raw disk is divided into data blocks, the raw disk space management allocation engine can be used to allocate storage space on the data blocks according to the object data to be stored, and then each object data is written into the corresponding storage space. The storage space can be recorded in the data block in a bitmap manner.

[0125] Step S303: construct a dictionary tree in the memory according to the name information of each object data and the location information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree.

[0126] In practical applications, each object data has unique name information, and after the object data is stored in the data block, the corresponding position information of the data object on the data block bitmap can be determined. Then, a dictionary tree can be constructed according to the name information and position information, and the dictionary tree stores the corresponding relationship between the name information and the position information of each data object. Thus, the dictionary tree is equivalent to an index, and the location information of the object data stored in the hard disk can be quickly queried through the dictionary tree, and then the object data can be read from the raw disk according to the position information.

[0127] A metadata engine may also be included in a stand-alone storage engine, so that a dictionary tree may be constructed through the metadata engine.

[0128] In the embodiment of the present invention, object data is directly stored in the raw disk without passing through the file system. The writing process only needs to be written once. Compared with the file system that requires multiple disk IOs, the data writing path is short and the writing process is convenient, thereby improving the data writing performance. In addition, metadata is stored in memory to achieve full memory indexing, which improves the disk reading performance.

[0129] In one embodiment of the present invention, the process of allocating storage space for each data block according to the object data to be stored in step S302 is specifically: dividing each data block into multiple write units, and using multiple cursors to allocate storage space for the object data to be stored on the bitmap of the data block, wherein each cursor allocates a preset number of write units according to the data amount of the object data to be stored.

[0130] In one embodiment of the present invention, in step 303, the process of constructing a dictionary tree in the memory according to the name information of each object data and the location information corresponding to the data block bitmap is as follows:

[0131] The name information of each object data is split according to characters, and a dictionary tree is constructed in sequence according to the split characters, and the corresponding character values ​​are stored in the non-leaf nodes of the dictionary tree, and the corresponding character values ​​and position information are stored in the leaf nodes of the dictionary tree.

[0132] The location information may include a data block identifier, an object data length, and an offset of the object data in the data block.

[0133] Furthermore, in an embodiment of the present invention, a stand-alone storage engine can realize data reading. Specifically, a user can input the name of the target object data to be read, so that the metadata engine in the stand-alone storage engine can obtain the target name information of the object data to be read, and perform traversal query in the dictionary tree according to the target name information to determine the target position information corresponding to the target name information; then use the position information to determine the data block identifier, and determine the target data block among multiple data blocks, determine the starting reading position of the target object data based on the offset in the target position information, and read the target object data from the starting reading position according to the object data length in the target position information.

[0134] In one embodiment of the present invention, the stand-alone storage engine also includes a snapshot module, and the data processing method also includes storing snapshot data generated based on the dictionary tree and bitmap at the current moment in the target disk after the snapshot is enabled, and recording the snapshot time.

[0135] In one embodiment of the present invention, the snapshot module in the stand-alone storage engine can generate operation logs based on operations during data storage and dictionary tree construction, so that after the snapshot is completed, the operation logs before the snapshot time can be deleted.

[0136] In one embodiment of the present invention, when the stand-alone storage engine is restarted, the snapshot data in the target disk is loaded into the memory, and the operation log is played back.

[0137] In an embodiment of the present invention, a stand-alone storage engine can divide a raw disk into multiple data blocks according to a preset number of bytes, and then allocate storage space for each data block according to the object data to be stored, and write each object data into the corresponding storage space, so that data can be directly written to the raw disk without going through the file system, thereby improving data writing efficiency. At the same time, each data block can use a bitmap to record the storage space; and then a dictionary tree can be constructed in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree, and realize full memory indexing, thereby enhancing the query and traversal capabilities and improving data reading efficiency.

[0138] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0139] Reference Figure 4 , shows a schematic diagram of the structure of a data processing device provided by an embodiment of the present invention, which is applied to a stand-alone storage engine and may specifically include the following modules:

[0140] The hard disk segmentation module 401 is used to segment the raw disk into multiple data blocks according to a preset number of bytes;

[0141] The object data writing module 402 is used to allocate storage space for each data block according to the object data to be stored, and write each object data into the corresponding storage space, wherein each data block uses a bitmap to record the storage space;

[0142] The dictionary tree construction module 403 is used to use the metadata engine to construct a dictionary tree in the memory according to the name information of each object data and the location information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree.

[0143] In an embodiment of the present invention, the object data writing module 402 may include the following submodules:

[0144] A write unit division submodule, used to divide each data block into multiple write units;

[0145] The cursor allocation submodule is used to allocate storage space for the object data to be stored on the bitmap of the data block using multiple cursors, wherein each cursor allocates a preset number of write units according to the data volume of the object data to be stored.

[0146] In one embodiment of the present invention, the dictionary tree construction module 403 may include the following submodules:

[0147] A character splitting submodule is used to split the name information of each object data according to characters;

[0148] The dictionary tree creation submodule is used to construct the dictionary tree in sequence according to the characters obtained by splitting;

[0149] The node data storage submodule is used to store the corresponding character values ​​in the non-leaf nodes of the dictionary tree, and store the corresponding character values ​​and the position information in the leaf nodes of the dictionary tree.

[0150] In an embodiment of the present invention, the location information includes a data block identifier, an object data length, and an offset of the object data in the data block.

[0151] In one embodiment of the present invention, the device further comprises:

[0152] A target name information acquisition module is used to acquire the target name information of the object data to be read;

[0153] A dictionary tree query module, used for performing a traversal query in the dictionary tree according to the target name information, and determining the target location information corresponding to the target name information;

[0154] A target data block determination module, configured to determine a target data block from among a plurality of data blocks based on a data block identifier in the target location information;

[0155] A reading position determination module is used to determine the starting reading position of the target object data based on the offset in the target position information;

[0156] The target object data reading module is used to read the target object data from the start reading position according to the object data length in the target position information.

[0157] In one embodiment of the present invention, the device further comprises:

[0158] The snapshot data storage module is used to store the snapshot data generated based on the dictionary tree and bitmap at the current moment in the target disk after the snapshot is enabled, and record the snapshot time.

[0159] In one embodiment of the present invention, the device further comprises:

[0160] The operation log deletion module is used to generate operation logs based on operations in the data storage and dictionary tree construction process, and delete the operation logs before the snapshot time after the snapshot is completed.

[0161] In one embodiment of the present invention, the device comprises:

[0162] The data recovery module is used to load the snapshot data in the target disk into the memory and replay the operation log when the stand-alone storage engine is restarted.

[0163] In an embodiment of the present invention, a stand-alone storage engine can divide a raw disk into multiple data blocks according to a preset number of bytes, and then allocate storage space for each data block according to the object data to be stored, and write each object data into the corresponding storage space, so that data can be directly written to the raw disk without going through the file system, thereby improving data writing efficiency. At the same time, each data block can use a bitmap to record the storage space; and then a dictionary tree can be constructed in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree, thereby realizing full memory indexing, thereby enhancing query and traversal capabilities and improving data reading efficiency.

[0164] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the computer program implements the above data processing method when executed by the processor.

[0165] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above data processing method is implemented.

[0166] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0167] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0168] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0169] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0170] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0172] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0173] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0174] The above provides a detailed introduction to a stand-alone storage engine and a data processing method and device. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A single-machine storage engine, characterized in that: The stand-alone storage engine includes a raw disk space management allocation engine and a metadata engine, wherein: The raw disk space management and allocation engine is used to divide the raw disk into multiple data blocks according to a preset number of bytes, allocate storage space to each data block according to the object data to be stored, and write each object data into the corresponding storage space, wherein each data block uses a bitmap to record the storage space; The metadata engine is used to construct a dictionary tree in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree.

2. The stand-alone storage engine according to claim 1, characterized in that: When the raw disk space management allocation engine is used to allocate storage space for each data block according to the object data to be stored, it is specifically used to: Each data block is divided into a plurality of write units, and a plurality of cursors are used to allocate storage space for the object data to be stored on the bitmap of the data block, wherein each cursor allocates a preset number of write units according to the data amount of the object data to be stored.

3. The stand-alone storage engine according to claim 1, characterized in that: When the metadata engine is used to construct a dictionary tree in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, it is specifically used to: The metadata engine is used to split the name information of each object data according to characters, and construct a dictionary tree in sequence according to the characters obtained by the split, and store the corresponding character values ​​in the non-leaf nodes of the dictionary tree, and store the corresponding character values ​​and the position information in the leaf nodes of the dictionary tree.

4. The stand-alone storage engine according to claim 3, characterized in that: The location information includes a data block identifier, an object data length, and an offset of the object data in the data block.

5. The stand-alone storage engine according to claim 4, characterized in that: The metadata engine is used to obtain target name information of the object data to be read, perform traversal query in the dictionary tree according to the target name information, and determine the target location information corresponding to the target name information; The bare disk space management allocation engine is used to determine a target data block among multiple data blocks based on a data block identifier in the target position information, determine a starting reading position of the target object data based on an offset in the target position information, and read the target object data from the starting reading position according to the object data length in the target position information.

6. The single-machine storage engine according to claim 1, characterized in that: The stand-alone storage engine also includes a snapshot module, which is used to store snapshot data generated based on the dictionary tree and bitmap at the current moment in the target disk after the snapshot is enabled, and record the snapshot time.

7. The stand-alone storage engine according to claim 6, characterized in that: The snapshot module is used to generate operation logs based on operations in the data storage and dictionary tree construction process, and after the snapshot is completed, the operation logs before the snapshot time are deleted.

8. The stand-alone storage engine according to claim 7, characterized in that: When the stand-alone storage engine is restarted, the snapshot module loads the snapshot data in the target disk into the memory and plays back the operation log.

9. A data processing method, characterized in that: Applied to a single-machine storage engine, the method includes: Split the raw disk into multiple data blocks according to the preset number of bytes; Allocate storage space for each data block according to the object data to be stored, and write each object data into the corresponding storage space, wherein each data block uses a bitmap to record the storage space; A dictionary tree is constructed in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree.

10. A data processing device, characterized in that: Applied to a single-machine storage engine, the device comprises: A hard disk segmentation module is used to segment a raw disk into multiple data blocks according to a preset number of bytes; The object data writing module is used to allocate storage space for each data block according to the object data to be stored, and write each object data into the corresponding storage space, wherein each data block uses a bitmap to record the storage space; The dictionary tree construction module is used to use the metadata engine to construct a dictionary tree in the memory according to the name information of each object data and the position information corresponding to the data block bitmap, so as to query and / or read the object data according to the dictionary tree.

Images