Index structure construction method and data indexing method

Abstract

The invention provides an index structure construction method and a data indexing method.The index structure construction method comprises the steps that in response to an index structure construction request, composite data and the data type of the composite data are obtained; when the data type is a first type, the offset of elements of the composite data in a preset storage unit is obtained, an index structure is determined based on metadata of the composite data and the offset, and the first type comprises at least one of a hash structure and a set structure; when the data type is a second type, elements of the composite data are obtained, an index structure is determined based on metadata of the composite data and the elements, and the second type comprises at least one of an ordered set and a queue.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data indexing, and in particular to an index structure construction method and a data indexing method. BACKGROUND

[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the present application described in the claims. The description herein does not constitute admission of prior art.

[0003] In the prior art, a unified data organization method and index structure are usually used to process different types of composite data. Although this method simplifies system design, it is difficult to fully exploit the characteristics of various data structures. Especially in the context of low modification frequency data storage, how to construct a dedicated index mechanism for different types of data to support efficient field-level query, range query, sorting query, and queue operation, and other complex access modes is a problem that needs to be solved.

[0004] In addition, in the context of multi-version concurrency control, each key corresponds to a unique version, and the version is not reused once it is released. Although this guarantees data consistency, it also puts higher requirements on the design of the index: it must be able to accurately locate the valid data under the specified version and achieve fast retrieval without frequent modification of the index.

[0005] When processing multiple types of composite data (such as Hash, Set, Zset, and List), how to efficiently implement fast query and version isolation management based on static write files becomes a key challenge to improve the overall performance of the system.

[0006] Therefore, there is an urgent need to propose an index structure construction method and a data indexing method that can solve the above technical problems. SUMMARY

[0007] Aspects of the present application provide an index structure construction method and a data indexing method to solve the technical problems existing in the prior art.

[0008] In an aspect of the present application, an index structure construction method is provided, which comprises: In response to an index structure construction request, composite data and the data type of the composite data are obtained; When the data type is a first type, the offset of the elements of the composite data in a preset storage unit is obtained, and an index structure is determined based on the metadata of the composite data and the offset, the first type including at least one of a hash structure and a set structure; When the data type is the second type, the elements of the composite data are obtained, and the index structure is determined based on the metadata of the composite data and the elements. The second type includes at least one of ordered sets and queues.

[0009] Further, when the data type is a first type, obtaining the offset of the element of the composite data in a preset storage unit, and determining the index structure based on the metadata of the composite data and the offset, includes: when the data type is a first type, obtaining the version number and field of the composite data, and obtaining the first offset of the element of the field in the preset storage unit; determining the primary key based on the version number, the field, and the data type, and determining the target value based on the first offset; and determining the index structure based on the primary key and the target value.

[0010] Further, when the data type is a first type, obtaining the offset of the element of the composite data in a preset storage unit, and determining the index structure based on the metadata of the composite data and the offset, includes: when the data type is a first type, obtaining the version number of the composite data, and obtaining the second offset of the first element corresponding to the version number in the preset storage unit; determining the primary key based on the version number, determining the target value based on the second offset, wherein the second offset corresponds to a unique element; and determining the index structure based on the primary key and the target value.

[0011] Further, when the data type is the second type, obtaining the elements of the composite data and determining the index structure based on the metadata of the composite data and the elements includes: when the data type of the composite data is a queue type, obtaining the version number, data type, elements, and absolute position of the elements in a preset storage unit; determining the primary key based on the version number, the queue type, and the absolute position; determining the target value based on the elements; and determining the index structure based on the primary key and the target value.

[0012] Furthermore, when the data type is a second type, obtaining the elements of the composite data and determining the index structure based on the metadata of the composite data and the elements includes: when the data type of the composite data is an ordered set type, obtaining the version number, data type, fields, and elements of the composite data; determining the primary key based on the version number, the ordered set type, and the fields; determining the target value based on the elements; and determining the index structure based on the primary key and the target value.

[0013] Furthermore, when the data type is a second type, the step of obtaining the elements of the composite data and determining the index structure based on the metadata of the composite data and the elements includes: when the data type of the composite data is an ordered set type, obtaining the version number, data type, fields, and elements of the composite data; determining the first data based on the version number, the ordered set type, the fields, and the elements; and constructing an index structure by arranging the first data in order of the size of the elements.

[0014] In another aspect, this application provides a data indexing method, the method being implemented based on an index structure constructed using the index structure construction method described above, the method comprising: In response to a data indexing request, retrieve the data index range, the data type of the data to be indexed, and the primary key of the data to be indexed; The data to be indexed is indexed from a pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed.

[0015] Further, the step of indexing the data to be indexed from the pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed includes: when the data index range is a local index and the data type is a first type, obtaining the primary key and the fields and version number of the primary key, determining a first offset based on the fields and the version number, and indexing the data to be indexed based on the first offset; when the data index range is a global index and the data type is a first type, obtaining the primary key and the version number of the primary key, determining a second offset based on the version number, and indexing the data to be indexed based on the second offset.

[0016] Further, the step of indexing the data to be indexed from the pre-constructed index structure based on the data index range, the data type, and the primary key of the data to be indexed includes: when the data index range is a local index and the data type is a queue, obtaining the primary key and its absolute position, and determining the data to be indexed based on the absolute position; when the data index range is a global index and the data type is a queue, determining the index range value, obtaining the primary key and its absolute position, determining the absolute position within the index range value, and determining the data to be indexed based on the absolute position.

[0017] Furthermore, the step of indexing the data to be indexed from the pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed includes: when the data type is an ordered set, obtaining the primary key and the version number of the primary key; obtaining the position information of the element corresponding to the version number from the pre-built index structure based on the version number; and determining the data to be indexed based on the position information.

[0018] This application proposes an index structure construction method and a data indexing method. The index structure construction method and data indexing method proposed in this application adopt different index structure construction rules and data indexing rules for different types of composite data. This not only gives full play to the advantages of each data type, but also better meets the needs of diverse application scenarios. On the one hand, the index structure construction method proposed in this application has high flexibility and adaptability: it designs special index structures and methods for different types of data, and can select the most suitable data structure according to specific application requirements, thereby improving the overall performance and response speed of the system. On the other hand, it maximizes resource utilization: through targeted design, it can minimize unnecessary computation and storage overhead while ensuring functionality, thereby improving resource utilization. Furthermore, it has the beneficial effects of easy maintenance and scalability: specific data structures and indexing methods are often accompanied by clear operational logic, which helps to simplify the design and maintenance of the system and facilitates future functional expansion. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating an index structure construction method provided in an embodiment of this application; Figure 2 A flowchart illustrating a data indexing method provided in another embodiment of this application; Figure 3 A schematic diagram of the structure of an index structure construction device provided in another embodiment of this application; Figure 4 This is a schematic diagram of the structure of a data indexing device provided in another embodiment of the present application; Figure 5This is a schematic diagram of the structure of an electronic device suitable for implementing the solutions in the embodiments of this application; The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0023] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0024] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0025] This application provides an index structure construction method, which includes at least the following steps: Step S101: In response to the index structure construction request, obtain the composite data and the data type of the composite data; Step S102: When the data type is the first type, the offset of the element of the composite data in the preset storage unit is obtained, and the index structure is determined based on the metadata of the composite data and the offset. The first type includes at least one of hash structure and set structure. Step S103: When the data type is the second type, the elements of the composite data are obtained, and the index structure is determined based on the metadata of the composite data and the elements. The second type includes at least one of ordered set and queue.

[0026] In practical scenarios, the execution entity of this method can be a user device, or a device formed by integrating a user device and a network device through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0027] Example 1

[0028] like Figure 1 As shown in the figure, this embodiment proposes a processing flow for an index structure construction method, which includes at least the following processing steps: Step S101: In response to the index structure construction request, obtain the composite data and the data type of the composite data; Step S102: When the data type is the first type, the offset of the element of the composite data in the preset storage unit is obtained, and the index structure is determined based on the metadata of the composite data and the offset. The first type includes at least one of hash structure and set structure. Step S103: When the data type is the second type, the elements of the composite data are obtained, and the index structure is determined based on the metadata of the composite data and the elements. The second type includes at least one of ordered set and queue.

[0029] Specifically, composite data refers to complex data types composed of multiple basic data types or structures. They typically contain multiple fields, each capable of storing different types of data, and these data types are logically related. Composite data allows us to organize related information together for easier management and manipulation.

[0030] In one implementation, the metadata of all composite data is stored uniformly in a separate device called `metaData`. This device is primarily responsible for storing the metadata information corresponding to each composite data key, including key attributes such as the current key's version number and the number of elements. The version is the most crucial information; each key uniquely corresponds to a version, and once a key is deleted, its corresponding version is no longer reused. This means that newly written keys will be assigned new versions; even if the same key is rewritten, it will be assigned a new version. When processing any request, the system will first retrieve the corresponding version information by querying `metaData`, or create a new version for newly written data.

[0031] In addition, for List data structures, metaData also records the absolute starting position of the data corresponding to the key to support efficient index-based access. Each key can contain multiple fields, and each field corresponds to its own value. This actual data is stored in a device called metaData.

[0032] Metadata is written sequentially by key, ensuring that different fields within the same version are ordered, and that order is maintained between different versions. This storage structure is implemented using low-modification-frequency storage files, meaning that the file content remains unchanged after data is written; indexes are built synchronously during data writing; and the index structure remains fixed after writing is complete.

[0033] In one embodiment, when the data type is a first type, obtaining the offset of the element of the composite data in a preset storage unit, and determining the index structure based on the metadata of the composite data and the offset, includes: when the data type is a first type, obtaining the version number and field of the composite data, and obtaining the first offset of the element of the field in the preset storage unit; determining the primary key based on the version number, the field, and the data type, and determining the target value based on the first offset; and determining the index structure based on the primary key and the target value.

[0034] In one embodiment, when the data type is a first type, obtaining the offset of the element of the composite data in a preset storage unit, and determining the index structure based on the metadata of the composite data and the offset, includes: when the data type is a first type, obtaining the version number of the composite data, and obtaining the second offset of the first element corresponding to the version number in the preset storage unit; determining the primary key based on the version number, determining the target value based on the second offset, wherein the second offset corresponds to a unique element; and determining the index structure based on the primary key and the target value.

[0035] Specifically, the first type of index includes FieldIndex and VersionIndex; FieldIndex is a hash index. The primary key of this index is the encoded key (i.e., version + data type + field), and the value of the index is the offset of the current field's data stored in the storage unit. The value can be obtained through this offset. FieldIndex is a hash index, and its query time complexity is O(1), which is faster than binary search. VersionIndex is also a hash index. The primary key of this index is the version (the version in the metadata), and the value is the offset of the first item corresponding to that version in the storage unit. The value can be obtained through this offset, and the search can proceed sequentially to find the next item until the required data is found or another version is encountered. Because VersionIndex is also a hash index, its query time complexity is O(1), which is faster than binary search.

[0036] In one embodiment, when the data type is a second type, obtaining the elements of the composite data and determining the index structure based on the metadata of the composite data and the elements includes: when the data type of the composite data is a queue type, obtaining the version number, data type, elements, and absolute position of the elements in a preset storage unit; determining a primary key based on the version number, the queue type, and the absolute position; determining a target value based on the elements; and determining the index structure based on the primary key and the target value.

[0037] Specifically, a List is a queue, ordered according to the order of writing. It supports writing from both ends and popping from both ends. The ListData encoding format has a primary key of: version + data type + position, and the data is the element itself. Here, the data type is List, the position is the absolute position within the list, and the index is ListIndex. The primary key of this index is version + data type + position, and the data is the offset of the element in the storage unit. This index is an array that records the index information of each element sequentially. Furthermore, the position is the absolute position of the element and has a fixed length. In one embodiment, when the data type is a second type, obtaining the elements of the composite data and determining the index structure based on the metadata of the composite data and the elements includes: when the data type of the composite data is an ordered set type, obtaining the version number, data type, fields, and elements of the composite data; determining the primary key based on the version number, the ordered set type, and the fields; determining the target value based on the elements; and determining the index structure based on the primary key and the target value.

[0038] In one embodiment, when the data type is a second type, the step of obtaining the elements of the composite data and determining the index structure based on the metadata of the composite data and the elements includes: when the data type of the composite data is an ordered set type, obtaining the version number, data type, fields, and elements of the composite data; determining first data based on the version number, the ordered set type, the fields, and the elements; and constructing an index structure by arranging the first data in order of the size of the elements.

[0039] Specifically, a Zset is an ordered set with a more complex structure. It needs to be sorted by score and can retrieve the score by field. To store these two types of data, the Zset needs to be encoded into two types: ZsetFieldData and ZsetScoreData. ZsetFieldData encoding format: primary key: version + data type + field, data: score. Here, the data type is ZsetField. ZsetScoreData encoding format: primary key: version + data type + score + field, data: empty. Here, the data type is ZsetScore. These two types of data correspond to different data types and different index structures. The index of ZsetFieldData: FieldIndex, the principle is the same as the FieldIndex of the Hash type. The Zscore command means to query the score of the current field under the current key. Using the key, query the metadata to get the key's version, and then use the version and field to query the FieldIndex to get its score value. ZsetScoreData index. There are two types: ZsetScoreVersion index and ZsetScoreOffset index. ZsetScoreOffset is an array that stores the offsets of all elements of zsetScore in the storage unit. The ZsetScoreVersion index records the index of this array. The two are used together to obtain the specific data. All score data under the current key are arranged in order. We call this data block a block. The primary key of the ZsetScoreVersion index is the version, and the data is the index of the first and last element of this block in ZsetScoreOffset. That is, the positions of the minimum and maximum scores.

[0040] Based on the index structure construction method described above, corresponding index structures are created according to different data types, and indexes are established under specific data organization forms, thereby significantly improving query efficiency. Compared with the general solution that can only rely on binary search for traversal, this solution uses hash indexes on datasets such as Hash, Set, ZsetFieldData, and ZsetScoreData, making the time complexity of the search operation O(1), which is significantly better than the traditional binary search method.

[0041] Although the index used on the List type is based on an array structure and implemented using binary search, the array is stored compactly, allowing multiple contiguous data entries to be loaded into the CPU cache in a single read, reducing multiple accesses to memory or disk. Compared to performing multiple binary search operations in a non-contiguous storage structure, this approach significantly improves query performance and achieves higher execution efficiency.

[0042] Example 2

[0043] like Figure 2 As shown, this embodiment provides a data indexing method. The method is implemented based on an index structure constructed using the index structure construction method described above. The method includes: Step S201: In response to the data indexing request, obtain the data index range, the data type of the data to be indexed, and the primary key of the data to be indexed; Step S202: Index the data to be indexed from the pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed.

[0044] In one embodiment, the step of indexing the data to be indexed from a pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed includes: when the data index range is a local index and the data type is a first type, obtaining the primary key and the fields and version number of the primary key, determining a first offset based on the fields and the version number, and indexing the data to be indexed based on the first offset; when the data index range is a global index and the data type is a first type, obtaining the primary key and the version number of the primary key, determining a second offset based on the version number, and indexing the data to be indexed based on the second offset.

[0045] In one embodiment, the step of indexing the data to be indexed from a pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed includes: when the data index range is a local index and the data type is a queue, obtaining the primary key and its absolute position, and determining the data to be indexed based on the absolute position; when the data index range is a global index and the data type is a queue, determining the index range value, obtaining the primary key and its absolute position, determining the absolute position within the index range value, and determining the data to be indexed based on the absolute position.

[0046] In one embodiment, the step of indexing the data to be indexed from a pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed includes: when the data type is an ordered set, obtaining the primary key and the version number of the primary key; obtaining the position information of the element corresponding to the version number from the pre-built index structure based on the version number; and determining the data to be indexed based on the position information.

[0047] As shown in Embodiments 1 and 2, this application proposes an index structure construction method and a data indexing method involving data types including hash structures, set structures, ordered sets, and queues.

[0048] In one implementation, when the data type is a hash structure, the index structure construction method includes constructing a FieldIndex and constructing a VersionIndex. The primary key of the constructed FieldIndex is version + data type + field, and its value is the offset of the data in that field within the storage unit. The primary key of the constructed VersionIndex is version, and its value is the offset of the first item corresponding to that version within the storage unit.

[0049] When the data type is a hash structure, the relevant commands for data indexing include the `Hget` and `Hgetall` commands. The `Hget` command executes as follows: it queries the metadata using the key to obtain the key's version; it queries the `FieldIndex` using the version and field to retrieve the value; and it returns the value. The `Hgetall` command executes as follows: it queries the metadata using the key to obtain the key's version; it queries the `VersionIndex` using the version to locate the first element of the current version; it iterates through the elements until it encounters the next version; and it returns all fields and values. In essence, when the data type is a hash structure, the data indexing steps involve first determining the version information, and then searching for data based on specific fields or version ranges.

[0050] In one implementation, when the data type is a set structure (Set type), the method for constructing the index structure is the same as the method for constructing the index structure corresponding to the hash structure, that is, it includes constructing FieldIndex and VersionIndex.

[0051] When the data type is a set structure, the relevant commands for data indexing include the Sismember and Smembers commands. The Sismember command executes as follows: it queries the metadata using the key to obtain the key's version; it queries the FieldIndex using the version and field to check if the element exists; and it returns the result indicating whether the element exists. The Smembers command executes as follows: it queries the metadata using the key to obtain the key's version; it queries the VersionIndex using the version to locate the first element of the current version; it iterates through the elements until it encounters the next version; and it returns all fields. In short, when the data type is a set structure, the data indexing process can be summarized as quickly locating set members using version information.

[0052] In one implementation, when the data structure is an ordered set (Zset type), the index structure construction method includes constructing ZsetFieldData and ZsetScoreData. Constructing ZsetFieldData involves a primary key of version + data type + field, and the data being score. Constructing ZsetScoreData involves a primary key of version + data type + score + field, and the data is empty. The construction rules for FieldIndex are consistent with those for Hash types. ZsetScoreVersion and ZsetScoreOffset are used for range queries sorted by score.

[0053] Specifically, the indexing mechanism of ZsetScoreData involves a Block, which refers to a contiguous block of data where all score data under the current key are arranged in an ordered manner; each block has a start position and an end position, which are represented by the index in the ZsetScoreOffset array.

[0054] The ZsetScoreVersion index structure includes a primary key: version; and data: the indices of the first and last elements of this block in the ZsetScoreOffset array, corresponding to the minimum and maximum scores. The ZsetScoreOffset index structure is an array that records the offset of each element in the storage unit, allowing for quick location of specific elements.

[0055] The indexing rules specifically include: Step 1: Querying Metadata using the Key: Input key: The user initiates a query request, providing a specific key (e.g., user_scores); Get version: The system uses the provided key to query the metadata and obtains the version number corresponding to the current key (e.g., 2025). Step 2: Searching the Block Range: Searching the ZsetScoreVersion index: Using the version obtained from the metadata as the primary key, the system searches in the ZsetScoreVersion index; this index returns the indices of the first and last elements of the block under that version in the ZsetScoreOffset array; assuming the returned result is [start_index, end_index], representing the positions of the minimum and maximum scores respectively; Step 3: Locating the Specific Element: Searching the ZsetScoreOffset index: Using start_index and end_index, the system searches from the ZsetScoreOffset array... Find the actual offsets of these elements in the storage unit; these offsets indicate the exact location of each element in the storage medium (such as the storage unit); Step 4: Execute the query operation: Execute the query according to the requirements: If you need to query a specific score (such as through the ZRANGEBYSCORE or ZREVRANGEBYSCORE command), then based on the given minimum and maximum score range; starting from start_index, traverse sequentially to end_index, checking whether the score of each element is within the specified range; for ZRANGEBYSCORE, traverse sequentially; for ZREVRANGEBYSCORE, traverse in reverse order; return all fields and scores that meet the conditions.

[0056] When the data type is an ordered set, the relevant commands for data indexing methods include the Zscore command, the Zrangebyscore command, and the Zrevrangebyscore command. The execution steps of the Zscore command are as follows: use the key to query the metadata to obtain the key's version; use the version and field to query the FieldIndex index to obtain the score; and return the score. The execution steps of the `ZrangebyScore` command include: querying the metadata using the key to obtain the key's version; querying the `ZsetScoreVersion` index using the version to obtain the start and end indices of the block; querying the `ZsetScoreOffset` index based on the start index to obtain the storage unit offset; iterating backward through the `ZsetScoreOffset` index to determine if the score is within the range; and returning all scores and fields that meet the conditions. The execution steps of the `ZrevrangebyScore` command include: querying the metadata using the key to obtain the key's version; querying the `ZsetScoreVersion` index using the version to obtain the start and end indices of the block; querying the `ZsetScoreOffset` index based on the end index to obtain the storage unit offset; iterating backward through the `ZsetScoreOffset` index to determine if the score is within the range; and returning all scores and fields that meet the conditions. Overall, when the data type is an ordered set, the data indexing steps can be summarized as first determining the version information, and then searching for data based on the score range or a specific field.

[0057] When the data type is a queue (List type), its corresponding encoding format is that the primary key is version + data type + position, and the data is the element itself; the index structure construction method includes building a ListIndex; the primary key of the built ListIndex is version + data type + position, and the data is the offset of the element in the storage unit.

[0058] When the data type is a queue (List type), the data indexing commands include the Lindex and Lrange commands. The Lindex command executes as follows: Query the metadata using the key to obtain the key's version; calculate the absolute position based on the absolute starting position of the metadata and the given index; query the ListIndex using the version and absolute position, and perform a binary search to obtain the storage unit offset; return the value. The Lrange command executes as follows: Query the metadata using the key to obtain the key's version; calculate the absolute position based on the given index and absolute starting position; query the ListIndex using the version and absolute position, and perform a binary search to obtain the storage unit offset; traverse n elements to the right using the binary search index; return these elements. In essence, when the data type is a queue, the data indexing steps involve first determining the version information, and then searching for the element based on the index and position.

[0059] In general, hash data types are primarily used to store structured data and support efficient single or batch queries; set data types are suitable for scenarios requiring element uniqueness, such as tag management; ordered set data types are suitable for supporting complex sorting and query ranges, and are applicable to leaderboards, priority queues, and other scenarios requiring sequential processing; queue data types are suitable for task queues, message passing, and other scenarios requiring sequential processing; Hash and Set focus more on member existence and single / batch retrieval; Zset emphasizes sorting by score and range queries; List supports fast access based on position and range queries. Hash and Set use similar FieldIndex and VersionIndex to accelerate lookups; Zset requires additional ZsetScoreVersion and ZsetScoreOffset to support its unique sorting functionality; List relies on a compact ListIndex array to optimize random access performance.

[0060] Based on the index structure construction method and data indexing method proposed in this application, a differentiated and efficient index structure is constructed for various composite data types on the basis of statically written and immutable metaData. Combined with metaData, precise version control and fast query capabilities are achieved, thereby significantly improving the system's read performance and query flexibility while ensuring data consistency and version isolation.

[0061] Example 3

[0062] like Figure 3As shown, this embodiment also provides an index structure construction device, the device comprising: a response unit for responding to an index structure construction request and obtaining composite data and the data type of the composite data; a construction unit for, when the data type is a first type, obtaining the offset of the elements of the composite data in a preset storage unit, and determining an index structure based on the metadata of the composite data and the offset, wherein the first type includes at least one of a hash structure and a set structure; and further for, when the data type is a second type, obtaining the elements of the composite data, and determining an index structure based on the metadata of the composite data and the elements, wherein the second type includes at least one of an ordered set and a queue.

[0063] Example 4

[0064] like Figure 4 As shown, this embodiment also provides a data indexing device, which is implemented based on the index structure constructed by the index structure construction device of claim 11, and the device includes: A response unit is used to respond to a data indexing request and obtain the data index range, the data type of the data to be indexed, and the primary key of the data to be indexed. An indexing unit is used to index the data to be indexed from a pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed.

[0065] Example 5

[0066] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the index structure construction method and data indexing method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0067] The electronic device can be a user device, or a device formed by integrating user devices and network devices through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0068] Figure 5 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The device 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes based on a program stored in a Read-Only Memory (ROM) 502 or a program loaded from a storage portion 508 into a Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0069] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), LED display, OLED display, etc., and speakers, etc.; a storage section 508 including one or more computer-readable media such as hard disk, optical disk, magnetic disk, semiconductor memory, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet.

[0070] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 501, it performs the functions defined in the methods of this application.

[0071] Example 6

[0072] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0073] Specifically, this embodiment may employ any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0074] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, device, or apparatus.

[0075] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0076] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0077] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or page components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces, or indirect couplings or communication connections between devices or units, and may be electrical, mechanical, or other forms.

[0080] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0082] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0084] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for constructing an index structure, characterized in that, The method includes: In response to the index structure construction request, obtain the composite data and the data type of the composite data; When the data type is the first type, the offset of the element of the composite data in the preset storage unit is obtained, and the index structure is determined based on the metadata of the composite data and the offset. The first type includes at least one of hash structure and set structure. When the data type is the second type, the elements of the composite data are obtained, and the index structure is determined based on the metadata of the composite data and the elements. The second type includes at least one of ordered sets and queues.

2. The index structure construction method according to claim 1, characterized in that, When the data type is a first type, the offset of the element of the composite data in a preset storage unit is obtained, and the index structure is determined based on the metadata of the composite data and the offset, including: When the data type is the first type, obtain the version number and field of the composite data, and obtain the first offset of the element of the field in the preset storage unit; The primary key is determined based on the version number, the field, and the data type; the target value is determined based on the first offset. The index structure is determined based on the primary key and the target value.

3. The index structure construction method according to claim 2, characterized in that, When the data type is a first type, the offset of the element of the composite data in a preset storage unit is obtained, and the index structure is determined based on the metadata of the composite data and the offset, including: When the data type is the first type, obtain the version number of the composite data, and obtain the second offset of the first element corresponding to the version number in the preset storage unit; The primary key is determined based on the version number, and the target value is determined based on the second offset, where the second offset corresponds to a unique element. The index structure is determined based on the primary key and the target value.

4. The index structure construction method according to claim 1, 2, or 3, characterized in that, When the data type is the second type, the elements of the composite data are obtained, and the index structure is determined based on the metadata of the composite data and the elements, including: When the data type of the composite data is a queue type, obtain the version number, data type, element, and absolute position of the element in the preset storage unit of the composite data; The primary key is determined based on the version number, the queue type, and the absolute position; the target value is determined based on the element. The index structure is determined based on the primary key and the target value.

5. The index structure construction method according to claim 4, characterized in that, When the data type is the second type, the elements of the composite data are obtained, and the index structure is determined based on the metadata of the composite data and the elements, including: When the data type of the composite data is an ordered set type, obtain the version number, data type, fields, and elements of the composite data; The primary key is determined based on the version number, the ordered set type, and the field; the target value is determined based on the element. The index structure is determined based on the primary key and the target value.

6. The index structure construction method according to claim 1 or 5, characterized in that, When the data type is the second type, the elements of the composite data are obtained, and the index structure is determined based on the metadata of the composite data and the elements, including: When the data type of the composite data is an ordered set type, obtain the version number, data type, fields, and elements of the composite data; The first data is determined based on the version number, the ordered set type, the field, and the element; The first data is arranged in order of size to construct an index structure.

7. A data indexing method, characterized in that, The method is implemented based on an index structure constructed using the index structure construction method according to any one of claims 1 to 6, and the method includes: In response to a data indexing request, retrieve the data index range, the data type of the data to be indexed, and the primary key of the data to be indexed; The data to be indexed is indexed from a pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed.

8. The data indexing method according to claim 7, characterized in that, The process of indexing the data to be indexed from a pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed includes: When the data index range is a local index and the data type is a first type, the primary key and the fields and version number of the primary key are obtained, a first offset is determined based on the fields and the version number, and the data to be indexed is indexed based on the first offset; When the data index range is a global index and the data type is a first type, the primary key and the version number of the primary key are obtained, a second offset is determined based on the version number, and the data to be indexed is indexed based on the second offset.

9. The data indexing method according to claim 7 or 8, characterized in that, The process of indexing the data to be indexed from a pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed includes: When the data index range is a local index and the data type is a queue, obtain the primary key and the absolute position of the primary key, and determine the data to be indexed based on the absolute position; When the data index range is a global index and the data type is a queue, determine the index range value, obtain the primary key and the absolute position of the primary key, determine the absolute position within the index range value, and determine the data to be indexed based on the absolute position.

10. The data indexing method according to claim 9, characterized in that, The process of indexing the data to be indexed from a pre-built index structure based on the data index range, the data type, and the primary key of the data to be indexed includes: When the data type is an ordered set, obtain the primary key and the version number of the primary key; Based on the version number, obtain the position information of the element corresponding to the version number from the pre-built index structure; The data to be indexed is determined based on the location information.

Images