Distributed index construction method and apparatus, and electronic device

Abstract

This application discloses a distributed index construction method, apparatus, and electronic device, relating to the field of distributed storage technology. The method includes: splitting a target key through an intermediate layer proxy to obtain a first key and multiple second keys; wherein the first key is associated with metadata used to determine the second keys, and each second key is associated with a portion of the member data of the target key. Based on the splitting processing logic of the intermediate layer proxy, this application achieves the transformation of the target key into a distributed key by splitting the target key into a first key associated with metadata and multiple second keys each carrying a portion of the member data. This effectively solves the technical problems of thread blocking and uneven cluster memory distribution caused by "large keys" in existing distributed Redis clusters, ensuring the stable and efficient operation of distributed Redis clusters in large data volume scenarios.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a distributed index construction method, apparatus and electronic device. Background Technology

[0002] In the context of the internet industry's rapid growth in data volume and traffic, Redis, as a mainstream key-value (KV) storage system, has been widely adopted. To meet the storage capacity and performance requirements of distributed scenarios, distributed storage systems built on Redis nodes and middleware proxies have become the mainstream deployment model in the industry. The middleware proxy is responsible for request forwarding and data routing, achieving distributed storage and load balancing by dividing data into fixed logical units (such as data slots), enabling clients to easily access the distributed Redis cluster.

[0003] However, in real-world business scenarios, some Redis composite data structures accumulate massive amounts of member data for their keys. In the industry, keys with such large amounts of member data (e.g., more than 5000) are typically defined as "large keys." In a distributed Redis cluster environment, due to the routing mechanism of the middleware proxy, all member data for these "large keys" is centrally stored on a single shard node. This not only directly leads to an imbalance in cluster memory distribution, preventing the full utilization of storage and computing resources on other shard nodes and resulting in resource waste, but also easily blocks the Redis single-threaded main thread and client processes. Ultimately, this creates a cluster performance bottleneck, severely limiting the core business requirements for massive data storage and high-concurrency access. Summary of the Invention

[0004] In view of this, this application provides a distributed index construction method, apparatus, and electronic device. It aims to solve the technical problems of thread blocking and uneven cluster memory distribution caused by "large keys" in existing distributed Redis clusters.

[0005] Firstly, this application provides a distributed index construction method applied to a distributed storage system, including: The target key is split by an intermediate proxy to obtain a first key and multiple second keys. The first key is associated with the metadata used to determine the second keys, and each second key is associated with a portion of the target key's member data.

[0006] Secondly, this application provides a distributed index building apparatus, which is applied to a distributed storage system and includes a processing module; The processing module is configured to split the target key through an intermediate layer proxy to obtain a first key and multiple second keys; wherein the first key is associated with the metadata used to determine the second key, and each second key is associated with a portion of the member data of the target key.

[0007] Thirdly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect.

[0008] In view of the above embodiments, this application, based on the splitting processing logic of the intermediate layer proxy, achieves the transformation of the target key into a distributed key by splitting the target key into a first key associated with metadata and multiple second keys that each carry a portion of the member data. Since multiple second keys can distribute the member data of the target key to different shard nodes in the cluster, it can avoid the uneven memory distribution caused by a single shard node centrally storing a large amount of member data, and make full use of the cluster sharding resources.

[0009] Furthermore, since the amount of member data for a single second key is significantly reduced after splitting, the operation time is greatly shortened, thus effectively avoiding client blocking, breaking through the single-key member data volume limit, and ensuring the stable and efficient operation of the distributed Redis cluster in big data scenarios.

[0010] It should be noted that the above content is only a general overview of the technical solution of this application. In order to enable those skilled in the art to clearly understand the core technical means of this application and to accurately implement this solution based on the content disclosed in the specification, the technical details of this application will be described in detail below in conjunction with specific embodiments. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the principles of this application.

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This paper presents a structural example diagram of a distributed Redis cluster provided in an embodiment of this application. Figure 2 A flowchart illustrating a distributed index construction method provided in an embodiment of this application is shown. Figure 3A flowchart illustrating a distributed index construction method provided in an embodiment of this application is shown. Figure 4 A flowchart illustrating a distributed index construction method provided in an embodiment of this application is shown. Figure 5 A flowchart illustrating a distributed index construction method provided in an embodiment of this application is shown. Figure 6 A flowchart illustrating a distributed index construction method provided in an embodiment of this application is shown. Figure 7 The diagram illustrates an example of the interaction logic of a distributed index construction method provided in an embodiment of this application. Figure 8 A schematic diagram of the structure of a distributed index building apparatus provided in an embodiment of this application is shown. Detailed Implementation

[0014] To facilitate the explanation of the embodiments of this application, some technical terms and technical means related to the embodiments of this application, as well as the application scenarios of the embodiments of this application, will be introduced first below.

[0015] A distributed Redis cluster is a distributed key-value storage system consisting of a storage layer and a middleware proxy. It achieves horizontal data scaling through sharding and slot mechanisms, aiming to overcome the memory, computation, and bandwidth bottlenecks of a single Redis instance, while allowing clients to connect to the cluster as easily as accessing a single Redis instance.

[0016] The middleware proxy is the core routing component of a distributed Redis cluster. It is responsible for receiving client requests, executing data routing logic, and forwarding requests, ensuring transparent access to the distributed cluster for clients.

[0017] Sharding technology is a technique that distributes data across multiple shard nodes to improve overall system performance, storage capacity, and availability.

[0018] The data slotting mechanism is used to logically divide data. The distributed Redis cluster pre-determines a fixed number of slots (e.g., 1024) and distributes these slots evenly to each shard node, serving as the core basis for data routing.

[0019] The hash data structure is a Redis data type that can store the mapping relationship between multiple fields and values. It is a commonly used type for storing large amounts of related data in business applications.

[0020] In the industry, a "large key" is usually defined as a Redis composite data structure key with more than 5,000 members.

[0021] like Figure 1 The diagram illustrates the structure of a distributed Redis cluster (hereinafter referred to as the cluster). This cluster structure comprises three core components: a client, a proxy layer, and a storage layer. The storage layer contains multiple shard nodes. The cluster has a pre-defined fixed number of slots (e.g., 1024), which are evenly distributed among the shard nodes, forming a shard-slot correspondence. For example, 1024 slots are allocated to shard node 1 and shard node 2, with shard node 1 handling slots 0-511 and 512-1023 respectively. The proxy layer includes multiple middleware proxies (1-3), each capable of accessing the shard nodes. When a client initiates a data operation request, the middleware proxy first calculates the hash value of the target key, then modulo the hash value with the total number of slots to obtain a unique slot number. Based on the shard-slot correspondence, the middleware proxy accurately forwards the request to the shard node corresponding to that slot, where the shard node ultimately performs the data read / write processing.

[0022] Due to limitations in current routing logic, regardless of whether the target key is a plain string structure or a Redis composite data structure (hash), it will only be routed to a single shard node. When a composite data structure key needs to store a large amount of member data (such as a "large key"), all its member data will be centrally stored on a single shard node. On the one hand, this will cause uneven distribution of cluster memory, failing to fully utilize the storage and computing resources of other shard nodes. On the other hand, read and write operations on "large keys" are time-consuming, easily blocking client processes and affecting business response efficiency.

[0023] This application provides a distributed index construction method, apparatus, and electronic device, aiming to solve problems such as thread blocking, uneven resource utilization, performance bottlenecks, and single-key data volume limitations caused by "large keys" in existing distributed Redis clusters. This application uses the splitting processing logic of an intermediate layer proxy to split "large keys" into multiple keys, achieving distributed storage and efficient processing of data within "large keys," fully releasing the potential of cluster resources, and reducing business development and maintenance costs.

[0024] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0025] like Figure 2As shown, this document describes the execution process of a distributed index construction method. This method can be applied to various distributed storage systems that support sharded storage and intermediate layer proxy routing. Examples include the distributed Redis cluster described above, distributed memory object caching system (Memcached) clusters, and distributed transactional key-value stores (such as TiKV). The following section uses a distributed Redis cluster as an example to detail the execution process of this method.

[0026] The method includes the following step S101: S101. The target key is split through an intermediate layer proxy to obtain the first key and multiple second keys.

[0027] The first key is associated with the metadata of the second key, and each second key is associated with a portion of the member data of the target key.

[0028] For example, the target key may refer to the key (hash type) of a Redis composite data structure that needs to store a large amount of member data in a business scenario. When the amount of member data exceeds 5,000, the target key is a "large key" that poses performance risks in existing technologies and is the object to be split in this method.

[0029] The first key (also known as the total key) refers to the metadata associated (storage) key generated after splitting the target key. The first key itself is not associated with the member data of the target key. In this method, the metadata associated with the first key is used to provide a basis for sharding location for subsequent business operations.

[0030] The second key (also known as the sharding key) refers to the key that associates (stores) multiple member data generated after splitting the target key. Each second key is associated with a portion of the member data of the target key, and the uniqueness of each second key is ensured by a preset identification rule, thus achieving effective distributed storage of data.

[0031] In some embodiments, the splitting process can be triggered by an instruction. For example, the business initiates a preset distributed creation command (such as dk.hcreate) and passes in the target key name (such as the first key identifier) ​​and the preset number of shards. After receiving the instruction, the middleware agent starts the splitting process.

[0032] In other embodiments, the splitting process can be based on automatic detection triggering. For example, the middleware agent monitors the amount of member data for the target key in real time, and when it detects that the number of its members is about to or has exceeded a preset number (such as 5000), it automatically triggers the splitting logic without requiring additional instructions from the business logic.

[0033] In some implementations, the middleware proxy first creates a first key and associates it with metadata, then generates multiple second keys according to preset identification rules, and associates some member data with each second key, thereby completing the splitting of the target key.

[0034] In some embodiments, after generating a first key and multiple second keys, the middleware agent maps the multiple second keys to different slots in the distributed Redis cluster through hash modulo, thereby distributing member data to various shard nodes for storage, while the first key is independently mapped to a specified slot, ensuring fast access to metadata.

[0035] By splitting the target key into a first key containing metadata and multiple second keys stored in a distributed manner, distributed storage of member data within the target key is achieved, avoiding client blocking and waste of cluster resources, and improving the performance of the distributed Redis cluster.

[0036] The distributed index construction method provided in this application embodiment will now be described in detail. This method consists of a distributed index construction phase and a distributed index application phase, with the two phases working together to achieve distributed storage and efficient operation of "large keys".

[0037] In some embodiments, for the distributed index construction phase, this application provides a splitting implementation method based on instruction triggering, based on the proxy layer and slot mechanism of the distributed Redis cluster.

[0038] First, receive the distributed creation command. This command includes the preset number of shards and the first key identifier of the target key.

[0039] For example, distributed creation commands can be proactively generated by the business layer based on actual storage needs.

[0040] In some scenarios, the Redis composite data structure key (hash type) that the business needs to store may have more than 5,000 expected members (i.e., there is a risk of it becoming a "large key"), or it is known that a large amount of related data needs to be stored (such as user order sets, massive configuration information, etc.). In this case, the business layer does not need to manually design sharding logic. It only needs to generate a distributed creation command according to a preset format to trigger the middleware agent to execute the "large key" splitting process.

[0041] This distributed creation command is, for example, a custom Redis extension command adapted to the scenario of this application. It is named "dk.hcreate". Here, "dk" is an abbreviation for distributed key, and "hcreate" corresponds to the creation semantics of a Redis hash type.

[0042] The distributed creation command is compatible with regular Redis commands and includes two core parameters. The first core parameter is the first key identifier of the target key (i.e., the target key name defined on the business side, such as "user_orders"), used to uniquely associate the "large key" that needs to be split. The second core parameter is the preset number of shards (such as 4, 8, or other positive integers), which is reasonably configured by the business based on the expected data volume and the number of shard nodes in the cluster. The value of the preset number of shards determines the granularity of data distribution storage.

[0043] The business layer sends the distributed creation command to the middleware agent of the distributed Redis cluster via a Redis client (such as Jedis or Redisson). Upon receiving the command, the middleware agent first verifies the command format, parameter integrity, and validity (e.g., whether the preset shard number is a positive integer). If the verification passes, the middleware agent parses the first key identifier and the preset shard number, and then proceeds to the subsequent sharding process.

[0044] like Figure 3 As shown, based on the distributed creation command, the following splitting operation is performed through an intermediate layer proxy: S201. Generate the first key based on the first key identifier.

[0045] For example, the middleware agent parses the first key identifier in the distributed creation command and generates a first key with the same name as that first key identifier, which serves as a dedicated storage carrier for metadata. For instance, when the first key identifier is parsed as "user_orders", the first key "user_orders" is generated.

[0046] S202. Associate the first key with metadata.

[0047] The metadata includes the preset number of shards. For example, the preset number of shards is a core parameter passed by the business through a distributed creation command (such as dk.hcreate), which specifies the total number of second keys (shard keys) after splitting the target "big key".

[0048] In some examples, the metadata also stores the total number of members. The total number of members is used to count the total amount of complete member data for the target key. It is initially 0 when the business writes member data via distributed commands (such as dk.hset). The middleware agent updates this value in real time based on the operation results (e.g., if one piece of data is written, the total number of members increases by 1). The business can quickly obtain the overall data scale of the target key by querying the first key, without having to traverse all the second keys to count.

[0049] In some embodiments, the metadata associated with the first key can be stored in the storage layer of the distributed Redis cluster. After the middleware agent writes the metadata to the first key, it maps the first key to the specified slot and shard node according to the cluster routing rules, thereby achieving persistent storage of metadata, ensuring data reliability, and is suitable for scenarios with high requirements for metadata consistency.

[0050] In some embodiments, the metadata associated with the first key can also be stored in a local cache (such as the memory cache of the proxy node) through an intermediate layer proxy. After the intermediate layer proxy queries the metadata of the first key, it caches it locally, and subsequent business operations can directly read it from the local cache without repeatedly accessing the storage layer. This significantly reduces the time spent querying metadata, improves the execution efficiency of distributed commands, and adapts to high-frequency read and write business scenarios.

[0051] S203. Generate a preset number of second keys based on the first key identifier and the fragment identifier.

[0052] Each second key has a different shard identifier. For example, the shard identifier is a unique identifier that distinguishes multiple second keys. This ensures that each second key is unique, guaranteeing that the member data of the target key can be accurately split into different second keys, and then distributed and stored across different shard nodes in the distributed Redis cluster, avoiding the drawbacks of centralized storage in a single shard.

[0053] For example, the middleware agent first determines the composition rules of the shard identifier (such as containing only a numeric suffix, or containing a fixed character hash value + numeric suffix), and then generates a corresponding number of unique identifiers based on the preset number of shards in the distributed creation command (such as generating 4 unique identifiers when the preset number of shards is 4), and ensures that the generated shard identifiers are compatible with the naming rules of the second key, and can be directly used to assemble the second key.

[0054] In some embodiments, the shard identifier can be flexibly designed according to cluster requirements. For example, consecutive integers such as 0, 1, 2, and 3 can be used. This design is simple in structure, easy to generate, and suitable for most scenarios. Another example is using a fixed character followed by a numeric suffix. The fixed character is a fixed value of the intermediate layer proxy (such as the MD5 value of the preset string "distributed_shard"), and the numeric suffix is ​​a consecutive integer, which can further enhance the uniqueness of the second key and prevent accidental business operations. Yet another example is calculating a hash of a preset field (such as the first key identifier) ​​and taking a portion of the value as the identifier, suitable for scenarios with high security requirements.

[0055] In some embodiments, the fragment identifier includes a first value (such as a fixed character) and a second value (such as a numeric suffix).

[0056] A hash function is used to calculate a first value from a preset string. This first value is a fixed value used by the intermediate layer proxy.

[0057] For example, the middleware proxy calculates a fixed-length hash value (such as the MD5 value "0ecbf9426bcfbd9a086ded5fc8c4eca8") on a fixed-preset string (such as "distributed_shard") using a preset hash function (such as MD5). This first value is a globally fixed value of the middleware proxy, and the first value of all second keys is consistent to ensure that the second key is not mistakenly manipulated by the business side and to improve uniqueness.

[0058] Generate a preset number of second values ​​for the number of shards. Each second value has a different value.

[0059] For example, the middleware agent generates a corresponding number of second values ​​with unique values ​​based on the preset number of shards in the distributed creation command. These values ​​are usually consecutive integers (e.g., when the preset number of shards is 4, the second value is 0, 1, 2, or 3). These values ​​are used to distinguish different second keys after the same target key is split, ensuring that the shard identifier of each second key is unique.

[0060] Based on the aforementioned shard identifiers, the middleware agent first obtains the first key identifier (e.g., "test") and the preset number of shards (e.g., 4) from the distributed creation command. Simultaneously, it determines the generated first value (a fixed value) and multiple different second values ​​(e.g., 0-3). Each second value is then combined with the first value in the format "first value + separator (e.g., underscore) + second value" to form unique shard identifiers corresponding to the preset number of shards (e.g., 4). Subsequently, using a unified naming convention of "first key identifier + separator (e.g., underscore) + shard identifier," the first key identifier is sequentially concatenated with each unique shard identifier to generate a second key matching the preset number of shards.

[0061] For example, if the preset number of shards is 4, the second key determined using the unified naming rule in the example above is: "test_0ecbf9426bcfbd9a086ded5fc8c4eca8_0"; "test_0ecbf9426bcfbd9a086ded5fc8c4eca8_1"; "test_0ecbf9426bcfbd9a086ded5fc8c4eca8_2"; "test_0ecbf9426bcfbd9a086ded5fc8c4eca8_3".

[0062] S204. According to the association strategy, associate the member data of the target key with the corresponding second key.

[0063] Among them, the member data associated with each second key together constitute the member data of the target key.

[0064] For example, the association strategy is the core mechanism by which the intermediate layer proxy accurately distributes the massive amount of member data of the target key to the corresponding second key (sharding key) according to preset rules. This is used to achieve even distribution of data among multiple second keys, avoid overload of a single second key, and provide a fast location basis for subsequent data read and write, adapting to the performance requirements of distributed storage scenarios.

[0065] In some embodiments, the middleware proxy, according to the association strategy, first accesses the specified slot where the first key (total key) is located to obtain the preset number of shards in the metadata, then performs a hash calculation (such as CRC16 algorithm) on the member data field passed in by the business, and takes the modulo with the preset number of shards. Based on the modulo result, it matches the corresponding second key generated in S203, completing the association between the member data and the second key.

[0066] After completing the above association, the middleware agent completes the routing mapping according to the slot mechanism of the distributed Redis cluster. For example, the middleware agent independently performs a hash calculation for each second key, and then takes the modulo of the total number of slots in the cluster (e.g., 1024) to obtain the corresponding slot number, so that the second key is distributed and mapped to different shard nodes (e.g., with 2 shards, the second key corresponds to the shards with slots 0-511 and 512-1023 respectively). After the mapping is completed, the business can transparently read and write the target key through distributed commands (e.g., dk.hset, dk.hget), without needing to pay attention to the underlying sharding and routing logic, and the operation experience is consistent with accessing a single-machine Redis.

[0067] If the splitting operation is successful, the middleware agent returns a creation success response (such as "OK") to the business. It confirms that the first key and the second key for the preset number of shards have been generated, and that the first key has been successfully associated with metadata (including the preset number of shards, total number of members, etc.).

[0068] In some embodiments, if the splitting operation fails, all generated keys are deleted. A splitting operation failure includes the failure to generate or associate the first key and any second key.

[0069] For example, generation failures may include: the first key triggering a cluster node exception during the creation of the specified slot; network timeout causing the creation command to fail to execute successfully; the second key failing to complete creation due to insufficient memory on the cluster shard node, permission restrictions, etc.; some second keys being created successfully, but the remaining second keys failing to be generated due to node failure, etc.

[0070] For example, association failures may include: algorithmic errors during hash calculation of member data fields, resulting in the inability to generate a valid hash value; failure to match the corresponding second key generated by S203 after modulo the hash value with the preset shard number; loss or tampering of the preset shard number in the metadata, causing the association logic to fail; and data loss during the writing of the mapping relationship between member data and the second key to the cluster, resulting in the failure to complete a valid association.

[0071] For example, if the splitting operation fails, the middleware agent triggers a rollback mechanism to delete all generated keys (including the first key that was successfully created and some second keys) to prevent invalid keys from remaining in the cluster from occupying storage resources or interfering with subsequent operations.

[0072] This application also provides an implementation method for determining the association strategy. For example... Figure 4 As shown, this embodiment includes the following steps S301 to S304: S301, Obtain the first data identifier of the member data of the target key.

[0073] The first data identifier corresponds to the unique distinguishing field of the target key member data, which is used to uniquely identify a single member data and is the basic input for subsequent hash calculations.

[0074] S302. Generate the hash value of the first data identifier using a hash function.

[0075] For example, a preset hash function is used to calculate the hash value of the first data identifier, ensuring that different member data identifiers can be converted into differentiated numerical results, thus providing a premise for uniform sharding.

[0076] S303. Take the hash value of the first data identifier modulo the preset number of fragments to obtain the first modulo value.

[0077] For example, the preset number of shards is the number of shards passed in the distributed creation command (such as dk.hcreate) (e.g., 4). The hash value of the first data identifier generated in step S302 is moduloed with the number of shards to obtain the first modulo value in the range of 0 to (number of shards minus 1) (e.g., when the number of shards is 4, the modulo value is 0, 1, 2, or 3).

[0078] S304. Associate the first modulus value with each fragment identifier to determine the association strategy.

[0079] For example, the first modulus value obtained in step S303 is mapped one-to-one with the shard identifier (e.g., modulus value 0 corresponds to the shard key with the numeric suffix 0), forming a mapping rule of "first data identifier to hash value to first modulus value to shard identifier to second key", which is the association strategy.

[0080] In some embodiments, for the distributed index application stage, this application provides a distributed index processing embodiment that supports transparent data operations based on the "large key" splitting and distributed storage requirements of Redis composite data structure.

[0081] Receive a business operation request. The business operation request includes a second key identifier, a second data identifier, and a business operation type.

[0082] For example, when a business initiates a data write request, the second key identifier is "test", the second data identifier is the order number "order_1001", and the business operation type is "hset". This data write request has the same command parameter format as when a business accesses a single-machine Redis instance.

[0083] like Figure 5 As shown, based on the business operation request, the following indexing operation is performed through the middleware proxy: S401. If the second key identifier matches the first key identifier, then obtain the metadata associated with the first key.

[0084] For example, if the second key identifier "test" matches the first key identifier, the middleware proxy queries the metadata corresponding to the first key. This includes core information such as the preset number of shards (e.g., 4) and the total number of members.

[0085] S402. Based on the first key identifier, the first value, the second data identifier, and the preset number of fragments included in the metadata, determine the target second key through a preset assembly strategy.

[0086] For example, the preset assembly strategy is the core strategy for achieving accurate data routing after the "large key" is split in this application. Through standardized calculation and assembly logic, it is ensured that the second data identifier can be uniformly mapped to the corresponding shard key, and the whole process is transparent to the business without the need for business intervention.

[0087] like Figure 6 As shown, S402 includes the following steps S4021 to S4023.

[0088] S4021. Generate the hash value of the second data identifier using a hash function.

[0089] For example, a hash value for the second data identifier "order_1001" is generated by using a hash algorithm such as CRC16.

[0090] S4022. Take the modulo between the hash value of the second data identifier and the preset number of fragments in the metadata to obtain the second modulo value.

[0091] For example, if the preset number of shards in the metadata is 4, the hash value of the second data identifier generated in step S4021 is modulo 4 to obtain a second modulo value in the range of 0 to 3. This second modulo value directly corresponds to the shard identifier of the second key.

[0092] S4023. Assemble the first key identifier, the first value, and the second modulus value to determine the target second key.

[0093] For example, the target second key is determined by assembling the data in the format of "first key identifier_first value_second modulus value".

[0094] S403. Perform data operations based on the target second key and business operation type to obtain the data operation results.

[0095] For example, if the business operation type is "hset", the middleware proxy forwards the operation request to the shard node corresponding to the target second key, executes the ordinary Redis hset command to write the data. After the operation is completed, it synchronously updates the metadata such as the total number of members of the first key, and returns the operation result (such as "successfully written" "number of rows affected 1") to the business, with the return format consistent with that of a single-machine Redis.

[0096] The above example enables businesses to achieve distributed massive data storage and access without needing to be aware of the "large key" splitting logic, and can follow the single-machine Redis operation habits. This solves the performance and resource distribution problems of "large keys", reduces development and maintenance costs, improves efficiency, and ensures a consistent operating experience.

[0097] like Figure 7 As shown, this example illustrates the core process of data writing performed by the business side using the distributed command dk.hset. The interaction logic for each step is as follows: S501, Business Initiation Operation Command.

[0098] The business side follows the operating habits of a single-machine Redis, and sends a distributed write command dk.hset to the middle-layer proxy, passing in the target key name, member data field (such as order number) and corresponding value.

[0099] S502, the middle layer proxy obtains metadata.

[0100] After receiving the request, the middleware agent initiates a query to the storage layer, accesses the specified slot where the first key (total key) is located, and obtains the preset number of shards in the metadata.

[0101] S503, intermediate layer agent assembles fragment key.

[0102] The intermediate layer proxy assembles the target second key (sharding key) according to a preset format based on the hash modulo result of the first key identifier, the fixed first value, and the member data fields.

[0103] S504, Intermediate Layer Proxy Forwarding Operation Command.

[0104] The middleware agent transforms the business's dk.hset request into a regular Redis hset command and forwards it to the storage layer (the shard node where the target's second key is located) to perform the data write operation.

[0105] S505, the intermediate layer agent returns the operation results to the business side.

[0106] After the storage layer (sharding node) completes its operation, it returns the result (such as "OK") to the middleware agent, which then passes the result to the business side. The return format is consistent with that of a single-machine Redis, ensuring that there is no difference in the business operation experience.

[0107] The above interaction process achieves full transparency of the business chain. Businesses only need to execute familiar Redis commands without having to worry about the underlying logic. The middleware agent automatically completes complex operations such as metadata query, shard key assembly, and data routing, which not only solves the performance bottleneck caused by "large keys" but also keeps up with the business's existing development habits, significantly reducing access costs.

[0108] Furthermore, this embodiment provides a distributed index building device applied to a distributed storage system, such as... Figure 8 As shown, the device includes a processing module 810. Wherein: The processing module 810 is configured to split the target key through an intermediate layer proxy to obtain a first key and multiple second keys; wherein the first key is associated with the metadata used to determine the second key, and each second key is associated with a portion of the member data of the target key.

[0109] In some embodiments, the processing module 810 is configured to receive a distributed creation command; the distributed creation command includes a preset number of shards and a first key identifier of the target key.

[0110] Based on the distributed creation command, the following splitting operation is performed through a middleware proxy: Generate the first key based on the first key identifier.

[0111] Associate the first key with metadata; the metadata includes the preset number of shards.

[0112] Based on the first key identifier and the fragment identifier, a preset number of second keys are generated; wherein, the fragment identifier of each second key is different.

[0113] According to the association strategy, the member data of the target key is associated with the corresponding second key; among them, the member data associated with each second key together constitute the member data of the target key.

[0114] In some embodiments, the processing module 810 is configured to obtain a first data identifier of the member data of the target key.

[0115] The hash value of the first data identifier is generated using a hash function.

[0116] The hash value of the first data identifier is moduloed by the preset number of fragments to obtain the first modulo value.

[0117] Associate the first modulus value with each fragment identifier to determine the association strategy.

[0118] In some embodiments, the processing module 810 is configured to calculate a first value by using a hash function on a preset string; the first value is a fixed value of the intermediate layer proxy.

[0119] Generate a preset number of second values ​​for each of the predefined number of shards; where each second value has a different value.

[0120] In some embodiments, the processing module 810 is configured to receive a service operation request; the service operation request includes a second key identifier, a second data identifier, and a service operation type.

[0121] Based on the business operation request, the following indexing operation is performed through the middleware proxy: If the second key identifier matches the first key identifier, then retrieve the metadata associated with the first key.

[0122] Based on the first key identifier, the first value, the second data identifier, and the preset number of fragments included in the metadata, the target second key is determined through a preset assembly strategy.

[0123] Data operations are performed based on the target's second key and the business operation type to obtain the data operation results.

[0124] In some embodiments, the processing module 810 is configured to generate a hash value for the second data identifier using a hash function.

[0125] The second modulo value is obtained by taking the modulo of the hash value of the second data identifier and the preset number of shards in the metadata.

[0126] Assemble the first key identifier, the first value, and the second modulus value to determine the target second key.

[0127] In some embodiments, the processing module 810 is configured to delete all generated keys if the splitting operation fails; wherein, the splitting operation failure includes failure to generate or associate the first key and any second key.

[0128] In some embodiments, the processing module 810 is configured to store metadata to a local cache via an intermediate layer proxy.

[0129] It should be noted that other corresponding descriptions of the functional units involved in the distributed index construction device provided in this embodiment can be found in the description of the distributed index construction method in the above embodiments, and will not be repeated here.

[0130] Based on the distributed index construction method shown in the above embodiments, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method shown in the above embodiments.

[0131] Based on the methods shown in the above embodiments, this embodiment also provides a computer program product on which a computer program is stored, and when the computer program product is executed by a processor, it implements the methods shown in the above embodiments.

[0132] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.). The storage medium includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0133] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0134] Based on the method shown in the above embodiments, and Figure 8 To achieve the above objectives, as illustrated in the virtual device embodiment, this application also provides an electronic device. This electronic device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to implement the method shown in the above embodiment.

[0135] Optionally, the aforementioned electronic device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optionally, a USB interface, a card reader interface, etc. Optionally, the network interface may include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0136] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. Compared with the current related technologies, this application, based on the splitting processing logic of the intermediate layer proxy, achieves the distributed key transformation of the target key by splitting the target key into a first key associated with metadata and multiple second keys that each carry a portion of the member data. Since multiple second keys can distribute the member data of the target key to different shard nodes in the cluster, it can avoid the uneven memory distribution caused by a single shard node centrally storing a large amount of member data, and make full use of the cluster sharding resources.

[0138] Furthermore, since the amount of member data for a single second key is significantly reduced after splitting, the operation time is greatly shortened, thus effectively avoiding client blocking, breaking through the single-key member data volume limit, and ensuring the stable and efficient operation of the distributed Redis cluster in big data scenarios.

[0139] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0140] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A distributed index construction method, characterized in that, Applications in distributed storage systems include: The target key is split by an intermediate layer proxy to obtain a first key and multiple second keys; wherein, the first key is associated with metadata used to determine the second keys, and each second key is associated with a portion of the member data of the target key.

2. The method according to claim 1, characterized in that, The process of splitting the target key through an intermediate layer proxy to obtain a first key and multiple second keys includes: Receive a distributed creation command; the distributed creation command includes a preset number of shards and a first key identifier for the target key; Based on the distributed creation command, the following splitting operation is performed through the intermediate layer proxy: Generate the first key based on the first key identifier; Associate the first key with metadata; wherein the metadata includes the preset number of shards; Based on the first key identifier and the fragment identifier, the preset number of second keys are generated; wherein, the fragment identifier of each second key is different; According to the association strategy, the member data of the target key is associated with the corresponding second key; wherein, the member data associated with each second key together constitute the member data of the target key.

3. The method according to claim 2, characterized in that, The method of associating the member data of the target key with the corresponding second key according to the association strategy further includes: Obtain the first data identifier of the member data of the target key; The hash value of the first data identifier is generated using a hash function; The hash value of the first data identifier is moduloed by the preset number of fragments to obtain the first modulo value; Associating the first modulus value with each of the fragment identifiers, the association strategy is determined.

4. The method according to claim 2 or 3, characterized in that, The fragment identifier includes a first value and a second value; The method further includes: The first value is generated by calculating a preset string using a hash function; the first value is a fixed value of the intermediate layer proxy. Generate the preset number of second values; wherein each of the second values ​​has a different value.

5. The method according to claim 4, characterized in that, The method further includes: Receive a business operation request; the business operation request includes a second key identifier, a second data identifier, and a business operation type; Based on the business operation request, the following indexing operation is performed through the intermediate layer proxy: If the second key identifier matches the first key identifier, then obtain the metadata associated with the first key; Based on the first key identifier, the first value, the second data identifier, and the preset number of fragments included in the metadata, the target second key is determined through a preset assembly strategy; Data operations are performed based on the target second key and the business operation type to obtain the data operation result.

6. The method according to claim 5, characterized in that, The step of determining the target second key based on the first key identifier, the first value, the second data identifier, and the preset number of fragments included in the metadata, through a preset assembly strategy, includes: The hash value of the second data identifier is generated using a hash function; The hash value of the second data identifier is moduloed by the preset number of fragments in the metadata to obtain the second modulo value; The first key identifier, the first value, and the second modulus value are assembled to determine the target second key.

7. The method according to claim 2 or 3, characterized in that, The method further includes: If the splitting operation fails, all generated keys are deleted; wherein, the splitting operation failure includes failure to generate or associate the first key and any of the second keys.

8. The method according to claim 2 or 3, characterized in that, After associating the first key with metadata, the method further includes: The metadata is stored in a local cache through the intermediate layer proxy.

9. A distributed index building apparatus, characterized in that, The device is used in a distributed storage system, and the device includes a processing module; The processing module is configured to split the target key through an intermediate layer proxy to obtain a first key and multiple second keys; wherein the first key is associated with metadata used to determine the second keys, and each second key is associated with a portion of the member data of the target key.

10. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

Images