A sharding storage method, device and equipment of a Redis cluster and a storage medium
By using hash operations and data offset pointers to shard data storage in the Redis cluster, the problems of memory waste and uneven load in the Redis cluster bitmap storage scheme are solved, achieving more efficient memory utilization and improved system scalability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA PING AN PROPERTY INSURANCE CO LTD
- Filing Date
- 2023-06-29
- Publication Date
- 2026-05-26
Smart Images

Figure CN116842012B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the fields of big data technology and financial insurance, specifically relating to a sharded storage method, device, equipment and storage medium for a Redis cluster. Background Technology
[0002] The Redis cluster bitmap storage scheme refers to a bitmap storage scheme implemented using Redis as the storage engine and its string data type. In this scheme, each bit is used to represent a certain state or identifier, which is suitable for processing massive integer data. For example, in a financial insurance product system, the Redis cluster bitmap storage scheme can be used to count whether a user has logged into the system.
[0003] However, in current Redis cluster bitmap storage solutions, data is usually stored sequentially in a single Redis node. When storing large amounts of data using this method, if the offset of a certain data is large, it will require a large amount of memory and generate a large offset value, resulting in serious memory waste. Furthermore, since all the data is stored on a single node, it is easy to cause uneven node load, weak fault tolerance, and affect the system's performance and scalability. Summary of the Invention
[0004] The purpose of this application is to propose a sharded storage method, apparatus, computer device, and storage medium for Redis clusters, so as to solve the technical problems of memory waste, uneven load, weak fault tolerance, and impact on system performance and scalability of the current Redis cluster bitmap storage scheme.
[0005] To address the aforementioned technical problems, this application provides a sharded storage method for a Redis cluster, employing the following technical solution:
[0006] A sharded storage method for a Redis cluster includes:
[0007] Receive data storage instructions and obtain the data to be stored corresponding to the data storage instructions;
[0008] Perform a hash operation on the data to be stored to obtain the hash value corresponding to the data to be stored, and get the first hash value;
[0009] The first hash value is divided into a first hash fragment and a second hash fragment;
[0010] Based on the first hash fragment, index the hash slot corresponding to the data to be stored in the Redis cluster to obtain the first hash slot;
[0011] Determine the data offset value of the second hash segment in the hash slot to obtain the first data offset value;
[0012] The storage bitmap position of the data to be stored in the Redis cluster is determined based on the first hash slot and the first data offset value. The storage state of the first storage bitmap position is then changed.
[0013] Furthermore, the storage state includes a first storage state and a second storage state. The storage bitmap position of the data to be stored in the Redis cluster is determined based on the first hash slot and the first data offset value to obtain the first storage bitmap position. The storage state of the first storage bitmap position is then changed, specifically including:
[0014] Generate a data offset pointer, and use the data offset pointer to traverse the first hash slot based on the first data offset value;
[0015] The storage bitmap position of the data to be stored in the Redis cluster is determined based on the traversal results of the data offset pointer, thus obtaining the first storage bitmap position;
[0016] Change the storage state of the first storage bitmap location from the first storage state to the second storage state.
[0017] Furthermore, after determining the location of the bitmap to be stored based on the first hash slot and the first data offset value, and changing the storage state of the bitmap location, the method further includes:
[0018] Receive data query instructions and retrieve the data to be queried corresponding to the data query instructions;
[0019] Perform a hash operation on the data to be queried to obtain the hash value corresponding to the data to be queried, and get the second hash value;
[0020] The second hash value is divided to obtain the third hash fragment and the fourth hash fragment;
[0021] Based on the third hash fragment, the second hash slot is obtained by indexing the hash slot corresponding to the data to be queried in the Redis cluster.
[0022] Determine the data offset value of the fourth hash segment in the hash slot to obtain the second data offset value;
[0023] The storage bitmap position of the data to be queried in the Redis cluster is determined based on the second hash slot and the second data offset value, thus obtaining the second storage bitmap position;
[0024] The storage status of the second storage bitmap location is detected, and the storage status detection result is used to determine whether the data to be queried has been stored as a bitmap.
[0025] Furthermore, the storage status of the second storage bitmap location is detected, and based on the storage status detection result, it is determined whether the data to be queried has been successfully stored as a bitmap. Specifically, this includes:
[0026] If the storage status of the second storage bitmap location is the first storage status, then it is determined that the data to be queried has not been fully stored in the bitmap.
[0027] Generate a first data query result based on the first storage state, and output the first data query result;
[0028] If the storage status of the second storage bitmap location is the second storage status, then it is determined that the data to be queried has been stored in the bitmap.
[0029] The second data query result is generated based on the second storage state and then output.
[0030] Furthermore, the first hash value is divided into a first hash fragment and a second hash fragment, specifically including:
[0031] Determine the index length of the hash slot in the Redis cluster;
[0032] The first hash value is obtained by performing bitwise operations based on the index length of the hash slot;
[0033] The second hash segment is obtained by performing a mask operation on the first hash value based on the first hash segment.
[0034] Furthermore, bitwise operations are performed on the first hash value based on the index length of the hash slot to obtain the first hash fragment, specifically including:
[0035] Determine the starting bit of the first hash value;
[0036] Starting from the beginning of the first hash value, perform a bit-by-bit right shift operation on the first hash value until the right shift length is equal to the index length of the hash slot, thus obtaining the first hash segment.
[0037] To address the aforementioned technical problems, this application also provides a sharded storage device for a Redis cluster, employing the following technical solution:
[0038] A sharded storage device for a Redis cluster, comprising:
[0039] The storage instruction receiving module is used to receive data storage instructions and obtain the data to be stored corresponding to the data storage instructions;
[0040] The first hash operation module is used to perform hash operation on the data to be stored, obtain the hash value corresponding to the data to be stored, and obtain the first hash value.
[0041] The first hash partitioning module is used to partition the first hash value to obtain the first hash fragment and the second hash fragment;
[0042] The first slot index module is used to index the hash slot corresponding to the data to be stored in the Redis cluster based on the first hash fragment, so as to obtain the first hash slot;
[0043] The first data offset module is used to determine the data offset value of the second hash fragment in the hash slot, and obtain the first data offset value;
[0044] The data bitmap storage module is used to determine the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtain the first storage bitmap position, and change the storage status of the first storage bitmap position.
[0045] Furthermore, the sharded storage device of a Redis cluster also includes:
[0046] The query instruction receiving module is used to receive data query instructions and obtain the data to be queried corresponding to the data query instructions;
[0047] The second hash operation module is used to perform hash operation on the data to be queried, obtain the hash value corresponding to the data to be queried, and obtain the second hash value.
[0048] The second hash partitioning module is used to partition the second hash value to obtain the third hash fragment and the fourth hash fragment;
[0049] The second slot index module is used to index the hash slot corresponding to the data to be queried in the Redis cluster based on the third hash fragment, and obtain the second hash slot.
[0050] The second data offset module is used to determine the data offset value of the fourth hash segment in the hash slot, and obtain the second data offset value.
[0051] The bitmap data query module is used to determine the storage bitmap position of the data to be queried in the Redis cluster based on the second hash slot and the second data offset value, and obtain the second storage bitmap position.
[0052] The storage status detection module is used to detect the storage status of the second storage bitmap location and determine whether the data to be queried has been stored as a bitmap based on the storage status detection result.
[0053] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:
[0054] A computer device includes a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the sharded storage method for a Redis cluster as described in any of the preceding claims.
[0055] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:
[0056] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the Redis cluster sharding storage method as described in any of the preceding descriptions.
[0057] Compared with the prior art, the embodiments of this application have the following main advantages:
[0058] This application discloses a sharded storage method, apparatus, device, and storage medium for a Redis cluster, belonging to the fields of big data technology and financial insurance. This application involves receiving a data storage instruction, obtaining the data to be stored corresponding to the instruction, performing a hash operation on the data to be stored to obtain a hash value, obtaining a first hash value, dividing the first hash value to obtain a first hash fragment and a second hash fragment, indexing the hash slot corresponding to the data to be stored in the Redis cluster based on the first hash fragment to obtain the first hash slot, determining the data offset value of the second hash fragment in the hash slot to obtain a first data offset value, determining the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtaining the first storage bitmap position, and changing the storage state of the first storage bitmap position. This application calculates the hash value of the data, indexes the hash slot in the Redis cluster using the hash value, and determines the data offset value using the hash value. Then, the data is evenly distributed across different Redis cluster nodes to achieve data sharding storage. When a large amount of data needs to be stored, sharding storage can evenly distribute the data across multiple nodes, avoiding excessive memory consumption on a single node and reducing the generation of gap values, thereby improving memory utilization efficiency and enhancing the scalability and load balancing of the system. Attached Figure Description
[0059] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0060] Figure 1An exemplary system architecture diagram is shown, in which this application can be applied;
[0061] Figure 2 A flowchart illustrating an embodiment of the sharded storage method for a Redis cluster according to this application is shown;
[0062] Figure 3 A schematic diagram of the structure of one embodiment of the sharded storage device for a Redis cluster according to this application is shown;
[0063] Figure 4 A schematic diagram of the structure of one embodiment of a computer device according to this application is shown. Detailed Implementation
[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.
[0065] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0067] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0068] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.
[0069] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.
[0070] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0071] It should be noted that the Redis cluster sharding storage method provided in this application embodiment is generally executed by the server, and correspondingly, the Redis cluster sharding storage device is generally set in the server.
[0072] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0073] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the sharded storage method for a Redis cluster according to this application. Embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0074] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.
[0075] Redis (Remote Dictionary Server) is an open-source in-memory data storage system, also known as a key-value database, cache, and message queue. It stores data in key-value pairs in memory for fast data access and read / write operations. Redis supports various data structures, including strings, hashes, lists, sets, and sorted sets. Each data structure has corresponding operation commands for performing CRUD (Create, Read, Update, Delete) operations.
[0076] The Redis cluster bitmap storage scheme refers to a bitmap storage scheme implemented using Redis as the storage engine and its string data type. In this scheme, each bit is used to represent a certain state or identifier, which is suitable for processing massive integer data. For example, in a financial insurance product system, the Redis cluster bitmap storage scheme can be used to count whether a user has logged into the system.
[0077] In current Redis cluster bitmap storage solutions, data is typically stored sequentially on a single Redis node. This fails to achieve load balancing, as all data is concentrated on a single node, easily leading to uneven node load, impacting system performance and scalability, and increasing the risk of data loss or unavailability. When storing large amounts of data using this solution, if a data item has a large offset, it requires significant memory, resulting in large offset values and severe memory waste. Furthermore, storing all data on a single node easily leads to uneven node load, weak fault tolerance, and negatively impacts system performance and scalability.
[0078] To address the aforementioned technical issues, this application discloses a sharded storage method, apparatus, device, and storage medium for a Redis cluster, belonging to the fields of big data technology and financial insurance. This application calculates the hash value of the data, indexes hash slots in the Redis cluster using the hash value, and determines the data offset value using the hash value. Then, the data is evenly distributed across different Redis cluster nodes to achieve sharded storage. When storing a large amount of data, sharded storage can evenly distribute the data across multiple nodes, avoiding excessive memory consumption on a single node and reducing gap values, thereby improving memory utilization efficiency and enhancing system scalability and load balancing.
[0079] In the Redis cluster bitmap storage scheme, sharding storage refers to dividing and distributing data to different Redis nodes according to certain rules or algorithms. This sharding storage method can achieve distributed data storage and load balancing, improving the system's scalability and concurrent processing capabilities.
[0080] Specifically, in a specific embodiment of this application, the process of fragmented storage is as follows:
[0081] Data is sharded using a hash algorithm: For each piece of data to be stored, a hash value is calculated by applying a hash algorithm. This hash value can be an integer or a string. Hash algorithms typically map input data to a fixed-length hash code.
[0082] Selecting storage nodes based on hash values: Hash values are mapped to hash slots in the Redis cluster. A hash slot is the basic unit used to distribute data in a Redis cluster. It is usually a fixed number of slots, such as the default 65536 slots. Data is mapped to the corresponding slots based on the high or low bits of the hash value.
[0083] Data is stored on the corresponding nodes: Based on the hash slot mapping results, the data is stored in the corresponding Redis nodes. Each node is responsible for managing a portion of the hash slots and maintaining the corresponding data.
[0084] By distributing data across multiple nodes, nodes can be easily added or removed to meet changing storage needs. Each node only needs to manage a portion of the data, reducing the load on a single node, improving system performance and throughput. Furthermore, because data is distributed across multiple nodes, if one node fails or becomes unavailable, data can still be retrieved from other available nodes, enhancing system availability and fault tolerance.
[0085] The Redis cluster sharding storage method includes the following steps:
[0086] S201, Receive data storage instruction, and obtain the data to be stored corresponding to the data storage instruction.
[0087] In this embodiment, after receiving a data storage instruction, the server parses the data storage instruction and retrieves the data to be stored from a preset database according to the data storage instruction. The data storage instruction may come from an application or other system components.
[0088] Furthermore, the storage state includes a first storage state and a second storage state. The storage bitmap position of the data to be stored in the Redis cluster is determined based on the first hash slot and the first data offset value to obtain the first storage bitmap position. The storage state of the first storage bitmap position is then changed, specifically including:
[0089] Generate a data offset pointer, and use the data offset pointer to traverse the first hash slot based on the first data offset value;
[0090] The storage bitmap position of the data to be stored in the Redis cluster is determined based on the traversal results of the data offset pointer, thus obtaining the first storage bitmap position;
[0091] Change the storage state of the first storage bitmap location from the first storage state to the second storage state.
[0092] In this embodiment, the data offset pointer indicates the specific storage location of the data to be stored within the node. Its value is calculated based on the first data offset value. The final data offset pointer is obtained by multiplying the data offset value by the storage space occupied by each piece of data in the node. In Redis, each piece of data is stored in a fixed size, so the data offset pointer can be calculated by multiplying by the size of each piece of data. For example, assuming each piece of data is N bytes and the data offset value is M, the formula for calculating the data offset pointer is: M*N. The calculated result is the specific storage location of the data to be stored within the node, which can be used to traverse hash slots and determine the storage bitmap position.
[0093] The traversal result refers to the specific position of the data offset pointer in the hash slot. Based on this position, the storage location of the data to be stored in the node can be determined. The storage bitmap position is a mechanism used to record the storage state of the node. Its value represents the storage state of a certain position in the node. For example, the first storage state is configured as "0" and the second storage state is configured as "1". The first storage state indicates that the data is not stored, and the second storage state indicates that the data has been stored. Changing the storage state of the first storage bitmap position from the first storage state to the second storage state, such as changing the storage state of the first storage bitmap position from "0" to "1", means that the storage location of the data to be stored is marked as used, so as to facilitate subsequent data reading and management.
[0094] S202, perform a hash operation on the data to be stored to obtain the hash value corresponding to the data to be stored, and obtain the first hash value.
[0095] In this embodiment, a suitable hash function is selected to map the data to be stored to an integer or a string. Common hash functions include MD5, SHA1, and MurmurHash. By performing a hash operation on the data to be stored, the hash value corresponding to the data to be stored is obtained, resulting in a first hash value. The first hash value can be used for hash slot indexing and determining the data offset, so as to determine the storage bitmap position of the data to be stored in the Redis cluster.
[0096] S203, divide the first hash value to obtain the first hash fragment and the second hash fragment.
[0097] In this embodiment, the first hash value can be divided using bitwise operations and masking operations. The division process can be implemented according to a specific hash algorithm, with the aim of dividing the hash value into two parts to facilitate the subsequent determination of hash slot indexes and data offsets. The first hash segment is the high-order hash segment of the first hash value, and the second hash segment is the low-order hash segment of the first hash value.
[0098] Furthermore, the first hash value is divided into a first hash fragment and a second hash fragment, specifically including:
[0099] Determine the index length of the hash slot in the Redis cluster;
[0100] The first hash value is obtained by performing bitwise operations based on the index length of the hash slot;
[0101] The second hash segment is obtained by performing a mask operation on the first hash value based on the first hash segment.
[0102] Furthermore, bitwise operations are performed on the first hash value based on the index length of the hash slot to obtain the first hash fragment, specifically including:
[0103] Determine the starting bit of the first hash value;
[0104] Starting from the beginning of the first hash value, perform a bit-by-bit right shift operation on the first hash value until the right shift length is equal to the index length of the hash slot, thus obtaining the first hash segment.
[0105] When indexing the hash slots corresponding to stored data in a Redis cluster, the hash fragment used is configurable. Redis allows changing the number of bits in the hash fragment by modifying the configuration. Specifically, the hash-bits configuration option can be used to specify the number of bits in the hash fragment. This option can be set to 16, 14, 12, or 10, indicating that a 16-bit, 14-bit, 12-bit, or 10-bit hash fragment is used to determine the slot index, respectively.
[0106] It should be noted that, by default, Redis clusters use the high 16 bits of the hash fragment as the slot index, that is, a 16-bit hash fragment is used to determine the slot to which the data belongs.
[0107] In this embodiment, taking the index length of 10 hash slots as an example, if hash is a 32-bit integer, it can be represented as a 32-bit binary number. The hash value is divided into high 10 bits and low 22 bits through bitwise operations. Starting from the starting bit of the hash value, the hash value is shifted 10 bits to the right through a right shift operation to obtain the value of the high 10 bits. The high 22 bits are masked by using a bitmask (0x3FFFFF) to retain the value of the low 22 bits. The offset value can be calculated by the following formula: offset = hash & 0x3FFFFF.
[0108] It's important to note that modifying the `hash-bits` configuration option will affect the hash slot allocation and data mapping method in the Redis cluster. Therefore, before modifying the hash bit length, careful evaluation and planning are necessary, and it's crucial to ensure consistent configuration across all nodes to avoid inconsistent data distribution and access anomalies.
[0109] S204: Based on the first hash fragment, index the hash slot corresponding to the data to be stored in the Redis cluster to obtain the first hash slot.
[0110] In this embodiment, the first hash slot is obtained by indexing the hash slot corresponding to the data to be stored in the Redis cluster based on the first hash fragment. The high-order hash fragment extracted from the hash value can be used as the basis for slot location, thus distributing the data across different slots and achieving sharded storage. Sharded storage can maximize memory utilization and avoid wasting memory resources. Through a good data sharding strategy, the amount of data stored on each node can be balanced, and redundant storage can be minimized, thereby reducing the waste of memory resources.
[0111] S205, determine the data offset value of the second hash segment in the hash slot, and obtain the first data offset value.
[0112] In this embodiment, the server determines the data offset value of the second hash fragment in the hash slot to obtain the first data offset value. Assuming the slot position is "slot", and data N needs to be stored in the Redis cluster with the corresponding node's slot position "slot", the low-order hash fragment of the hash value corresponding to data N can be used as the identifier bit in the Bitmap. As needed, the storage status setting of the identifier in the Bitmap can be changed from "0" to "1" to indicate the storage of data N.
[0113] S206. Determine the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtain the first storage bitmap position, and change the storage status of the first storage bitmap position.
[0114] In this embodiment, the storage bitmap position of the data to be stored in the Redis cluster is determined based on the first hash slot and the first data offset value, i.e., the identifier bit in the Bitmap. This yields the first storage bitmap position, and its storage state is changed to represent the storage of the data to be stored. The Bitmap is a mechanism used by Redis to record the storage state of each node. It divides each node into several storage bitmap positions, each corresponding to a data storage state. The purpose here is to record the storage position of the data to be stored in the storage bitmap and change its storage state for subsequent data reading and management.
[0115] In the above embodiments, this application calculates the hash value of the data, indexes the hash slot in the Redis cluster using the hash value, and determines the data offset value using the hash value. Then, the data is evenly distributed to different Redis cluster nodes to achieve data sharding storage. When a large amount of data needs to be stored, sharding storage can evenly distribute the data to multiple nodes, avoiding excessive memory consumption of a single node and reducing the generation of gap values, thereby improving memory utilization efficiency and enhancing the scalability and load balancing of the system.
[0116] Furthermore, after determining the location of the bitmap to be stored based on the first hash slot and the first data offset value, and changing the storage state of the bitmap location, the method further includes:
[0117] Receive data query instructions and retrieve the data to be queried corresponding to the data query instructions;
[0118] Perform a hash operation on the data to be queried to obtain the hash value corresponding to the data to be queried, and get the second hash value;
[0119] The second hash value is divided to obtain the third hash fragment and the fourth hash fragment;
[0120] Based on the third hash fragment, the second hash slot is obtained by indexing the hash slot corresponding to the data to be queried in the Redis cluster.
[0121] Determine the data offset value of the fourth hash segment in the hash slot to obtain the second data offset value;
[0122] The storage bitmap position of the data to be queried in the Redis cluster is determined based on the second hash slot and the second data offset value, thus obtaining the second storage bitmap position;
[0123] The storage status of the second storage bitmap location is detected, and the storage status detection result is used to determine whether the data to be queried has been stored as a bitmap.
[0124] When it is necessary to query whether a certain data exists, the same hash function is used to calculate its hash value, the high-order hash fragment is taken out as the basis for the slot position, and the low-order hash fragment is used to determine the data offset in order to find the corresponding node, slot position and the identifier in the Bitmap, and then the storage status of the corresponding identifier in the Bitmap is queried.
[0125] In this embodiment, when the server receives a data query instruction, it obtains the data to be queried corresponding to the data query instruction, performs a hash operation on the data to be queried using the same hash function, obtains the hash value corresponding to the data to be queried, and obtains a second hash value. The second hash value is divided using the same partitioning rule to obtain a third hash fragment and a fourth hash fragment. Based on the third hash fragment, the hash slot corresponding to the data to be queried is indexed in the Redis cluster to obtain a second hash slot. The data offset value of the fourth hash fragment in the hash slot is determined to obtain a second data offset value. Based on the second hash slot and the second data offset value, the storage bitmap position of the data to be queried in the Redis cluster and the identifier bit in the Bitmap are determined to obtain a second storage bitmap position. The storage status of the second storage bitmap position is detected, and the storage status detection result is used to determine whether the data to be queried has been stored in the bitmap.
[0126] Furthermore, the storage status of the second storage bitmap location is detected, and based on the storage status detection result, it is determined whether the data to be queried has been successfully stored as a bitmap. Specifically, this includes:
[0127] If the storage status of the second storage bitmap location is the first storage status, then it is determined that the data to be queried has not been fully stored in the bitmap.
[0128] Generate a first data query result based on the first storage state, and output the first data query result;
[0129] If the storage status of the second storage bitmap location is the second storage status, then it is determined that the data to be queried has been stored in the bitmap.
[0130] The second data query result is generated based on the second storage state and then output.
[0131] In this embodiment, if the storage state of the second storage bitmap location is the first storage state, it is determined that the data to be queried has not been completely stored in the bitmap. A first data query result is generated based on the first storage state and output, indicating that the data to be queried has not been stored. If the storage state of the second storage bitmap location is the second storage state, it is determined that the data to be queried has been completely stored in the bitmap. A second data query result is generated based on the second storage state and output, indicating that the data to be queried has been stored.
[0132] In this embodiment, the present application calculates the hash value of the data, indexes the hash slot in the Redis cluster using the hash value, and determines the data offset value using the hash value. Then, the storage status of the data is quickly queried using the hash slot and the data offset value, thereby improving the data query efficiency of the Redis cluster.
[0133] In the above embodiments, this application discloses a sharded storage method for a Redis cluster, belonging to the fields of big data technology and financial insurance. This application receives a data storage instruction, obtains the data to be stored corresponding to the data storage instruction, performs a hash operation on the data to be stored to obtain a hash value corresponding to the data to be stored, obtains a first hash value, divides the first hash value to obtain a first hash fragment and a second hash fragment, indexes the hash slot corresponding to the data to be stored in the Redis cluster based on the first hash fragment, obtains the first hash slot, determines the data offset value of the second hash fragment in the hash slot, obtains a first data offset value, determines the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtains the first storage bitmap position, and changes the storage state of the first storage bitmap position. This application achieves data sharding storage by calculating the hash value of the data, indexing hash slots in the Redis cluster using the hash value, and determining the data offset value using the hash value. The data is then evenly distributed across different Redis cluster nodes. When storing large amounts of data, sharding distributes the data evenly across multiple nodes, avoiding excessive memory consumption on a single node and reducing gaps, thus improving memory utilization efficiency and enhancing system scalability and load balancing. Furthermore, this application also improves the efficiency of Redis cluster data retrieval by calculating the hash value of the data, indexing hash slots in the Redis cluster using the hash value, and determining the data offset value using the hash value. The hash slot and data offset value then allow for quick querying of the data's storage status.
[0134] In this embodiment, the Redis cluster's sharded storage method runs on electronic devices (e.g., Figure 1 The server shown can receive instructions or acquire data via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future wireless connection methods.
[0135] It should be emphasized that, to further ensure the privacy and security of the data to be stored, the data can also be stored in a node of a blockchain.
[0136] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium, and when executed, they can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).
[0138] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0139] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a sharded storage device for a Redis cluster, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0140] like Figure 3 As shown, the Redis cluster sharded storage device 300 described in this embodiment includes:
[0141] The storage instruction receiving module 301 is used to receive data storage instructions and obtain the data to be stored corresponding to the data storage instructions.
[0142] The first hash operation module 302 is used to perform hash operation on the data to be stored, obtain the hash value corresponding to the data to be stored, and obtain the first hash value.
[0143] The first hash partitioning module 303 is used to partition the first hash value to obtain a first hash fragment and a second hash fragment;
[0144] The first slot index module 304 is used to index the hash slot corresponding to the data to be stored in the Redis cluster based on the first hash fragment, and obtain the first hash slot.
[0145] The first data offset module 305 is used to determine the data offset value of the second hash segment in the hash slot and obtain the first data offset value.
[0146] The data bitmap storage module 306 is used to determine the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtain the first storage bitmap position, and change the storage state of the first storage bitmap position.
[0147] Furthermore, the storage states include a first storage state and a second storage state, and the data bitmap storage module 306 specifically includes:
[0148] Generate a data offset pointer, and use the data offset pointer to traverse the first hash slot based on the first data offset value;
[0149] The bitmap position determination unit is used to determine the storage bitmap position of the data to be stored in the Redis cluster based on the traversal result of the data offset pointer, and obtain the first storage bitmap position;
[0150] The storage state change unit is used to change the storage state of the first storage bitmap location from the first storage state to the second storage state.
[0151] Furthermore, the sharded storage device of a Redis cluster also includes:
[0152] The query instruction receiving module is used to receive data query instructions and obtain the data to be queried corresponding to the data query instructions;
[0153] The second hash operation module is used to perform hash operation on the data to be queried, obtain the hash value corresponding to the data to be queried, and obtain the second hash value.
[0154] The second hash partitioning module is used to partition the second hash value to obtain the third hash fragment and the fourth hash fragment;
[0155] The second slot index module is used to index the hash slot corresponding to the data to be queried in the Redis cluster based on the third hash fragment, and obtain the second hash slot.
[0156] The second data offset module is used to determine the data offset value of the fourth hash segment in the hash slot, and obtain the second data offset value.
[0157] The bitmap data query module is used to determine the storage bitmap position of the data to be queried in the Redis cluster based on the second hash slot and the second data offset value, and obtain the second storage bitmap position.
[0158] The storage status detection module is used to detect the storage status of the second storage bitmap location and determine whether the data to be queried has been stored as a bitmap based on the storage status detection result.
[0159] Furthermore, the storage status detection module specifically includes:
[0160] The first inspection result unit is used to determine that the data to be queried has not been completely stored in the bitmap when the storage state of the second storage bitmap position is the first storage state.
[0161] The first query result output unit is used to generate a first data query result based on a first storage state and output the first data query result;
[0162] The second inspection result unit is used to determine that the data to be queried has been stored in the bitmap when the storage state of the second storage bitmap location is the second storage state.
[0163] The second query result output unit is used to generate a second data query result based on the second storage state and output the second data query result.
[0164] Furthermore, the first hash partitioning module 303 specifically includes:
[0165] The index length determination unit is used to determine the index length of a hash slot in a Redis cluster.
[0166] The hash bit operation unit is used to perform bit operations on the first hash value based on the index length of the hash slot to obtain the first hash fragment;
[0167] The hash mask operation unit is used to perform a mask operation on the first hash value based on the first hash fragment to obtain the second hash fragment.
[0168] Furthermore, the hash bit operation unit specifically includes:
[0169] The start bit determination sub-unit is used to determine the start bit of the first hash value;
[0170] The bit-by-bit right shift subunit is used to perform a bit-by-bit right shift operation on the first hash value, starting from the starting bit, until the right shift length is equal to the index length of the hash slot, thus obtaining the first hash segment.
[0171] In the above embodiments, this application discloses a sharded storage device for a Redis cluster, belonging to the fields of big data technology and financial insurance. This application receives a data storage instruction, obtains the data to be stored corresponding to the data storage instruction, performs a hash operation on the data to be stored to obtain a hash value corresponding to the data to be stored, obtains a first hash value, divides the first hash value to obtain a first hash fragment and a second hash fragment, indexes the hash slot corresponding to the data to be stored in the Redis cluster based on the first hash fragment, obtains the first hash slot, determines the data offset value of the second hash fragment in the hash slot, obtains a first data offset value, determines the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtains the first storage bitmap position, and changes the storage state of the first storage bitmap position. This application achieves data sharding storage by calculating the hash value of the data, indexing hash slots in the Redis cluster using the hash value, and determining the data offset value using the hash value. The data is then evenly distributed across different Redis cluster nodes. When storing large amounts of data, sharding distributes the data evenly across multiple nodes, avoiding excessive memory consumption on a single node and reducing gaps, thus improving memory utilization efficiency and enhancing system scalability and load balancing. Furthermore, this application also improves the efficiency of Redis cluster data retrieval by calculating the hash value of the data, indexing hash slots in the Redis cluster using the hash value, and determining the data offset value using the hash value. The hash slot and data offset value then allow for quick querying of the data's storage status.
[0172] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.
[0173] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0174] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.
[0175] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for the sharded storage method of Redis cluster. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.
[0176] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions for the sharded storage method of the Redis cluster.
[0177] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.
[0178] In the above embodiments, this application discloses a computer device belonging to the fields of big data technology and financial insurance. This application receives a data storage instruction, obtains the data to be stored corresponding to the data storage instruction, performs a hash operation on the data to be stored to obtain a hash value corresponding to the data to be stored, obtains a first hash value, divides the first hash value to obtain a first hash fragment and a second hash fragment, indexes the hash slot corresponding to the data to be stored in the Redis cluster based on the first hash fragment to obtain the first hash slot, determines the data offset value of the second hash fragment in the hash slot to obtain a first data offset value, determines the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtains the first storage bitmap position, and changes the storage state of the first storage bitmap position. This application achieves data sharding storage by calculating the hash value of the data, indexing hash slots in the Redis cluster using the hash value, and determining the data offset value using the hash value. The data is then evenly distributed across different Redis cluster nodes. When storing large amounts of data, sharding distributes the data evenly across multiple nodes, avoiding excessive memory consumption on a single node and reducing gaps, thus improving memory utilization efficiency and enhancing system scalability and load balancing. Furthermore, this application also improves the efficiency of Redis cluster data retrieval by calculating the hash value of the data, indexing hash slots in the Redis cluster using the hash value, and determining the data offset value using the hash value. The hash slot and data offset value then allow for quick querying of the data's storage status.
[0179] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the Redis cluster sharding storage method described above.
[0180] In the above embodiments, this application discloses a computer-readable storage medium, belonging to the fields of big data technology and financial insurance. This application receives a data storage instruction, obtains the data to be stored corresponding to the data storage instruction, performs a hash operation on the data to be stored to obtain a hash value corresponding to the data to be stored, obtains a first hash value, divides the first hash value to obtain a first hash fragment and a second hash fragment, indexes the hash slot corresponding to the data to be stored in the Redis cluster based on the first hash fragment to obtain the first hash slot, determines the data offset value of the second hash fragment in the hash slot to obtain a first data offset value, determines the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtains the first storage bitmap position, and changes the storage state of the first storage bitmap position. This application achieves data sharding storage by calculating the hash value of the data, indexing hash slots in the Redis cluster using the hash value, and determining the data offset value using the hash value. The data is then evenly distributed across different Redis cluster nodes. When storing large amounts of data, sharding distributes the data evenly across multiple nodes, avoiding excessive memory consumption on a single node and reducing gaps, thus improving memory utilization efficiency and enhancing system scalability and load balancing. Furthermore, this application also improves the efficiency of Redis cluster data retrieval by calculating the hash value of the data, indexing hash slots in the Redis cluster using the hash value, and determining the data offset value using the hash value. The hash slot and data offset value then allow for quick querying of the data's storage status.
[0181] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0182] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0183] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.
Claims
1. A sharded storage method for a Redis cluster, characterized in that, include: Receive a data storage instruction and obtain the data to be stored corresponding to the data storage instruction; Perform a hash operation on the data to be stored to obtain the hash value corresponding to the data to be stored, and obtain the first hash value; The first hash value is divided to obtain a first hash fragment and a second hash fragment; Based on the first hash fragment, the hash slot corresponding to the data to be stored is indexed in the Redis cluster to obtain the first hash slot; Determine the data offset value of the second hash segment in the hash slot to obtain the first data offset value; The storage bitmap position of the data to be stored in the Redis cluster is determined based on the first hash slot and the first data offset value, the first storage bitmap position is obtained, and the storage status of the first storage bitmap position is changed. The step of dividing the first hash value to obtain a first hash fragment and a second hash fragment specifically includes: Determine the index length of the hash slots in the Redis cluster; The first hash value is bitwise operated on based on the index length of the hash slot to obtain the first hash fragment; The second hash segment is obtained by performing a mask operation on the first hash value based on the first hash segment.
2. The sharded storage method for a Redis cluster as described in claim 1, characterized in that, The storage states include a first storage state and a second storage state. The step of determining the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtaining the first storage bitmap position, and changing the storage state of the first storage bitmap position specifically includes: Generate a data offset pointer, and use the data offset pointer to traverse the first hash slot based on the first data offset value; The storage bitmap position of the data to be stored in the Redis cluster is determined based on the traversal result of the data offset pointer, thus obtaining the first storage bitmap position; Change the storage state of the first storage bitmap location from the first storage state to the second storage state.
3. The Redis cluster sharding storage method as described in claim 2, characterized in that, After determining the location of the bitmap to be stored based on the first hash slot and the first data offset value, and changing the storage state of the bitmap location, the method further includes: Receive a data query instruction and obtain the data to be queried corresponding to the data query instruction; Perform a hash operation on the data to be queried to obtain the hash value corresponding to the data to be queried, and obtain the second hash value; The second hash value is divided to obtain a third hash fragment and a fourth hash fragment; Based on the third hash fragment, the hash slot corresponding to the data to be queried is indexed in the Redis cluster to obtain the second hash slot; Determine the data offset value of the fourth hash segment in the hash slot to obtain the second data offset value; The storage bitmap position of the data to be queried in the Redis cluster is determined based on the second hash slot and the second data offset value, thus obtaining the second storage bitmap position; The storage status of the second storage bitmap location is detected, and the storage status detection result is used to determine whether the data to be queried has been stored as a bitmap.
4. The Redis cluster sharding storage method as described in claim 3, characterized in that, The step of detecting the storage status of the second storage bitmap position and determining whether the data to be queried has been successfully stored as a bitmap based on the storage status detection result specifically includes: If the storage state of the second storage bitmap position is the first storage state, then it is determined that the data to be queried has not been fully stored in the bitmap. Generate a first data query result based on the first storage state, and output the first data query result; If the storage state of the second storage bitmap location is the second storage state, then it is determined that the data to be queried has been stored in the bitmap. A second data query result is generated based on the second storage state, and the second data query result is output.
5. The sharded storage method for a Redis cluster as described in claim 1, characterized in that, The step of performing bitwise operations on the first hash value based on the index length of the hash slot to obtain the first hash fragment specifically includes: Determine the starting bit of the first hash value; Starting from the beginning of the first hash value, the first hash value is shifted right bit by bit until the shift length is equal to the index length of the hash slot, thus obtaining the first hash segment.
6. A sharded storage device for a Redis cluster, characterized in that, include: A storage instruction receiving module is used to receive a data storage instruction and obtain the data to be stored corresponding to the data storage instruction. The first hash operation module is used to perform a hash operation on the data to be stored, obtain the hash value corresponding to the data to be stored, and obtain the first hash value. The first hash partitioning module is used to partition the first hash value to obtain a first hash fragment and a second hash fragment; The first slot indexing module is used to index the hash slot corresponding to the data to be stored in the Redis cluster based on the first hash fragment, so as to obtain the first hash slot. The first data offset module is used to determine the data offset value of the second hash fragment in the hash slot, and obtain the first data offset value; The data bitmap storage module is used to determine the storage bitmap position of the data to be stored in the Redis cluster based on the first hash slot and the first data offset value, obtain the first storage bitmap position, and change the storage state of the first storage bitmap position. The first hash partitioning module specifically includes: An index length determination unit is used to determine the index length of the hash slot in the Redis cluster; A hash bit operation unit is used to perform bit operations on the first hash value based on the index length of the hash slot to obtain the first hash fragment; The hash mask operation unit is used to perform a mask operation on the first hash value based on the first hash fragment to obtain the second hash fragment.
7. The sharded storage device for a Redis cluster as described in claim 6, characterized in that, The sharded storage device of the Redis cluster also includes: The query instruction receiving module is used to receive data query instructions and obtain the data to be queried corresponding to the data query instructions; The second hash operation module is used to perform a hash operation on the data to be queried, obtain the hash value corresponding to the data to be queried, and obtain the second hash value. The second hash partitioning module is used to partition the second hash value to obtain a third hash fragment and a fourth hash fragment; The second slot index module is used to index the hash slot corresponding to the data to be queried in the Redis cluster based on the third hash fragment, so as to obtain the second hash slot. The second data offset module is used to determine the data offset value of the fourth hash segment in the hash slot, and obtain the second data offset value; The bitmap data query module is used to determine the storage bitmap position of the data to be queried in the Redis cluster based on the second hash slot and the second data offset value, and obtain the second storage bitmap position. The storage status detection module is used to detect the storage status of the second storage bitmap location and determine whether the data to be queried has been stored as a bitmap based on the storage status detection result.
8. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the Redis cluster sharding storage method as described in any one of claims 1 to 5.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the Redis cluster sharding storage method as described in any one of claims 1 to 5.