A data storage method and system for a distributed architecture

Abstract

The application discloses a data storage method and system for a distributed architecture, which comprises the following steps: firstly, acquiring the load of each node in the distributed architecture, and mapping the address of each node in a hash ring according to a preset quantity ratio, wherein the node has a plurality of corresponding virtual nodes in the hash ring; then, mapping all data in the hash ring based on the unique number of each piece of data in the data set to be stored and storing the data in the virtual nodes, wherein when the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to a virtual node with a load lower than a second preset threshold, so that the problem of unbalanced load among nodes during data storage in the distributed architecture can be effectively avoided, and normal execution of data storage is ensured.

Description

Technical Field

[0001] This invention belongs to the field of data storage technology, specifically relating to a data storage method and system for distributed architecture. Background Technology

[0002] With the rapid development of the Internet and artificial intelligence, the scale of data is growing explosively. At the same time, there are higher requirements for data storage and management, such as real-time performance, accuracy and scalability. Distributed architecture, also known as distributed caching system, is specifically designed to process large-scale data and support large-scale cluster computing, providing high reliability, high throughput and simple access methods for distributed storage.

[0003] In a distributed architecture, the randomness strategy for data storage can easily lead to uneven data distribution across nodes, causing load imbalance between cluster nodes and making it difficult for storage tasks to execute normally.

[0004] Therefore, how to avoid the problem of unbalanced load among nodes during data storage in a distributed architecture and ensure the normal execution of data storage is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to solve the technical problem in the prior art that when storing data in a distributed architecture, the load imbalance between nodes is easy to occur, which leads to the inability to store data normally.

[0006] To achieve the above-mentioned technical objectives, in one aspect, the present invention provides a data storage method for a distributed architecture, the method comprising:

[0007] The load of each node in the distributed architecture is obtained, and the addresses of each node are mapped in a hash ring according to a preset ratio. Each node has multiple corresponding virtual nodes in the hash ring.

[0008] Based on the unique number of each data in the dataset to be stored, all data are mapped in the hash ring and stored in the virtual nodes. When the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to a virtual node with a load below a second preset threshold.

[0009] Furthermore, the step of mapping all data in the hash ring and storing it within the virtual node based on the unique number of each data item in the dataset to be stored specifically includes:

[0010] The hash value of each piece of data is determined based on its unique identifier.

[0011] Based on the hash value, the corresponding data is mapped to the hash ring, and when the hash value corresponds to the hash position of a certain virtual node, the data corresponding to the hash value is stored in the virtual node corresponding to the hash value. Specifically, storing in the virtual node means storing in the node corresponding to the virtual node.

[0012] Furthermore, the hash ring is divided into multiple independent intervals, and each interval contains a virtual node corresponding to a node.

[0013] Furthermore, the method also includes partitioning the node into data, specifically including:

[0014] Define a state space, an action space, and a reward function. The state space reflects the state of data partitions and the state of load in a node. The action space is a set of actions, specifically including partitioning operations, data copying operations, and data migration operations. The reward function is the feedback reward after the agent takes any action.

[0015] Based on the state space, determine the current state s, and determine the reward at the next moment after taking action a in the current state s;

[0016] The total revenue of all rewards is determined based on a Markov reward chain;

[0017] The behavioral value function is determined based on the sum of the benefits, and the optimal data partitioning strategy is determined based on the behavioral value function.

[0018] Furthermore, the method also includes: generating a change record when data updates occur in each of the nodes, and saving the change record and metadata.

[0019] Furthermore, the method also includes synchronizing the data of each node through a preset control mechanism and handling concurrent write conflicts of each node.

[0020] On the other hand, the present invention also provides a data storage system for a distributed architecture, the system comprising:

[0021] The hash module is used to obtain the load of each node in the distributed architecture and map the address of each node in the hash ring according to a preset ratio. Each node has multiple corresponding virtual nodes in the hash ring.

[0022] The storage module is used to map all data in the hash ring based on the unique number of each data in the dataset to be stored and store it in the virtual node. When the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to the virtual node with a load below a second preset threshold.

[0023] This invention provides a data storage method and system for a distributed architecture. Compared with existing technologies, this method first obtains the load of each node in the distributed architecture and maps the addresses of each node in a hash ring according to a preset ratio. Each node has multiple corresponding virtual nodes in the hash ring. Then, based on the unique number of each piece of data in the dataset to be stored, all data are mapped in the hash ring and stored in the virtual nodes. When the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to a virtual node with a load below a second preset threshold. This effectively avoids the problem of unbalanced load among nodes during data storage in a distributed architecture and ensures the normal execution of data storage.

[0024] This method also provides a method to determine the optimal data partitioning strategy through reinforcement learning, which can reduce the cost of cross-regional communication. It is highly adaptive and can automatically optimize data partitioning based on real-time data load and system status. The optimization effect is better, effectively dealing with complex data distribution and uncertainty, thereby improving data efficiency, reducing resource waste, and improving system performance. It has a strong ability to handle large-scale data partitioning problems, with higher computational efficiency and more accurate strategies. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 The diagram shown is a flowchart illustrating a data storage method for a distributed architecture provided in an embodiment of this specification.

[0027] Figure 2 The diagram shown is a structural schematic of a data storage system for a distributed architecture provided in an embodiment of this specification. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] like Figure 1The diagram illustrates a data storage method for a distributed architecture provided in an embodiment of this specification. While this specification provides the method operation steps or apparatus structure shown in the following embodiments or figures, based on conventional methods or without creative effort, the method or apparatus may include more or fewer operation steps or module units after partial merging. In steps or structures where there is no logically necessary causal relationship, the execution order of these steps or the module structure of the apparatus is not limited to the execution order or module structure shown in the embodiments or figures of this specification. When the method or module structure is applied in actual devices, servers, or terminal products, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiments or figures (e.g., in a parallel processor or multi-threaded processing environment, or even in a distributed processing or server cluster implementation environment).

[0030] The data storage methods for distributed architectures provided in the embodiments of this specification are as follows: Figure 1 As shown, the method specifically includes the following steps:

[0031] Step S101: Obtain the load of each node in the distributed architecture, and map the address of each node in the hash ring according to a preset ratio. Each node has multiple corresponding virtual nodes in the hash ring.

[0032] In this embodiment, the hash ring is divided into multiple independent intervals, each interval containing a virtual node corresponding to a node, which can improve the load balancing effect and make the data more evenly distributed; when a new node is added or an old node is removed, the data migration is smoother.

[0033] Specifically, this application employs a consistent hashing algorithm to achieve load balancing among nodes, thereby enabling good horizontal scaling. When adding or deleting nodes, only a small amount of data needs to be relocated within the hash ring, effectively solving the problems of data node changes and data allocation. However, when the number of nodes in the consistent hashing algorithm is too small, data skew can occur due to uneven node distribution, meaning that most data is stored on the same server, while only a small amount is stored on other servers. To avoid this, a virtual node mechanism is introduced into the consistent hashing algorithm. Multiple hashes are calculated for each server node, and a virtual node is placed at each calculation location to reduce data skew.

[0034] More specifically, first, obtain the node's performance parameters, namely the corresponding server memory usage, and calculate the node's server memory utilization rate to obtain the server node's load status. The server memory utilization rate is calculated as follows:

[0035] ;

[0036] Where Pt is the memory utilization rate of server t, CUi is the memory usage of server t, and CTi is the memory capacity of server t.

[0037] The server node IP address is used as input, and the FNV-1a hash function is used to calculate the hash value, which is then mapped onto the hash ring.

[0038] Here, offset_basis is the base offset, which will be used to store the string "chongo<Landon Curt Noll> Using " / \\.. / \\" as input, the hash value calculated using the FNV-0 algorithm is the offset_basis value. The formula for the FNV-0 algorithm is the same as that for FNV-1a, except that the offset_basis in the FNV-0 algorithm is a constant value of 0.

[0039] FNV_Prime represents the FNV hash prime number, calculated using the following formula:

[0040] ;

[0041] When n = 32, When n≥64, When b is an integer, .

[0042] Calculate the virtual node positions on the hash ring. Map the server node's IP address to the hash ring, and then map virtual nodes within the hash ring at a preset ratio of 1:N. The hash ring is divided into N+1 independent intervals, with nodes distributed relatively evenly within each interval. Therefore, in a consistent hash ring with a 1:N virtual node mapping, assuming the initial hash position of a physical node is P, the hash positions of all mapped nodes represented by that node can be represented as follows:

[0043] ;

[0044] Step S102: Based on the unique number of each piece of data in the dataset to be stored, map all the data in the hash ring and store them in the virtual nodes. When the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to a virtual node with a load below a second preset threshold.

[0045] In this embodiment of the application, the step of mapping all data in the hash ring and storing them within the virtual node based on the unique number of each data item in the dataset to be stored specifically includes:

[0046] The hash value of each piece of data is determined based on its unique identifier.

[0047] Based on the hash value, the corresponding data is mapped to the hash ring, and when the hash value corresponds to the hash position of a certain virtual node, the data corresponding to the hash value is stored in the virtual node corresponding to the hash value. Specifically, storing in the virtual node means storing in the node corresponding to the virtual node.

[0048] Specifically, each piece of data to be stored is assigned a unique number to ensure its uniqueness. This number serves as the key, and its hash value is calculated and mapped onto a hash ring. If the hash value of the data corresponds exactly to the hash position of a virtual node on the hash ring, then the data is stored on the server node corresponding to that virtual node, which is also the node in the distributed architecture. Otherwise, the data will search for the first matching server node in a clockwise direction along the hash ring.

[0049] When a request is assigned to a server node with a load exceeding 85%, the request will be forwarded to other servers with lower loads to ensure that the memory usage of each server does not exceed 85%. When the memory usage of all servers exceeds 85%, the system will start new load balancers to distribute the load.

[0050] When a server node fails, the data is relocated to the next node on the hash ring in a clockwise direction. In other words, the data on the failed node is mapped to a new node on the ring.

[0051] When a new node is added to the system, the data between the new node's position and its nearest neighbor in a counter-clockwise direction on the hash ring needs to be remapped to the new node.

[0052] In this embodiment of the application, the method further includes data partitioning of the node, specifically including:

[0053] Define a state space, an action space, and a reward function. The state space reflects the state of data partitions and the state of load in a node. The action space is a set of actions, specifically including partitioning operations, data copying operations, and data migration operations. The reward function is the feedback reward after the agent takes any action.

[0054] Based on the state space, determine the current state s, and determine the reward at the next moment after taking action a in the current state s;

[0055] The total revenue of all rewards is determined based on a Markov reward chain;

[0056] The behavioral value function is determined based on the sum of the benefits, and the optimal data partitioning strategy is determined based on the behavioral value function.

[0057] Specifically, the data partitioning problem can be understood as the process of dividing a dataset into several non-overlapping subsets under specific constraints. During data partitioning, various factors need to be considered, such as the size and complexity of the data, access patterns, computational requirements, and hardware resource limitations.

[0058] Reinforcement learning consists of an agent and an environment. In addition to these two core components, a reinforcement learning system also includes a policy, a reward signal, and a value function. The policy determines the action the agent should take in a given state. When the agent performs an action in a given environmental state, the environment provides a reward signal, which is a single scalar value called the reward. Through the reward, the agent can determine whether its action was appropriate or inappropriate.

[0059] Solving the horizontal partitioning problem in distributed storage systems using reinforcement learning algorithms requires viewing the partitioning process as an interaction between an agent and its environment. Specifically, the three core elements of reinforcement learning that map the horizontal partitioning problem to this problem are as follows:

[0060] ① State: State describes the current system status during partitioning and is usually represented in the form of a data structure. In the horizontal partitioning problem of distributed databases, state can include the following information: the table structure in the database, the amount of data in the tables, the access frequency, and the load of each server, etc.

[0061] ② Action: An action is an operation that an agent can perform. In the horizontal partitioning problem, actions might include migrating or replicating a table from one server to another, or splitting the table into smaller parts so that it can be more rationally distributed across different servers.

[0062] ③ Reward: The reward is the feedback signal that the agent receives during its interaction with the environment, which guides it to make better decisions. In the horizontal partitioning problem, the reward value can be calculated based on system performance indicators (such as query response time, network bandwidth utilization, and server load balancing) as well as the cost and time of partitioning operations.

[0063] Based on the above three elements, a reinforcement learning model is constructed using a Markov decision process to solve the horizontal data partitioning problem in a distributed storage system. The data partitioning is described in detail below:

[0064] (a) Define a state space St∈S to reflect the data partition status and load of the server node. S is a set of states, including but not limited to: the amount of data contained in each partition, data distribution, read / write request rate, memory usage, and system resources; t=1,2,3,...

[0065] (b) Define the action space At∈A, where A is a set of actions, including partitioning operations (adding partitions, merging partitions, adjusting partition ranges), copying operations (data copying, deletion copying), and data migration, etc.

[0066] (c) Define a reward function Rt∈R, where R is the set of rewards. The effect of an agent taking a certain action is evaluated based on the following indicators: query return time, system load balancing, resource utilization, cross-partition communication overhead, etc.

[0067] (d) The agent selects an action based on the partition state, receives a reward from the environment, and then proceeds to the next state. Simultaneously, the agent updates its partitioning strategy based on environmental reward information and its current state. This agent-environment interaction process is repeated using a Markov decision process as a reinforcement learning algorithm until the agent learns the optimal partitioning strategy.

[0068] According to the Markov decision process, the current data partition adopts the strategy of action a in state s. This can be expressed by the formula:

[0069] ;

[0070] Where P represents the state transition probability, and the formula for the state transition probability is:

[0071] ;

[0072] In the formula, For data partitioning, execute the strategy At that time, the state changed from Go to The probability, Current state Take action Transition to state The probability of.

[0073] Take action in a state The formula for the reward obtained in the next moment is:

[0074] ;

[0075] in, express Take in state The expectation that an action will be rewarded in the next moment. In the state The following strategy The rewards received , In the state Take action below Then at the next moment Receive rewards Expected value .

[0076] On a Markov reward chain, from The sum of all rewards with diminishing returns from the moment onward. for:

[0077] ;

[0078] in It is a discount factor. , Let T be the last action taken after time t. For example, if 10 actions were taken after time t during data partitioning, resulting in 10 next-moment states, then T equals 10. The calculation is the sum of the diminishing returns obtained from these 10 next-moment states.

[0079] The policy-based behavior value function is:

[0080] ;

[0081] In the formula, Given the current state, this represents the expected reward for performing an action. To execute the strategy In progress The total amount of rewards that may be obtained in the future.

[0082] Where V is the policy-based state value function:

[0083] ;

[0084] ;

[0085] In the formula, Let E be the value function of the state at the next time step, and E be the expected value of the value function of the state at the next time step.

[0086] Finally, the optimal data partitioning strategy is found through the optimal action value function.

[0087] ;

[0088] in This represents the maximum value among the numerous action-state value functions generated under all data partitioning strategies. To maximize the optimal behavior value function Used to find Maximize the action , In addition to maximizing the optimal behavior value function Other than these, In the state Take action below The optimal policy is a function that assigns a probability value to each state-action pair, indicating the likelihood of taking that action in that state. The action value function is derived from the state value function, which can be obtained by weighted summation of the state value functions based on the policy's state probability distribution. The optimal action value function is the upper bound of the action value function, giving the best possible outcome of performing a certain action in a given state. The action value function based on a specific policy, however, represents the actual outcome under that policy and will not exceed the optimal outcome.

[0089] In this embodiment of the application, the method further includes: when data updates occur in each of the nodes, generating a change record, saving the change record and metadata, synchronizing the data of each of the nodes through a preset control mechanism, and handling concurrent write conflicts of each of the nodes.

[0090] Specifically, this application's solution can use Git (an open-source distributed version control system) to efficiently manage data, resolve issues such as data conflicts, concurrent operations, and document redundancy, achieve version traceability, and specifically address the chaotic data management situation.

[0091] A snapshot is a description of the state of data at a specific point in time. Git creates a snapshot of all files when a commit is made and manages it through an index. If a file has not changed, Git uses a link to the previous file, thus improving efficiency. The advantage of snapshots is that version switching is fast, and you can directly restore from the target snapshot.

[0092] As a distributed version control system, Git allows clients to have a complete repository containing all historical versions, branches, and tags, with most operations performed locally. Remote repositories, on the other hand, function like a central server in a centralized system. Furthermore, Git ensures data integrity through SHA-1 hash checksums, using these checksums to reference data, and automatically detects any changes to files managed by Git.

[0093] Based on the three functional characteristics of the Git system mentioned above, data version control is implemented, data changes are tracked, and data traceability and recovery are achieved.

[0094] Based on the above-described data storage method for distributed architecture, one or more embodiments of this specification also provide a platform or terminal for data storage in a distributed architecture. This platform or terminal may include devices, software, modules, plug-ins, servers, clients, etc., using the methods described in the embodiments of this specification, combined with necessary hardware implementation devices. Based on the same innovative concept, the systems in one or more embodiments provided in this specification are as described in the following embodiments. Since the implementation schemes and methods for solving the system problem are similar, the specific system implementation in the embodiments of this specification can refer to the implementation of the aforementioned methods. Repeated descriptions will not be repeated. The terms "unit" or "module" used below can refer to a combination of software and / or hardware that performs a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementations, and a combination of software and hardware, are also possible and contemplated.

[0095] Specifically, Figure 2 This is a schematic diagram of the module structure of an embodiment of a data storage system for a distributed architecture provided in this specification, as shown below. Figure 2 As shown, the data storage system for distributed architectures provided in this specification includes:

[0096] The hash module is used to obtain the load of each node in the distributed architecture and map the address of each node in the hash ring according to a preset ratio. Each node has multiple corresponding virtual nodes in the hash ring.

[0097] The storage module is used to map all data in the hash ring based on the unique number of each data in the dataset to be stored and store it in the virtual node. When the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to the virtual node with a load below a second preset threshold.

[0098] It should be noted that the system described above may include other implementation methods based on the description of the corresponding method embodiments. The specific implementation methods can be referred to the description of the corresponding method embodiments above, and will not be elaborated here.

[0099] This application also provides an electronic device, including:

[0100] processor;

[0101] Memory used to store the processor's executable instructions;

[0102] The processor is configured to perform the methods provided in the embodiments described above.

[0103] The electronic device provided in this application stores the executable instructions of the processor in a memory. When the processor executes the executable instructions, it can first obtain the load of each node in the distributed architecture and map the addresses of each node in a hash ring according to a preset ratio. Each node has multiple corresponding virtual nodes in the hash ring. Then, based on the unique number of each piece of data in the dataset to be stored, all data are mapped in the hash ring and stored in the virtual nodes. When the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to a virtual node with a load below a second preset threshold. This can effectively avoid the problem of unbalanced load between nodes during data storage in the distributed architecture and ensure the normal execution of data storage.

[0104] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0105] The methods or apparatus described in the embodiments provided in this specification can implement business logic through a computer program and record it on a storage medium. The storage medium can be read and executed by a computer to achieve the effects of the solutions described in the embodiments of this specification, such as:

[0106] The load of each node in the distributed architecture is obtained, and the addresses of each node are mapped in a hash ring according to a preset ratio. Each node has multiple corresponding virtual nodes in the hash ring.

[0107] Based on the unique number of each data in the dataset to be stored, all data are mapped in the hash ring and stored in the virtual nodes. When the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to a virtual node with a load below a second preset threshold.

[0108] The storage medium can include physical devices for storing information, typically digitizing the information and then storing it using electrical, magnetic, or optical methods. The storage medium can include: devices that store information using electrical energy, such as various types of memory, like RAM and ROM; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, bubble memory, and USB flash drives; and devices that store information using optical methods, such as CDs or DVDs. Of course, there are other readable storage media, such as quantum memories and graphene memories.

[0109] The embodiments in this specification are not limited to conforming to industry communication standards, standard computer resource data update and data storage rules, or the situations described in one or more embodiments of this specification. Slightly modified implementations based on certain industry standards or custom methods or embodiments can also achieve the same, equivalent, or similar, or predictable, implementation effects as described above. Embodiments that utilize these modified or modified methods for data acquisition, storage, judgment, and processing still fall within the scope of optional implementations of the embodiments in this specification.

[0110] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0111] The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or plug-ins may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0112] These computer program instructions can also be loaded onto a computer or other programmable resource data updating device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0114] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A data storage method for a distributed architecture, characterized in that, The method includes: The load of each node in the distributed architecture is obtained, and the addresses of each node are mapped in a hash ring according to a preset ratio. Each node has multiple corresponding virtual nodes in the hash ring. Based on the unique number of each piece of data in the dataset to be stored, all data are mapped in the hash ring and stored in the virtual nodes. When the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to a virtual node with a load below a second preset threshold. The method further includes partitioning the node into data, specifically including: Define a state space, an action space, and a reward function. The state space reflects the state of data partitions and the state of load in a node. The action space is a set of actions, which specifically include partitioning operations, data copying operations, and data migration operations. The reward function is the feedback reward after the agent takes any action. Based on the state space, determine the current state s, and determine the reward at the next moment after taking action a in the current state s; The total revenue of all rewards is determined based on a Markov reward chain; The behavioral value function is determined based on the sum of the revenues, and the optimal data partitioning strategy is determined based on the behavioral value function.

2. The data storage method for a distributed architecture as described in claim 1, characterized in that, The process of mapping all data in the dataset to be stored onto the hash ring and storing it within the virtual node, based on the unique identifier of each data entry, specifically includes: The hash value of each piece of data is determined based on its unique identifier. Based on the hash value, the corresponding data is mapped to the hash ring, and when the hash value corresponds to the hash position of a certain virtual node, the data corresponding to the hash value is stored in the virtual node corresponding to the hash value. Specifically, storing in the virtual node means storing in the node corresponding to the virtual node.

3. The data storage method for a distributed architecture as described in claim 1, characterized in that, The hash ring is divided into multiple independent intervals, and each interval contains a virtual node corresponding to a node.

4. The data storage method for a distributed architecture as described in claim 1, characterized in that, The method further includes: when data updates occur in each of the nodes, generating a change record and saving the change record and metadata.

5. The data storage method for a distributed architecture as described in claim 1, characterized in that, The method also includes synchronizing the data of each node through a preset control mechanism and handling concurrent write conflicts of each node.

6. A data storage system for a distributed architecture, characterized in that, The system includes: The hash module is used to obtain the load of each node in the distributed architecture and map the address of each node in the hash ring according to a preset ratio. Each node has multiple corresponding virtual nodes in the hash ring. The storage module is used to map all data in the hash ring based on the unique number of each data in the dataset to be stored and store it in the virtual node. When the load of any virtual node exceeds a first preset threshold, the storage request is forwarded to the virtual node with a load below a second preset threshold. This also includes data partitioning of the nodes, specifically including: Define a state space, an action space, and a reward function. The state space reflects the state of data partitions and the state of load in a node. The action space is a set of actions, which specifically include partitioning operations, data copying operations, and data migration operations. The reward function is the feedback reward after the agent takes any action. Based on the state space, determine the current state s, and determine the reward at the next moment after taking action a in the current state s; The total revenue of all rewards is determined based on a Markov reward chain; The behavioral value function is determined based on the sum of the revenues, and the optimal data partitioning strategy is determined based on the behavioral value function.

Images