A method and device for distributed data storage

Abstract

The embodiment of the present application discloses a data distributed storage method and device, the method comprising: sharding the data to be stored to obtain the initial sharding result; using the reinforcement learning model to predict the heat of the data shard according to the historical access data to obtain the access heat of the data shard; determining the number of copies based on the access heat and importance of the data shard, generating shard copies of the data shard according to the number of copies, and storing the data shards and shard copies in the storage node according to the access heat and node load status score; when the storage node load is unbalanced, data migration is performed on the data shards stored in the storage node, and when the node health status of a storage node is abnormal or the stored data shards are lost, data reconstruction is performed on the data shards stored in the storage node. The present application can alleviate the problems of uneven access load, waste of resources and data loss in the distributed storage system.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a method and device for distributed data storage. Background Art

[0002] With the development of big data technology, the rapid growth of data storage has made it difficult for traditional centralized storage models to meet actual needs. Distributed storage provides higher storage capacity, reliability and performance by distributing data to multiple storage nodes.

[0003] However, in practical applications, distributed storage systems face problems such as uneven data distribution, overloaded access to hotspot shards, unbalanced node loads, and data loss due to storage node failures. Summary of the invention

[0004] The present application provides a data distributed storage method, which effectively addresses the problems of uneven access load, resource waste, and data loss in a distributed storage system.

[0005] This application provides the following solutions:

[0006] According to a first aspect, a method for distributed data storage is provided, the method comprising: using a data sharding algorithm to perform sharding processing on data to be stored to obtain an initial sharding result, wherein the initial sharding result includes more than one data shard; using a reinforcement learning model to predict the heat of the data shard according to historical access data to obtain the access heat of the data shard; determining the number of copies based on the access heat and importance of the data shard, and generating shard copies of the data shard according to the number of copies, wherein determining the number of copies based on the access heat and importance of the data shard comprises: increasing the number of copies for data shards with high access heat and / or high importance, and reducing the number of copies for data shards with low access heat and / or low importance on the premise of meeting the minimum number of copies; real-time monitoring of the multi-dimensional load indicators of each storage node to obtain the node load status score of each storage node, the multi-dimensional The load indicators include: CPU usage, storage capacity utilization, bandwidth occupancy, network delay and node health status; according to the access popularity and the node load status score, the data shards and the shard copies are stored in the storage node; when the storage node load is unbalanced, the data shards stored in the storage node are migrated; the data migration includes: sorting the data shards according to the access popularity, and migrating the shards with low access frequency first; using the shard compression technology to compress the data shards with low access frequency and then transmit them, and determining the target node and migration rate according to the load indicators; when the node health status of a certain storage node is abnormal or the stored data shards are lost, the data shards stored in the storage node are reconstructed to obtain the reconstructed shard data, and according to the health status of each node, a storage node is selected to store the reconstructed shard data.

[0007] According to an achievable method in an embodiment of the present application, the use of a reinforcement learning model to predict the heat of the data shards based on historical access data includes: constructing a reinforcement learning state space with access frequency, access interval and access request source as input, using a policy gradient algorithm to continuously optimize the access heat prediction strategy, and generating future access heat scores for each data shard.

[0008] According to an achievable method in an embodiment of the present application, increasing the number of copies of a shard with high access popularity includes: grading the shards according to the access popularity of the data shards to obtain data shards with high access popularity, and increasing the number of copies of the shards with high access popularity using a preset formula.

[0009] According to an achievable method in an embodiment of the present application, the use of shard compression technology to compress and transmit data shards with low access frequency includes: compressing the structured data in the data shards using a column storage-based compression algorithm, and compressing the unstructured data in the data shards based on a dictionary.

[0010] According to an achievable method in an embodiment of the present application, the data reconstruction of the data shards stored in the storage node includes: dividing the lost shards into key shards and ordinary shards according to the access popularity and the importance of the shards, using an improved check code mechanism to reconstruct the data, and based on the redundant characteristics of the shard data and the check code, preferentially restoring the key shard data in combination with the association information between the shards.

[0011] According to an achievable method in an embodiment of the present application, the data migration also includes: scanning the storage status of the data blocks in the sharded data, marking the discontinuous data blocks in the data shards distributed at different addresses or nodes; and preferentially migrating the discontinuous data blocks in the shards based on the marking results.

[0012] According to an achievable method in an embodiment of the present application, selecting a storage node for storing the reconstructed shard data based on the health status of each node includes: selecting a storage node whose remaining storage capacity is greater than a preset threshold, sorting the storage nodes according to the health status of the nodes, and preferentially selecting a storage node with a higher ranking to store the reconstructed shard data.

[0013] According to a second aspect, a data distributed storage device is provided, the device comprising: an initial sharding unit, configured to perform sharding processing on the data to be stored using a data sharding algorithm to obtain an initial sharding result, wherein the initial sharding result includes more than one data shard; a replica generation unit, configured to perform heat prediction on the data shard according to historical access data using a reinforcement learning model to obtain the access heat of the data shard; determine the number of replicas based on the access heat and importance of the data shard, and generate shard replicas of the data shard according to the number of replicas, wherein determining the number of replicas based on the access heat and importance of the data shard comprises: increasing the number of replicas for data shards with high access heat and / or high importance, and reducing the number of replicas for data shards with low access heat and / or low importance on the premise of meeting the minimum number of replicas; a load evaluation unit, configured to monitor the multidimensional load indicators of each storage node in real time, obtain the node load status score of each storage node, the multidimensional load indicators of each storage node The load indicators include: CPU usage, storage capacity utilization, bandwidth occupancy, network delay and node health status; a data storage unit is configured to store the data shards and the shard copies in the storage node according to the access popularity and the node load status score; a data migration unit is configured to migrate the data shards stored in the storage node when the storage node load is unbalanced; the data migration includes: sorting the data shards according to access popularity and migrating the shards with low access frequency first; using shard compression technology to compress the data shards with low access frequency and then transmit them, and determining the target node and migration rate according to the load indicator; a data reconstruction unit is configured to reconstruct the data shards stored in the storage node when the node health status of a certain storage node is abnormal or the stored data shards are lost, obtain the reconstructed shard data, and select the storage node for storing the reconstructed shard data according to the health status of each node.

[0014] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0015] According to a fourth aspect, an electronic device is provided, comprising: one or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, execute the steps of any one of the methods described in the first aspect.

[0016] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0017] (1) This application realizes efficient data sharding and access popularity prediction by combining data sharding algorithm and reinforcement learning model, so that the number of replicas can be dynamically adjusted according to access popularity and importance, thereby effectively improving data access efficiency and storage resource utilization. At the same time, real-time monitoring of storage node load and reasonable allocation of data shards according to load status help balance node pressure and enhance system stability and reliability. In addition, through dynamic migration and exception handling mechanism, the system can adaptively adjust in the case of load imbalance or node abnormality to ensure data security and access performance.

[0018] (2) By constructing a reinforcement learning state space, taking access frequency, interval, and request source as input, and optimizing the prediction strategy using the policy gradient algorithm, we can accurately predict the future access popularity of data shards. This prediction mechanism improves the accuracy of popularity judgment, allowing the system to take adjustment measures in advance, thereby further optimizing storage layout and resource allocation efficiency.

[0019] (3) Using a hierarchical sorting mechanism to prioritize high-access hotspot shards and dynamically adjust their number of replicas through a preset formula can avoid resource waste while maintaining high data availability. This strategy adaptively increases the number of replicas of popular data, optimizes the response speed of data access, and improves the overall system performance and user experience.

[0020] (4) Adaptive compression algorithms are used for structured data and unstructured data, effectively reducing bandwidth usage and storage consumption during data migration. Through shard compression technology, the system can more efficiently transmit low-frequency data shards, reduce network burden, and improve the speed and reliability of data migration.

[0021] (5) By utilizing the improved checksum mechanism and combining the access popularity and importance of the shards, key shard data is restored first, which enhances the efficiency and accuracy of data reconstruction. With the assistance of associated information, the system can quickly restore key data, thereby ensuring the availability of high-priority data, while shortening the time for data reconstruction and improving disaster recovery capabilities.

[0022] (6) By scanning the discontinuous storage status of sharded data and migrating discontinuous data blocks first, the fragmentation problem in sharded data storage is effectively reduced, while optimizing the transmission efficiency during data migration. This strategy reduces the complexity of the migration process and improves the data consistency and availability of the system.

[0023] (7) Based on the health status of the nodes and the remaining storage capacity, the storage nodes for the reconstructed data are prioritized and selected to ensure that the reconstructed data is stored on nodes with better health status, thereby reducing the risk of system failure. This dynamic allocation strategy further optimizes data distribution and improves the system's fault tolerance and overall operating efficiency.

[0024] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of a process flow of a data distributed storage method provided in an embodiment of the present application;

[0027] Figure 2 A schematic block diagram of a data distributed storage device provided in an embodiment of the present application;

[0028] Figure 3 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0030] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0031] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0032] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0033] There are already some technologies that shard data through simple sharding strategies (such as consistent hashing algorithms), but they have limited optimization capabilities in dealing with dynamic changes in access popularity and imbalanced loads on storage nodes. Therefore, how to design a more efficient distributed storage method that can dynamically adjust the distribution of data shards and the number of copies while improving the overall performance and reliability of the system is an urgent problem to be solved in this field.

[0034] In view of this, the present application provides a new idea. In order to facilitate the understanding of the present application, the system architecture on which the present application is based is first described. Figure 1 The data distributed storage method flow chart provided in the embodiment of the present application is as follows: Figure 1 As shown in , the method may include the following steps:

[0035] Step 101: shard the data to be stored using a data sharding algorithm to obtain an initial sharding result, which includes more than one data shard.

[0036] Step 102: Utilize a reinforcement learning model to predict the popularity of the data shard based on historical access data to obtain the access popularity of the data shard; determine the number of replicas based on the access popularity and importance of the data shard, and generate shard replicas of the data shard based on the number of replicas, wherein determining the number of replicas based on the access popularity and importance of the data shard includes: increasing the number of replicas for data shards with high access popularity and / or high importance, and reducing the number of replicas for data shards with low access popularity and / or low importance while meeting the minimum number of replicas.

[0037] Step 103: Monitor the multi-dimensional load indicators of each storage node in real time and obtain the node load status score of each storage node. The multi-dimensional load indicators include: CPU usage, storage capacity utilization, bandwidth occupancy, network delay and node health status.

[0038] Step 104: According to the access popularity and node load status score, the data shards and shard copies are stored in the storage node.

[0039] Step 105: When the storage node load is unbalanced, data shards stored in the storage node are migrated; data migration includes: sorting the data shards according to access popularity, and migrating shards with low access frequency first; using shard compression technology to compress and transmit data shards with low access frequency, and determining the target node and migration rate according to the load index.

[0040] Step 106: When the node health status of a storage node is abnormal or the stored data slices are lost, data is reconstructed for the data slices stored in the storage node to obtain reconstructed slice data, and a storage node is selected to store the reconstructed slice data according to the health status of each node.

[0041] It can be seen from the above process that this application realizes efficient data sharding and access popularity prediction through the combination of data sharding algorithm and reinforcement learning model, so that the number of copies can be dynamically adjusted according to access popularity and importance, thereby effectively improving data access efficiency and storage resource utilization. At the same time, real-time monitoring of storage node load and reasonable allocation of data shards according to load status help balance node pressure and enhance system stability and reliability. In addition, through dynamic migration and exception handling mechanisms, the system can adaptively adjust in the event of load imbalance or node abnormality to ensure data security and access performance.

[0042] The following describes in detail each step in the above process and the effects that can be further produced in combination with the embodiments. First, the above step 101, i.e. "using a data sharding algorithm to shard the data to be stored to obtain an initial sharding result, the initial sharding result including more than one data shard", is described in detail in combination with the embodiments.

[0043] When storing data, the present application first divides the large-scale data set to be stored into multiple smaller data blocks, namely data shards, so as to efficiently store and access them in a distributed storage system. Data sharding is a common technology in distributed systems. By dividing data into multiple parts, the utilization of storage capacity can be improved, the performance of data access can be optimized, and the pressure on a single storage node can be reduced. Each data shard can be stored on a different storage node, so that parallel access and load balancing of data can be achieved.

[0044] The present application utilizes a data sharding algorithm to divide the data to be stored into multiple data slices, wherein different sharding algorithms can be used to implement data sharding. For example, algorithms such as hash sharding or consistent hash sharding. The hash sharding algorithm uses a hash function to shard the data and maps the data to different shards according to the hash value. The hash value of each data item determines which shard it belongs to. For example, the data can be hashed according to the ID field of the data, and the hash value can be obtained and then allocated to the corresponding shard by taking the remainder or the like. This method ensures that the distribution of the shards is more uniform and is suitable for scenarios with uneven access patterns; consistent hashing is a more flexible sharding algorithm that can effectively cope with the dynamic addition or deletion of storage nodes without the need for a large amount of data migration. Through consistent hashing, the shards can be mapped to a virtual ring, and then the data items can be allocated to the corresponding positions according to the hash value. When a node changes, only a very small amount of shard data needs to be migrated, avoiding the migration of the full amount of data. This algorithm is often used in distributed storage systems with dynamic scale expansion. And technical personnel in the field can flexibly choose according to actual needs.

[0045] The following is a detailed description of step 102 in conjunction with an embodiment, namely, "using a reinforcement learning model to predict the popularity of the data shard based on historical access data to obtain the access popularity of the data shard; determining the number of replicas based on the access popularity and importance of the data shard, and generating shard replicas of the data shard based on the number of replicas, wherein determining the number of replicas based on the access popularity and importance of the data shard includes: increasing the number of replicas for data shards with high access popularity and / or high importance, and reducing the number of replicas for data shards with low access popularity and / or low importance while meeting the minimum number of replicas."

[0046] Reinforcement Learning (RL) is a machine learning method that learns the optimal decision-making strategy through interaction with the environment. In this application, the reinforcement learning model is used to analyze historical access data and predict the access popularity of data shards. The core idea of ​​reinforcement learning is that the model gradually optimizes its behavior strategy based on environmental feedback. In a data distributed storage system, the reinforcement learning model can continuously adjust and optimize the access popularity prediction of shards by learning from historical access data.

[0047] Specifically, the reinforcement learning model evaluates the access popularity of each data shard based on historical access patterns, and infers future access trends based on this popularity information. The historical access popularity data of each data shard is an important input to the reinforcement learning model. The model generates a predicted value based on this information, which reflects the possible access frequency of the data shard in the future.

[0048] As an implementable method, the present application may include, when using a reinforcement learning model to predict the access heat of data shards: constructing a reinforcement learning state space with access frequency, access interval, access request source, etc. as input, and using a policy gradient algorithm to continuously optimize the access heat prediction strategy to generate the future access heat score of each data shard.

[0049] Among them, the access frequency refers to the number of times a data shard is accessed within a certain time range. Data shards with higher frequencies usually indicate higher popularity; the access interval refers to the time interval between two accesses. Shards with shorter access intervals may indicate that they have higher popularity or are frequently requested; the access request source includes the source information of the user or system accessing the data. Different access request sources may mean that the access popularity of a data shard is affected by different types of requests. For example, some request sources may represent requests from high-priority business systems, which means that these shards have higher access priorities. These input features constitute the state space of the reinforcement learning model, and the model learns and predicts the future access popularity of each data shard by processing these features.

[0050] In order to effectively learn from the constructed state space and make optimal decisions, the reinforcement learning model adopts the policy gradient algorithm. The policy gradient algorithm is a method based on optimization strategy. By directly optimizing the strategy, the model can take the best action under a given state. In the technical solution of the present application, the goal of the policy gradient algorithm is to continuously optimize the access heat prediction strategy so that the access heat prediction value of each data shard is as accurate as possible. The advantage of the policy gradient algorithm is that it maximizes the reward by gradually adjusting the strategy, thereby achieving accurate prediction of future access heat. In each step, the model will take an action (i.e., predict the access heat of the shard) based on the current state (for example, the access frequency and access interval of the data shard), and feedback the results based on the actual access situation to further adjust the prediction strategy.

[0051] After predicting the access popularity of data shards, the number of replicas is dynamically adjusted according to the access popularity and importance of the shards. Shards with higher access popularity mean that the data will be accessed frequently, so more replicas are needed to improve data availability and access speed; on the contrary, shards with lower access popularity may not need too many replicas, which can reduce the waste of storage space. When determining the number of replicas, in addition to access popularity, the importance of the data shard also needs to be considered. For example, some important data, even if they are less frequently accessed, still need to maintain a large number of replicas to ensure data security and reliability.

[0052] Specifically, the following steps may be included: increasing the number of copies of data shards with high access popularity or / and high importance to improve data redundancy and accessibility. Under the premise of meeting the minimum number of copies, reducing the number of copies of data shards with low access popularity or / and low importance, thereby saving storage space and reducing the load on storage nodes.

[0053] As an implementable method, increasing the number of copies of shards with high access popularity includes the following steps: Increasing the number of copies of shards with high access popularity includes: grading the shards according to the access popularity of the data shards to obtain data shards with high access popularity, and increasing the number of copies of the shards with high access popularity by a preset formula. Among them, hierarchical sorting refers to dividing the shards into different levels (such as high, medium, and low access popularity levels) according to certain rules according to the access popularity of the data shards, and sorting the shards within each level. Its main function is to help the system identify which shards require a higher number of copies or priority storage resources to achieve optimal allocation of resources. Specifically, data shards with an access frequency greater than 90 times / hour can be defined as high access popularity, data shards with an access frequency between 30 times / hour and 90 times / hour can be defined as medium access popularity, and data shards with an access frequency less than 30 times / hour can be defined as low access popularity.

[0054] For high-access popularity shards, the number of replicas is adjusted dynamically. By using the preset formula, the number of replicas can be flexibly increased according to the actual access popularity and storage requirements. The preset formula can be set flexibly. For example, the preset formula can be: Number of replicas = basic number of replicas + f (access popularity, node load, storage capacity) Among them, f() is a dynamic function that can adjust the number of replicas based on factors such as access popularity, node load and storage capacity.

[0055] The following describes in detail the above step 103, i.e., "monitoring the multidimensional load indicators of each storage node in real time to obtain the node load status score of each storage node, wherein the multidimensional load indicators include: CPU usage, storage capacity utilization, bandwidth occupancy, network delay and node health status" in conjunction with an embodiment.

[0056] This application scores the node load status based on the multi-dimensional load indicators of each storage node. The multi-dimensional load indicator is a key factor in evaluating the storage node load status and covers multiple system-level performance indicators, including the following aspects:

[0057] CPU usage: CPU usage is an important indicator to measure whether the storage node has sufficient computing resources. High CPU usage usually means that the storage node is performing heavy computing tasks, which may lead to slower response speed or increased latency. Therefore, the system needs to monitor the CPU usage in real time. When the CPU usage is too high, the system can take measures, such as migrating tasks to nodes with lower loads, to avoid excessive load.

[0058] Storage capacity utilization: Storage capacity utilization reflects the ratio of used and remaining storage space on a storage node. If the storage capacity utilization of a storage node is close to 100%, it means that the node is close to full storage capacity, which may affect data writing and reading performance. Real-time monitoring of this indicator can help the system to warn in advance and prevent data loss or performance degradation due to node exhaustion of storage resources.

[0059] Bandwidth utilization: Bandwidth utilization indicates the ratio of the data read and write speed of a storage node to the bandwidth. If the I / O bandwidth utilization is too high, it may mean that a bottleneck has occurred in data transmission, resulting in a decrease in the read and write performance of the storage system. By monitoring the I / O bandwidth utilization, the system can effectively evaluate whether the node is overly dependent on the network bandwidth, and then decide whether it is necessary to adjust the data distribution or migrate shards to reduce I / O pressure.

[0060] Network latency: Network latency is a measure of the time it takes for data to be transmitted over a network, usually measured in milliseconds. Higher network latency can significantly reduce data access speed and processing efficiency, especially in scenarios where stored data needs to be accessed frequently. Real-time monitoring of network latency can help the system identify network bottlenecks and take timely optimization measures, such as reallocating data or adjusting data access paths, to reduce the impact of latency on system performance.

[0061] Node health status: Node health status reflects the health of storage node hardware and software, such as hard disk health, temperature, memory usage, operating system status, etc. If a storage node fails or has an abnormal condition, data may become inaccessible or lost. Real-time monitoring of node health status helps to detect potential problems as early as possible, take appropriate repair or migration measures, and avoid large-scale system failures.

[0062] Through real-time monitoring of the above multi-dimensional load indicators, a comprehensive node load status score can be calculated for each storage node. This score will reflect the current load status and health level of the storage node, and is usually based on the weighted synthesis of multiple load indicators. Specifically, parameters such as CPU usage, storage capacity utilization, bandwidth occupancy, and network latency can be normalized and assigned corresponding weights; for node health status, such as whether the node has a hardware failure, the health status can be quantified as 1 for completely healthy and 0 for failure; these indicators are weighted and summed to obtain the node load status score.

[0063] The score provides a quantitative evaluation standard for the system, which can help the system make more intelligent decisions. For example, a node with a low load status score may be selected to store highly accessed shards, or as the preferred node for migration when the load is unbalanced.

[0064] The above step 104, namely "score the data shards and shard copies in the storage nodes according to the access popularity and the node load status", is described in detail below in conjunction with the embodiments.

[0065] In this application, data shards and shard copies are stored according to the access popularity and node load status scores. Nodes with lower scores indicate lighter loads and better performance for taking on new tasks. Based on the load score, this application can prioritize allocating high-access popularity shards to nodes with lower scores to balance the load pressure between nodes and avoid performance bottlenecks caused by high-heat data being concentrated in certain nodes.

[0066] For shards with high access popularity, their copies can be preferentially stored on nodes with lower loads in different regions to improve access reliability and concurrency; for shards with low access popularity, the copies can be concentrated on a few nodes with lower loads to save storage resources.

[0067] As an implementable method, when storing data shards and shard copies in storage nodes, a multi-dimensional allocation matrix can be established for each data shard and shard copy according to its access popularity and node load status score, and high access popularity shards are preferentially stored in storage nodes with sufficient bandwidth and lower load scores.

[0068] The following is a detailed description of step 105, namely, "When the storage node load is unbalanced, data migration is performed on the data shards stored in the storage node; the data migration includes: sorting the data shards according to access popularity, and migrating the shards with low access frequency first; using shard compression technology to compress the data shards with low access frequency and then transmit them, and determining the target node and migration rate according to the load indicator" in conjunction with the embodiment.

[0069] When the storage node load is unbalanced, the application starts a data migration operation to transfer part of the data from the node with higher load to the node with lower load, so as to balance the resource usage of the overall storage system. The load status of the storage node is evaluated by real-time monitoring of multi-dimensional load indicators. It can be evaluated by the node load status score or by one or more indicators in the load indicators. For example, the data migration operation can be started when the load status score of some nodes is significantly increased or lower than the set threshold.

[0070] To optimize migration efficiency and reduce the impact on system performance, the migration operation first sorts the data shards by access popularity. Low-access frequency shards have low real-time performance requirements for the system, and migrating them will not significantly affect the user's access experience. Prioritizing the migration of these shards can reduce the interference of the migration operation on system performance and ensure efficient access to high-access popularity shards on the original node.

[0071] For shards with low access frequency, compress the data before migration. You can choose a suitable compression algorithm (such as lossless compression or mild lossy compression) based on the shard content to ensure that the data can be accurately restored after migration while minimizing the computing overhead.

[0072] As an implementable method, an adaptive compression algorithm can be selected according to the structural characteristics of the shard content, and the structured data can be compressed by a compression algorithm based on column storage, and the unstructured data can be compressed based on a dictionary. Specifically, each data shard may contain structured data (such as database records, table data) and unstructured data (such as text files, pictures, and videos). The key to shard compression technology is to match the adaptive compression algorithm according to the data characteristics to obtain the best compression effect and decompression efficiency. For structured data, a compression algorithm based on column storage, such as Run-Length Encoding (RLE) or Delta Encoding, is used to effectively reduce data redundancy. For unstructured data, a dictionary-based compression algorithm is used to replace repeated strings or byte sequences with shorter codes by taking advantage of the repetitiveness in the data. A dynamic dictionary is constructed to gradually fill in common strings or patterns according to the content in the shard. The repeated parts in the shard content are replaced with index values ​​in the dictionary, thereby significantly reducing the amount of data.

[0073] The target node is the node to which the shard is migrated. The determination of the target node is based on multi-dimensional considerations: for example, based on network latency, the target node with lower network latency than the current node is selected first to speed up the migration and reduce latency. Based on storage capacity utilization, the node with lower storage capacity utilization is selected first as the target node to balance the storage resource distribution of the entire distributed storage.

[0074] In order to prevent the migration operation from causing excessive pressure on the current system operation, this application dynamically adjusts the migration rate: when the network bandwidth is sufficient and the target node load is low, the migration is accelerated to restore the load balance as soon as possible; when the network is congested or the target node load score increases, the migration rate is reduced to ensure that normal services are not affected.

[0075] The data blocks in the shard may be distributed on different physical addresses or nodes during storage due to insufficient storage node space, interruption of the write process, or historical migration operations, resulting in non-contiguous storage. This discontinuity will reduce data access efficiency and increase system read latency.

[0076] As an implementable solution, the present application gives priority to migrating discontinuous data blocks in data shards during data migration. Specifically, the storage status of the data blocks in the shards is scanned, and discontinuous blocks in the shards that are distributed at different addresses or nodes are marked; based on the marking results, the discontinuous data blocks in the shards are preferentially migrated. At the same time, during the migration process, the order of migration can be further modified based on parameters such as the dispersed location of the data blocks, the importance of the block data, the network transmission conditions, and the load status of the target storage node. For example, data blocks with high relevance to critical businesses and severe discontinuity are placed at the top of the migration task list, and low-priority shard migration tasks are postponed, such as data blocks with low access popularity and low dispersion.

[0077] The following is a detailed description of step 106, i.e., "when the node health status of a storage node is abnormal or the stored data shards are lost, reconstructing the data shards stored in the storage node to obtain reconstructed shard data, and selecting a storage node for storing the reconstructed shard data based on the health status of each node" in conjunction with the embodiments.

[0078] This application evaluates the health status of storage nodes through real-time monitoring indicators. When an abnormal condition occurs in a node, such as hardware failure alarm, disk read and write error rate, network connection stability, node response time, etc., this application will mark the node as an unhealthy node and start the data reconstruction process to avoid data unavailability or permanent loss.

[0079] For node data reconstruction, a checksum mechanism can be used for reconstruction. The checksum mechanism is a data redundancy technology widely used in distributed storage systems. It aims to restore data when the original data is lost or damaged by adding checksum data.

[0080] As an implementable method, this application uses an improved check code mechanism for data reconstruction. The improved check code mechanism is optimized based on the traditional check code, mainly solving problems such as efficiency, storage cost and reliability. Traditional check code mechanisms include common parity check codes, erasure codes (such as Reed-Solomon codes), LDPC (Low-Density Parity-Check Code), etc.

[0081] The present application is based on the redundancy characteristics of shard data and check codes, combined with the association information between shards to give priority to restoring key shard data. By analyzing the access popularity and importance of shard data, the redundancy ratio of the check code is dynamically adjusted to improve the recovery priority and efficiency of key data shards. Introduce association information between data shards, such as partial redundancy of adjacent shards or metadata indexes of specific shards, so as to provide additional recovery clues when some shards are lost. Specifically, the state of the storage node is detected in real time through the node health monitoring module to determine whether any data shards are lost. If the loss of a shard is detected, the data reconstruction process is started immediately. The improved check code mechanism allows the system to use the remaining shard data and check codes to restore the original data when a small number of data shards are lost. For example: for Reed-Solomon codes, if there are k original shards and r check shards, any k shards can reconstruct the lost data. In the improved mechanism, the value of r is dynamically adjusted so that the key data shards have higher recovery fault tolerance when stored. According to the access popularity and importance of the shards, the lost shards are divided into key shards and ordinary shards. Key shards are restored first to ensure that high-priority data is available as soon as possible. At the same time, the metadata information of each shard (such as the checksum relationship between adjacent shards) can be recorded to improve the traceability and recovery efficiency of shard data. The partial redundant information embedded in adjacent shards is used to reduce the reliance on complete checksums. For example, data fragments in high-frequency access shards can be used to assist in the recovery of low-frequency access shards.

[0082] The reconstructed shard data needs to be stored again in healthy storage nodes to ensure system reliability and data persistence. When selecting nodes, you can give priority to nodes with high health scores to avoid the risk of data loss again; you can also choose nodes with low current load scores to ensure that new data will not cause node overload. At the same time, you must also consider optimizing data distribution to avoid too many shard copies being concentrated in a few nodes, ensure that data is evenly distributed, and improve the system's fault tolerance.

[0083] As an implementable method, the step of selecting a storage node for the reconstructed shard data based on the node health status includes: selecting a storage node whose remaining storage capacity is greater than a preset threshold, sorting the storage nodes according to the node health status, and preferentially selecting a storage node with a higher ranking to store the reconstructed shard data.

[0084] In the data distributed storage method of the present application, the system provides a flexible and efficient data call mechanism to meet the data retrieval and processing requirements in various scenarios. When a specific data needs to be retrieved, the system quickly locates the storage node where the corresponding data shard is located by querying the unique key value or other identification information of the data, and efficiently obtains the required data fragments from the target node. In addition, when it is necessary to summarize or aggregate the distributed stored data, the system supports the execution mode of distributed computing. Specifically, predefined calculations or processing tasks can be performed on each storage node on the data shards stored there to fully utilize the parallel processing capabilities of the distributed storage architecture. Subsequently, the calculation results of each node will be transmitted to a central node or summary location, where they will be uniformly integrated and analyzed to generate global results. Through this flexible call and aggregation mechanism, the method of the present application can significantly improve the efficiency of data retrieval and processing, while reducing the occupation of system resources and the burden of network transmission, and is suitable for high-performance distributed storage and computing scenarios.

[0085] The above method provided in the embodiment of the present application can be applied to a variety of application scenarios, including but not limited to: in a database management system, the database system can improve the performance and scalability of the database by using the method in the present application, and sharding can split a large database into manageable small blocks to reduce query time; in a distributed file system, the distributed file system can use the method in the present application to store large files or objects, thereby improving the performance and capacity of the file system; cloud service providers can use the method in the present application to distribute user data to provide high availability and scalability; social media platforms use the method in the present application to store and manage user-generated content to support large-scale user access; the Internet of Things system can use the method in the present application to manage and store data generated by sensors for real-time monitoring and analysis.

[0086] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] According to an embodiment of another aspect, a data distributed storage device is provided. Figure 2 A schematic block diagram of the data distributed storage device according to an embodiment is shown as follows: Figure 2 As shown, the device 200 includes:

[0088] The initial sharding unit 201 is configured to perform sharding processing on the data to be stored by using a data sharding algorithm to obtain an initial sharding result, wherein the initial sharding result includes more than one data shard;

[0089] The replica generation unit 202 is configured to use a reinforcement learning model to predict the popularity of data shards based on historical access data to obtain the access popularity of the data shards; determine the number of replicas based on the access popularity and importance of the data shards, and generate shard replicas of the data shards based on the number of replicas, wherein determining the number of replicas based on the access popularity and importance of the data shards includes: increasing the number of replicas for data shards with high access popularity and / or high importance, and reducing the number of replicas for data shards with low access popularity and / or low importance while meeting the minimum number of replicas.

[0090] The load evaluation unit 203 is configured to monitor the multi-dimensional load indicators of each storage node in real time and obtain the node load status score of each storage node. The multi-dimensional load indicators include: CPU usage, storage capacity utilization, bandwidth occupancy, network delay and node health status.

[0091] The data storage unit 204 is configured to store the data shards and the shard replicas in a storage node according to the access popularity and the node load status score.

[0092] The data migration unit 205 is configured to migrate data shards stored in the storage node when the storage node load is unbalanced; the data migration includes: sorting the data shards according to access popularity, and giving priority to migrating shards with low access frequency; using shard compression technology to compress the data shards with low access frequency and then transmit them, and determining the target node and migration rate according to the load indicator.

[0093] The data reconstruction unit 206 is configured to reconstruct the data shards stored in a storage node when the node health status of the storage node is abnormal or the stored data shards are lost, obtain reconstructed shard data, and select a storage node for storing the reconstructed shard data according to the health status of each node.

[0094] As one of the feasible ways, when the replica generation unit 202 uses the reinforcement learning model to predict the heat of the data shard according to the historical access data, it can be configured as follows: constructing a reinforcement learning state space with access frequency, access interval and access request source as input, and using the policy gradient algorithm to continuously optimize the access heat prediction strategy to generate the future access heat score of each data shard.

[0095] As one of the achievable methods, when increasing the number of replicas for a shard with high access popularity, the replica generation unit 202 can be configured as follows: grading the shards according to the access popularity of the data shards to obtain data shards with high access popularity, and increasing the number of replicas of the shards with high access popularity by a preset formula.

[0096] As one of the feasible ways, when the data migration unit 205 compresses the data shards with low access frequency and transmits them after compression by using the shard compression technology, it can be configured as follows: compressing the structured data in the data shards by using a column storage-based compression algorithm, and compressing the unstructured data in the data shards based on a dictionary.

[0097] As one of the feasible ways, the data reconstruction unit 206 can be configured as follows when reconstructing the data shards stored in the storage node: according to the access popularity and importance of the shards, the lost shards are divided into key shards and common shards, and the data is reconstructed using an improved checksum mechanism, and based on the redundant characteristics of the shard data and the checksum, the key shard data is restored preferentially in combination with the association information between the shards.

[0098] As one of the achievable methods, the data migration unit 205 can be configured during data migration to: scan the storage status of data blocks in the sharded data, mark discontinuous data blocks distributed at different addresses or nodes in the data shards; and prioritize migrating discontinuous data blocks in the shards based on the marking results.

[0099] As one of the feasible ways, when the data reconstruction unit 206 selects a storage node for storing the reconstructed shard data according to the health status of each node, it can be configured as follows: selecting a storage node whose remaining storage capacity is greater than a preset threshold, sorting the storage nodes according to the health status of the nodes, and giving priority to selecting a storage node with a higher sorting to store the reconstructed shard data.

[0100] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0102] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0103] And an electronic device, comprising:

[0104] One or more processors; and a memory associated with the one or more processors, the memory being used to store program instructions, and the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0105] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0106] in, Figure 3The architecture of the electronic device is shown as an example, which may include a processor 310, a video display adapter 311, a disk drive 312, an input / output interface 313, a network interface 314, and a memory 320. The processor 310, the video display adapter 311, the disk drive 312, the input / output interface 313, the network interface 314, and the memory 320 may be communicatively connected via a communication bus 330.

[0107] The processor 310 may be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in this application.

[0108] The memory 320 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 320 can store an operating system 321 for controlling the operation of the electronic device 300, and a basic input and output system (BIOS) 322 for controlling the low-level operation of the electronic device 300. In addition, a web browser 323, a data storage management system 324, and a data distributed storage device 325, etc. can also be stored. The above-mentioned data distributed storage device 325 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 320 and is called and executed by the processor 310.

[0109] The input / output interface 313 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0110] The network interface 314 is used to connect to a communication module (not shown) to achieve communication interaction between the device and other devices. The communication module can achieve communication through a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WIFI, Bluetooth, etc.).

[0111] The bus 330 comprises a pathway for transmitting information between the various components of the device (eg, the processor 310 , the video display adapter 311 , the disk drive 312 , the input / output interface 313 , the network interface 314 , and the memory 320 ).

[0112] It should be noted that, although the above device only shows a processor 310, a video display adapter 311, a disk drive 312, an input / output interface 313, a network interface 314, a memory 320, a bus 330, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0113] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0114] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A data distributed storage method, characterized in that: The method comprises: Using a data sharding algorithm to perform sharding processing on the data to be stored, and obtaining an initial sharding result, wherein the initial sharding result includes more than one data shard; Using a reinforcement learning model, the data shard is predicted to have high heat according to historical access data, and the access heat of the data shard is obtained; the number of copies is determined based on the access heat and importance of the data shard, and a shard copy of the data shard is generated according to the number of copies, wherein determining the number of copies based on the access heat and importance of the data shard includes: increasing the number of copies of data shards with high access heat and / or high importance, and reducing the number of copies of data shards with low access heat and / or low importance on the premise of meeting the minimum number of copies; Monitor the multi-dimensional load indicators of each storage node in real time and obtain the node load status score of each storage node, wherein the multi-dimensional load indicators include: CPU usage, storage capacity utilization, bandwidth occupancy, network delay and node health status; According to the access popularity and the node load status score, the data shard and the shard replica are stored in the storage node; When the storage node load is unbalanced, data migration is performed on the data slices stored in the storage node; the data migration includes: sorting the data slices according to access popularity, migrating the slices with low access frequency first, compressing the data slices with low access frequency and transmitting them by slice compression technology, and determining the target node and migration rate according to the load index; When the node health status of a storage node is abnormal or the stored data shards are lost, the data shards stored in the storage node are reconstructed to obtain reconstructed shard data, and a storage node is selected to store the reconstructed shard data based on the health status of each node.

2. The data distributed storage method according to claim 1, characterized in that: The method of using a reinforcement learning model to predict the heat of the data shards based on historical access data includes: constructing a reinforcement learning state space with access frequency, access interval and access request source as input, using a policy gradient algorithm to continuously optimize the access heat prediction strategy, and generating a future access heat score for each data shard.

3. The data distributed storage method according to claim 1, characterized in that: Increasing the number of copies of a shard with high access popularity includes: grading the shards according to the access popularity of the data shards to obtain data shards with high access popularity, and increasing the number of copies of the shards with high access popularity according to a preset formula.

4. The data distributed storage method according to claim 1, characterized in that: The method of compressing and transmitting the data slices with low access frequency by using the slice compression technology includes: compressing the structured data in the data slices by using a compression algorithm based on column storage, and compressing the unstructured data in the data slices based on a dictionary.

5. The data distributed storage method according to claim 1, characterized in that: The data reconstruction of the data shards stored in the storage node includes: dividing the lost shards into key shards and common shards according to the access popularity and the importance of the shards, using an improved check code mechanism to reconstruct the data, and based on the redundant characteristics of the shard data and the check code, preferentially restoring the key shard data in combination with the association information between the shards.

6. The data distributed storage method according to claim 1, characterized in that: The data migration also includes: Scan the storage status of data blocks in the sharded data, and mark discontinuous data blocks in the data shards that are distributed at different addresses or nodes; Based on the marking result, discontinuous data blocks in the shard are preferentially migrated.

7. The data distributed storage method according to claim 1, characterized in that: The method of selecting a storage node for storing the reconstructed shard data based on the health status of each node includes: selecting a storage node whose remaining storage capacity is greater than a preset threshold, sorting the storage nodes according to the health status of the nodes, and preferentially selecting a storage node with a higher sorting level to store the reconstructed shard data.

8. A data distributed storage device, characterized in that: The device comprises: An initial sharding unit is configured to perform sharding processing on the data to be stored using a data sharding algorithm to obtain an initial sharding result, wherein the initial sharding result includes more than one data shard; The replica generation unit is configured to use a reinforcement learning model to predict the heat of the data shard according to historical access data to obtain the access heat of the data shard; determine the number of replicas based on the access heat and importance of the data shard, and generate a shard replica of the data shard according to the number of replicas, wherein determining the number of replicas based on the access heat and importance of the data shard includes: increasing the number of replicas for data shards with high access heat and / or high importance, and reducing the number of replicas for data shards with low access heat and / or low importance on the premise of meeting the minimum number of replicas; A load evaluation unit is configured to monitor multi-dimensional load indicators of each storage node in real time and obtain a node load status score of each storage node, wherein the multi-dimensional load indicators include: CPU usage, storage capacity utilization, bandwidth occupancy, network latency and node health status; A data storage unit, configured to store the data shard and the shard replica in the storage node according to the access heat and the node load status score; A data migration unit is configured to migrate the data slices stored in the storage node when the storage node load is unbalanced; the data migration includes: sorting the data slices according to access popularity, and migrating the slices with low access frequency first; using the slice compression technology to compress the data slices with low access frequency before transmission, and determining the target node and migration rate according to the load indicator; The data reconstruction unit is configured to reconstruct the data shards stored in a storage node when the node health status of a storage node is abnormal or the stored data shards are lost, obtain reconstructed shard data, and select a storage node for storing the reconstructed shard data according to the health status of each node.

9. An electronic device, characterized in that: include: one or more processors; And a memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, the steps of the method described in any one of claims 1 to 7 are executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images