Data storage method and device, equipment and medium

Abstract

The embodiment of the invention provides a data storage method and device, equipment and a medium. The method comprises the steps that a data fragmentation strategy, a storage node strategy, a data fragmentation selection strategy and to-be-processed data are acquired; determining at least one data fragment according to the data fragment strategy; determining a storage node of each data fragment in the at least one data fragment according to the at least one data fragment, a storage node strategy and the at least one preset storage node; determining at least one data component of the to-be-processed data according to the to-be-processed data; according to a data fragment selection strategy, at least one data component is written into the corresponding data fragment, and through a clear data fragment strategy, a storage node strategy and the data fragment selection strategy, the data storage flexibility, reliability and access efficiency are comprehensively improved, and the management cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a data storage method, device, equipment and medium. Background Art

[0002] In today's information society, data has become the core asset of enterprises and organizations, and its scale and complexity are growing at an unprecedented rate. Faced with such a huge amount of data, how to efficiently and reliably store and manage this data has become a key issue that needs to be solved urgently. Traditional data storage solutions, such as centralized storage or simple distributed storage, have been unable to meet the increasingly diverse storage needs and high-concurrency access scenarios. On the one hand, with the diversification of data types, from structured data to unstructured data, and then to semi-structured data, the demand for data storage has become increasingly complex. Different data types have different requirements for storage methods, access speeds, and data processing capabilities, which makes it difficult for a single data storage solution to meet the needs. On the other hand, with the rapid development of technologies such as cloud computing, big data, and the Internet of Things, the concurrency of data access has increased dramatically. In high-concurrency scenarios, how to ensure rapid response and efficient processing of data has become another major challenge facing data storage systems. Traditional storage solutions are often limited by architecture design and storage media performance, and are unable to meet such high-concurrency access requirements. In addition, the popularity of heterogeneous storage environments has also made data storage and management more complicated. Different storage devices, storage protocols, and storage interfaces require more flexible and intelligent management strategies to ensure data consistency and availability.

[0003] In summary, the current data storage and management field urgently needs a new solution that is highly flexible, scalable, and intelligent to adapt to the ever-changing data storage needs and high-concurrency access scenarios. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a data storage method, apparatus, device and medium that overcome the above problems or at least partially solve the above problems.

[0005] In order to solve the above problem, an embodiment of the present invention discloses a data storage method, the method comprising:

[0006] Obtain data sharding strategy, storage node strategy, data sharding selection strategy and data to be processed;

[0007] Determine at least one data shard according to the data sharding strategy;

[0008] Determine a storage node for each data shard in the at least one data shard according to the at least one data shard, the storage node strategy and at least one preset storage node;

[0009] Determining at least one data component of the data to be processed according to the data to be processed;

[0010] According to the data shard selection strategy, the at least one data component is written into the corresponding data shard.

[0011] Optionally, the storage node strategy includes a polling strategy; and determining the storage node of each data shard in the at least one data shard according to the at least one data shard and the storage node strategy includes:

[0012] According to the polling strategy and the at least one data shard, a storage node for each data shard in the at least one data shard is determined.

[0013] Optionally, the storage node strategy includes a weight strategy; and determining the storage node of each data shard in the at least one data shard according to the at least one data shard and the storage node strategy includes:

[0014] Obtaining a weight corresponding to the at least one preset storage node;

[0015] A storage node for each data shard in the at least one data shard is determined according to the at least one preset storage node, a weight corresponding to the at least one preset storage node, and the at least one data shard.

[0016] Optionally, the storage node policy includes a resource policy; and determining the storage node of each data shard in the at least one data shard according to the at least one data shard and the storage node policy includes:

[0017] Obtaining resource utilization information corresponding to the at least one preset storage node;

[0018] A storage node for each data slice in the at least one data slice is determined according to resource utilization information corresponding to the at least one preset storage node, the at least one preset storage node, and the at least one data slice.

[0019] Optionally, determining at least one data component of the data to be processed according to the data to be processed and the data sharding strategy includes:

[0020] Obtain data processing strategies;

[0021] At least one data component of the data to be processed is determined according to the data processing strategy, the data sharding strategy and the data to be processed.

[0022] Optionally, the data processing strategy includes a data prefix processing strategy, and determining at least one data component of the data to be processed according to the data processing strategy, the data sharding strategy and the data to be processed includes:

[0023] Determining preliminary processing data according to the data prefix processing strategy and the data to be processed;

[0024] At least one data component of the data to be processed is determined according to the preliminary processed data, the data processing strategy, the data sharding strategy and the data to be processed.

[0025] Optionally, the method further comprises:

[0026] Obtain at least one data shard copy corresponding to the at least one data shard;

[0027] After writing the at least one data component into the corresponding data shard according to the data shard selection strategy, the data component in the at least one data shard is synchronized to the at least one data shard replica according to a preset synchronization algorithm.

[0028] On the other hand, an embodiment of the present invention further discloses a data storage device, the device comprising:

[0029] A data sharding strategy determination module is used to obtain data sharding strategy, storage node strategy, data sharding selection strategy and data to be processed;

[0030] A data shard determination module, configured to determine at least one data shard according to the data shard strategy;

[0031] A storage node allocation module, configured to determine a storage node for each data shard in the at least one data shard according to the at least one data shard, the storage node policy and at least one preset storage node;

[0032] A data component determination module, used to determine at least one data component of the data to be processed according to the data to be processed;

[0033] A data storage module is used to write the at least one data component into the corresponding data slice according to the data slice selection strategy.

[0034] Optionally, the storage node strategy includes a polling strategy, and the storage node allocation module includes:

[0035] A polling submodule is used to determine the storage node of each data shard in the at least one data shard according to the polling strategy and the at least one data shard.

[0036] Optionally, the storage node strategy includes a weight strategy; and the storage node allocation module includes:

[0037] A weight acquisition submodule, used to acquire a weight corresponding to the at least one preset storage node;

[0038] A weight determination submodule is used to determine the storage node of each data slice in the at least one data slice according to the at least one preset storage node, the weight corresponding to the at least one preset storage node and the at least one data slice.

[0039] Optionally, the storage node policy includes a resource policy; and the storage node allocation module includes:

[0040] A resource acquisition submodule, used to acquire resource utilization information corresponding to the at least one preset storage node;

[0041] The resource determination submodule is used to determine the storage node of each data slice in the at least one data slice according to the resource utilization information corresponding to the at least one preset storage node, the at least one preset storage node and the at least one data slice.

[0042] Optionally, the data component determination module includes:

[0043] A data processing strategy acquisition submodule is used to acquire data processing strategies;

[0044] The first component determination submodule is used to determine at least one data component of the data to be processed according to the data processing strategy, the data sharding strategy and the data to be processed.

[0045] Optionally, the data processing strategy includes a data prefix processing strategy, and the first component determination submodule includes:

[0046] a preliminary processing data determining unit, configured to determine preliminary processing data according to the data prefix processing strategy and the data to be processed;

[0047] The second component determination unit is used to determine at least one data component of the data to be processed according to the preliminary processed data, the data processing strategy, the data sharding strategy and the data to be processed.

[0048] Optionally, the device further comprises:

[0049] A shard copy determination submodule, used to obtain at least one data shard copy corresponding to the at least one data shard;

[0050] A data synchronization submodule is used to synchronize the data component in the at least one data shard to the at least one data shard replica according to a preset synchronization algorithm after writing the at least one data component into the corresponding data shard according to the data shard selection strategy.

[0051] Correspondingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the various steps of the above-mentioned data storage method embodiment when executed by the processor.

[0052] Accordingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the above-mentioned data storage method embodiment is implemented.

[0053] The embodiment of the present invention includes the following advantages: by acquiring the data sharding strategy, the storage node strategy, the data sharding selection strategy and the data to be processed, the embodiment of the present invention can comprehensively acquire the key information required for data storage and management. This comprehensive information acquisition method ensures the accuracy and efficiency in the data processing process. By determining at least one data shard according to the data sharding strategy, the data can be flexibly divided into appropriate data shards according to the characteristics and requirements of the data. This data sharding method not only improves the flexibility of data storage, but also optimizes the data access path and reduces the access delay. By determining the storage node of each data shard in the at least one data shard according to the at least one data shard, the storage node strategy and at least one preset storage node, the storage node can be intelligently selected. This storage node selection method ensures the reliability and availability of data, balances the node load, and improves the overall performance of the storage system. By determining at least one data component of the data to be processed according to the data to be processed, the embodiment of the present invention can perform refined analysis and processing of the data to be processed and extract key information from the data. This data analysis method improves the accuracy and efficiency of data processing and provides convenience for subsequent data storage and access. By writing the at least one data component into the corresponding data shard according to the data shard selection strategy, the embodiment of the present invention can dynamically select the most suitable data shard for data storage, thereby optimizing the storage distribution and access path of the data. This data shard selection method reduces data conflicts and write bottlenecks, and improves the concurrent processing capability of data writing and system throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flowchart of a data storage method embodiment of the present invention;

[0055] Figure 2 It is a schematic diagram of a combination strategy of a data storage method of the present invention;

[0056] Figure 3 is a schematic diagram of a combination strategy of another data storage method of the present invention;

[0057] Figure 4 It is a relationship diagram between data shards and data copies of the present invention;

[0058] Figure 5 It is a structural block diagram of an embodiment of a data storage device of the present invention. DETAILED DESCRIPTION

[0059] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] One of the core concepts of the embodiments of the present invention is to combine multiple data sharding strategies, data sharding selection strategies, and storage node strategy combinations to meet metadata storage needs in different scenarios and significantly improve the read and write performance of metadata.

[0061] Reference Figure 1 , shows a flow chart of the steps of an embodiment 1 of the XX method of the present invention, which may specifically include the following steps:

[0062] Step 101, obtaining a data sharding strategy, a storage node strategy, a data sharding selection strategy, and data to be processed;

[0063] At the beginning of the method, it is necessary to obtain the data sharding strategy, storage node strategy, and data sharding selection strategy to guide how to perform subsequent data storage; specifically, the data sharding strategy is used to determine how to split a large data set into multiple smaller, manageable shards. It specifies which specific data sharding method should be used to perform the splitting operation to ensure the balance and efficiency of data during storage and processing. The storage node strategy is used to determine how to distribute data shards to different storage nodes. It can be formulated based on multi-dimensional factors such as business needs and system load to optimize the efficiency of data storage and access. The data sharding selection strategy is used to determine how to select the appropriate data shard for operation when data needs to be processed. It can be formulated based on the characteristics of the data, the requirements of the query, or the performance of the system.

[0064] In one example, the data sharding selection strategy can be: 3. Data distribution supports simple hashing, range, and distributed consistent hashing. Simple and distributed consistent hashing have good data balance, and range distribution provides good data locality. Users can define their own data distribution method to determine the key

[0065] Step 102, determining at least one data shard according to the data sharding strategy;

[0066] According to the specific data sharding strategy, at least one data shard that needs to be involved in this processing can be determined;

[0067] In one example, the data sharding strategy can be a fixed number of data shards: Under this strategy, the data is split into a predetermined number of shards, and this number is fixed in the sharding strategy. Regardless of the amount of data, it will be split according to the set number of shards. For example, if the policy stipulates that 4 shards are to be created, then regardless of the amount of data, the data will be split into 4 parts.

[0068] The data sharding strategy can also be a variable number of data shards: Unlike a fixed number, the number of shards in this strategy is not predetermined, but is determined based on the amount of data or some dynamic conditions. For example, the number of shards can be automatically increased according to the growth of data, or the number of shards can be dynamically adjusted according to the load of the system.

[0069] The data sharding strategy can also be range sharding: Range sharding is a method of sharding data according to the range of a certain attribute of the data, such as timestamp, ID, etc. Each shard contains data within a specific range. For example, data can be sharded into one shard per day, one shard per month, or one shard per year according to the time range.

[0070] Data sharding strategies can also be hash sharding: perform hash operations on a certain attribute of the data, such as user ID, storage key, etc., and allocate data slices based on the hash value; list sharding: divide data according to predefined lists or collections, and each list item corresponds to a data slice; and composite sharding: combine multiple sharding strategies to allocate data to achieve more complex sharding logic.

[0071] Step 103, determining a storage node for each data shard in the at least one data shard according to the at least one data shard, the storage node strategy and at least one preset storage node;

[0072] The system needs to select a suitable storage node for each data shard based on the determined data shards, storage node strategy, and preset storage node information.

[0073] In one example, the storage node policy may also include factors such as load balancing, geographic location priority, network latency optimization, data security, etc. The system will evaluate the suitability of each storage node based on these policies. The preset storage nodes are nodes that already exist in the system and can be used to store data. These nodes may be distributed in different geographical locations and have different performance and availability characteristics.

[0074] In one embodiment, the storage node strategy includes a polling strategy, and step 102 includes the following sub-steps:

[0075] Sub-step S11, determining a storage node for each data shard in the at least one data shard according to the polling strategy and the at least one data shard.

[0076] The round-robin strategy is a simple scheduling algorithm that allocates tasks or resources in a preset order.

[0077] In one example, when the system uses a polling strategy to allocate storage nodes, a polling sequence is first initialized. This sequence is usually a circular queue that includes all available storage nodes. Then, all data shards determined in step 102 are traversed. For each data shard, the system needs to select a storage node for it. This method can allocate a storage node to each data shard in a fair and simple process. Although this polling strategy is simple, it is very effective in situations where the performance of storage nodes is similar and the load balancing requirements are not high. It can ensure that the load between storage nodes is relatively balanced while reducing the complexity of system implementation.

[0078] In one embodiment, the storage node strategy includes a weight strategy, and step 102 may further include the following sub-steps:

[0079] Sub-step S21, obtaining a weight corresponding to the at least one preset storage node;

[0080] When using a weight strategy, you first need to obtain the weight information of each preset storage node. The weight is usually a numerical value that represents the relative importance or ability of the storage node in processing data or providing services. The weight can be determined based on a variety of factors, such as the performance, capacity, availability, geographic location, network latency, etc. of the storage node.

[0081] Sub-step S22, determining a storage node for each data shard in the at least one data shard according to the at least one preset storage node, the weight corresponding to the at least one preset storage node and the at least one data shard.

[0082] In one example, when the system uses a weighted strategy to allocate storage nodes, it first builds a candidate list of storage nodes based on the preset storage nodes and their weights. This list contains all available storage nodes and their respective weights; next, the system calculates the probability of each storage node being selected based on its weight, thereby obtaining a probability distribution. Then, the system randomly selects a storage node for each data shard based on the calculated selection probability. In this way, the method can consider the weight factor of the storage node and select a relatively better storage node for each data shard. This weighted strategy performs well in considering factors such as storage node performance differences, load balancing, and data security, and can help the system achieve more efficient and reliable data storage and management.

[0083] In one embodiment, the storage node policy includes a resource policy, and step 102 may further include the following sub-steps:

[0084] Sub-step S31, obtaining resource utilization information corresponding to the at least one preset storage node;

[0085] When using resource policies as storage node policies, the system will first collect resource utilization information for each preset storage node. This resource information may cover multiple aspects, such as the remaining amount of storage space, CPU and memory usage, I / O operation frequency, network bandwidth usage, and node response time.

[0086] In one example, the collection method may be to use a dedicated monitoring tool or system service to track the resource usage of the storage node in real time; or to obtain its current resource utilization data by periodically sending query requests to the storage node; or to perform in-depth analysis of the log files generated by the storage node to extract relevant information on resource usage.

[0087] Sub-step S32, determining a storage node for each data slice in the at least one data slice according to resource utilization information corresponding to the at least one preset storage node, the at least one preset storage node and the at least one data slice.

[0088] The system will select the most suitable storage node for each data shard based on the collected resource utilization information, combined with the preset storage node list and the data shards to be allocated.

[0089] In one example, when the system applies resource policies to allocate storage nodes, it will give priority to storage nodes with sufficient resources, such as ample storage space, high CPU and memory idle rates, to ensure efficient data storage and access; on this basis, it can also select storage nodes with the best performance, such as nodes with fast I / O operations and low network latency, based on the characteristics of data sharding and access patterns. This approach can achieve load balancing of the entire system by avoiding overuse of certain node resources and idleness of other node resources through reasonable data allocation.

[0090] Step 104, determining at least one data component of the data to be processed according to the data to be processed;

[0091] In one example, the method of determining the data components may be: comprehensively analyzing the input data to be processed, including the data type such as text, image, audio, etc., the structure such as table, sequence, graph, etc., and the meaning and value range of each field or attribute contained in the data.

[0092] After the analysis is completed, the basic units or elements that make up the data, i.e., data components, are identified. These data components may be a column in a data table, a word in a text, a pixel or pixel block in an image, an audio frame in an audio, etc.

[0093] Determine how many data components are contained in the data to be processed, and the specific type of each data component. For example, in text processing, data components may be words, phrases, or sentences; in image processing, data components may be color channels, texture features, or edge information.

[0094] According to the identified data components, the data components are extracted from the data to be processed. This involves operations such as parsing, splitting, and converting the data to ensure that each data component can be correctly identified and extracted. It should be clear that the data components can also represent other types of data. This example only exemplifies one of the methods. The method for determining the data components can also be other methods, and the embodiments of the present invention do not limit this.

[0095] In one embodiment, step 104 includes the following sub-steps:

[0096] Sub-step S41, obtaining a data processing strategy;

[0097] The data processing strategy defines how to process, analyze or transform the data to be processed. This strategy may include data preprocessing methods such as cleaning, normalization, encoding, etc., feature selection standards, data conversion methods, and any specific data processing requirements or constraints. The data processing strategy can be determined by reading predefined processing strategies from configuration files or databases; obtaining processing strategies through user input or interface selection; dynamically generating processing strategies based on data type, source or application scenario, etc.

[0098] In one example, the data processing strategy may be hash feature extraction, which uses a hash function to perform a hash operation on any part of the storage key or data to generate a hash value of a fixed length. This hash value can be used as a unique identifier or feature of the data for data retrieval, deduplication, distribution, etc. The advantage of hash extraction is that it can quickly map a large amount of data into a smaller space while maintaining the uniqueness and consistency of the data.

[0099] Sub-step S42, determining at least one data component of the data to be processed according to the data processing strategy, the data sharding strategy and the data to be processed.

[0100] In one embodiment, the data processing strategy includes a data prefix processing strategy, and step S42 includes the following sub-steps:

[0101] Sub-step S51, determining preliminary processing data according to the data prefix processing strategy and the data to be processed;

[0102] Sub-step S52, determining at least one data component of the data to be processed according to the preliminary processed data, the data processing strategy, the data sharding strategy and the data to be processed.

[0103] In one example, the data prefix processing strategy can be a prefix extraction of a specified length, which involves extracting a prefix of a specified length from a storage key. The prefix is ​​the beginning of the storage key, which may contain information about data classification, source, or other important information. By extracting the prefix, the system can preliminarily classify or group the data, which helps with subsequent data processing and management. A storage key is an identifier used to uniquely identify a data item or data record in the field of data storage and management. It is usually a string, a number, or a combination of the two, used to locate and retrieve specific data in a storage system.

[0104] In another example, the data processing strategy can also combine prefix extraction and hash extraction. For example, prefix extraction is used for preliminary classification: first, the system performs prefix extraction on the storage key and divides the data into different categories or groups according to the prefix. Next, the system can perform hash extraction on the data in each category or group to generate a unique hash value as the identifier of the data. These hash values ​​can be used for data retrieval, deduplication and distribution. Through the hash value, the system can quickly locate the location of the data in the storage system, and achieve fast access and efficient management of the data. Prefix extraction can provide preliminary classification information for the data, which helps the system better understand and manage the data. : Hash extraction can generate a unique hash value as the identifier of the data, ensuring the uniqueness and consistency of the data, while facilitating the retrieval and distribution of the data. By combining prefix extraction and hash extraction, the system can locate and process data more quickly and improve the efficiency of data processing.

[0105] Step 105: write the at least one data component into the corresponding data shard according to the data shard selection strategy.

[0106] The system will decide how to split the data into data slices and determine where each data slice should be stored based on the preset data shard selection strategy. Data shard selection strategies include simple hashing, range distribution, distributed consistent hashing, and other methods. Simple hashing is to calculate the hash value of the data and map it to a limited number of data slices to achieve uniform distribution of data. Range distribution is to allocate data slices according to the range of a certain attribute of the data (such as timestamp, ID, etc.). This method can provide better data locality, that is, adjacent or related data is more likely to be stored in the same data slice, thereby optimizing query and access performance. Distributed consistent hashing is a hash algorithm widely used in distributed systems. It can maintain the consistency of data distribution when nodes dynamically join or leave. This method also has good data balance and can adapt to the dynamic changes of distributed systems.

[0107] For example, users can customize the data distribution method according to their own needs and data characteristics, and decide which key (or attribute) is used for the allocation of data slices.

[0108] In one embodiment, the method further comprises:

[0109] At least one data component of the data to be processed is determined based on the preliminary processed data, the data processing strategy, the data sharding strategy and the data to be processed. Generating data components based on the data after prefix extraction can make the generated data components more quickly located and retrieved, and can also participate in subsequent operations more conveniently.

[0110] In one embodiment, the method further comprises:

[0111] Obtain at least one data shard copy corresponding to the at least one data shard;

[0112] In the data processing and storage architecture, in order to ensure high availability and fault tolerance of data, data sharding and data replication strategies are often adopted.

[0113] First, you need to clarify the replica information corresponding to each data shard. These replicas can be set up for data redundancy, load balancing, or disaster recovery. The system will determine which data shards need to create replicas and where these replicas should be stored based on the preset replica strategy, such as a fixed number of replicas for each data shard, or dynamically adjusting the number of replicas based on the importance and access frequency of the data.

[0114] After writing the at least one data component into the corresponding data shard according to the data shard selection strategy, the data component in the at least one data shard is synchronized to the at least one data shard replica according to a preset synchronization algorithm.

[0115] After the data is successfully written to the data shard, the system needs to ensure that the data is also copied to the corresponding shard replica. This step is completed through a preset synchronization algorithm. The synchronization algorithm can be: Asynchronous replication: After the data is written to the primary shard, it is copied to the replica in an asynchronous manner. This method may sacrifice a certain degree of data consistency in exchange for higher write performance. Synchronous replication: When the data is written to the primary shard, it is also synchronously copied to all replicas to ensure that the data is consistent at all locations, but it may increase the delay of writing. Semi-synchronous replication: Between asynchronous and synchronous, after the data is written to the primary shard, it is necessary to wait for at least one replica to confirm that the data has been received before the write is considered successful. And other technologies, it should be clear that the synchronization algorithm can also be other technologies, and the embodiments of the present invention are not limited to this.

[0116] In one embodiment, before obtaining the data sharding strategy, the storage node strategy, the data sharding selection strategy, and the data to be processed, it is necessary to first determine the interface type of the access request;

[0117] In one example, the interface type of the access request includes: transaction type, batch type and simple type;

[0118] Reference Figure 2 , showing a schematic diagram of a combined strategy of a data storage method of the present invention:

[0119] First, the access interface types are batch type interface and transaction type interface. The batch type interface can realize batch addition, deletion and modification functions, and needs to provide atomicity, that is, either all are successful or one fails and then rollback; the transaction type interface needs to provide transactional addition, deletion, modification and query functions to meet the ACID characteristics (atomicity, consistency, isolation, persistence), and then the data shard generator generates data shards according to the fixed sharding strategy; the storage node allocates storage nodes to the data shards according to the polling strategy; after the storage nodes are allocated to the data shards, the data is allocated to the data shards according to the simple hash strategy, and finally, the storage engine stores the data in a read-friendly access mode and a persistent storage mode. The read-friendly access mode means that the mode is designed to optimize read operations to improve read efficiency and performance; persistence refers to the long-term storage of data on non-volatile storage devices (such as hard disks, SSDs, cloud storage, etc.) so that the data can be retained after the program or system is shut down.

[0120] Reference Figure 3 , showing a schematic diagram of a combination strategy of another data storage method of the present invention:

[0121] First, the access interface types are batch type interface and transaction type interface. The simple type interface supports simple addition, deletion and modification. Then the data shard generator generates data shards according to the fixed range sharding strategy. The storage node allocates storage nodes to the data shards according to the resource strategy. After the storage nodes are allocated to the data shards, the data is allocated to the data shards according to the range strategy. Finally, the storage engine performs storage range read optimization according to the access mode of range optimization and the storage method of persistence. It is an optimization strategy for query operations of databases or storage systems, which aims to improve the efficiency of retrieving data within a specified range. This optimization technology is particularly suitable for scenarios where large amounts of data need to be processed and query operations are frequent.

[0122] Reference Figure 4 , showing a relationship diagram between data sharding and data replicas of the present invention;

[0123] As shown in the figure, after the system generates Shard 1, Shard 2, ... Shard N, a total of N shards, according to the data sharding strategy; a master copy and several slave copies are assigned to each data shard. The number of slave copies can be designed according to business needs. When data needs to be written to a data shard, it is first written to the master copy of the data, and then the written data is synchronized to the remaining slave copies through a strong consistency algorithm.

[0124] By acquiring the data sharding strategy, storage node strategy, data sharding selection strategy and data to be processed, the embodiment of the present invention can comprehensively acquire the key information required for data storage and management. This comprehensive information acquisition method ensures the accuracy and efficiency in the data processing process. By determining at least one data shard according to the data sharding strategy, the data can be flexibly divided into appropriate data shards according to the characteristics and requirements of the data. This data sharding method not only improves the flexibility of data storage, but also optimizes the data access path and reduces the access delay. By determining the storage node of each data shard in the at least one data shard according to the at least one data shard, the storage node strategy and at least one preset storage node, the storage node can be intelligently selected. This storage node selection method ensures the reliability and availability of data, while balancing the node load and improving the overall performance of the storage system. By determining at least one data component of the data to be processed according to the data to be processed, the embodiment of the present invention can perform refined analysis and processing of the data to be processed and extract key information from the data. This data analysis method improves the accuracy and efficiency of data processing and provides convenience for subsequent data storage and access. By writing the at least one data component into the corresponding data shard according to the data shard selection strategy, the embodiment of the present invention can dynamically select the most suitable data shard for data storage, thereby optimizing the storage distribution and access path of the data. This data shard selection method reduces data conflicts and write bottlenecks, and improves the concurrent processing capability of data writing and system throughput.

[0125] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0126] Reference Figure 5 , shows a structural block diagram of an embodiment of the XX device of the present invention, which may specifically include the following modules:

[0127] The data sharding strategy determination module 201 is used to obtain the data sharding strategy, storage node strategy, data sharding selection strategy and data to be processed;

[0128] A data shard determination module 202, configured to determine at least one data shard according to the data shard strategy;

[0129] A storage node allocation module 03, configured to determine a storage node for each data shard in the at least one data shard according to the at least one data shard, the storage node policy and at least one preset storage node;

[0130] A data component determination module 204, configured to determine at least one data component of the data to be processed according to the data to be processed;

[0131] The data storage module 205 is used to write the at least one data component into the corresponding data slice according to the data slice selection strategy.

[0132] In one embodiment, the storage node strategy includes a polling strategy, and the storage node allocation module includes:

[0133] A polling submodule is used to determine the storage node of each data shard in the at least one data shard according to the polling strategy and the at least one data shard.

[0134] In one embodiment, the storage node strategy includes a weight strategy; the storage node allocation module includes:

[0135] A weight acquisition submodule, used to acquire a weight corresponding to the at least one preset storage node;

[0136] A weight determination submodule is used to determine the storage node of each data slice in the at least one data slice according to the at least one preset storage node, the weight corresponding to the at least one preset storage node and the at least one data slice.

[0137] In one embodiment, the storage node policy includes a resource policy; the storage node allocation module includes:

[0138] A resource acquisition submodule, used to acquire resource utilization information corresponding to the at least one preset storage node;

[0139] The resource determination submodule is used to determine the storage node of each data slice in the at least one data slice according to the resource utilization information corresponding to the at least one preset storage node, the at least one preset storage node and the at least one data slice.

[0140] In one embodiment, the data component determination module includes:

[0141] A data processing strategy acquisition submodule is used to acquire data processing strategies;

[0142] The first component determination submodule is used to determine at least one data component of the data to be processed according to the data processing strategy, the data sharding strategy and the data to be processed.

[0143] In one embodiment, the data processing strategy includes a data prefix processing strategy, and the first component determination submodule includes:

[0144] a preliminary processing data determining unit, configured to determine preliminary processing data according to the data prefix processing strategy and the data to be processed;

[0145] The second component determination unit is used to determine at least one data component of the data to be processed according to the preliminary processed data, the data processing strategy, the data sharding strategy and the data to be processed.

[0146] In one embodiment, the device further comprises:

[0147] A shard copy determination submodule, used to obtain at least one data shard copy corresponding to the at least one data shard;

[0148] A data synchronization submodule is used to synchronize the data component in the at least one data shard to the at least one data shard replica according to a preset synchronization algorithm after writing the at least one data component into the corresponding data shard according to the data shard selection strategy.

[0149] By acquiring the data sharding strategy, storage node strategy, data sharding selection strategy and data to be processed, the embodiment of the present invention can comprehensively acquire the key information required for data storage and management. This comprehensive information acquisition method ensures the accuracy and efficiency in the data processing process. By determining at least one data shard according to the data sharding strategy, the data can be flexibly divided into appropriate data shards according to the characteristics and requirements of the data. This data sharding method not only improves the flexibility of data storage, but also optimizes the data access path and reduces the access delay. By determining the storage node of each data shard in the at least one data shard according to the at least one data shard, the storage node strategy and at least one preset storage node, the storage node can be intelligently selected. This storage node selection method ensures the reliability and availability of data, while balancing the node load and improving the overall performance of the storage system. By determining at least one data component of the data to be processed according to the data to be processed, the embodiment of the present invention can perform refined analysis and processing of the data to be processed and extract key information from the data. This data analysis method improves the accuracy and efficiency of data processing and provides convenience for subsequent data storage and access. By writing the at least one data component into the corresponding data shard according to the data shard selection strategy, the embodiment of the present invention can dynamically select the most suitable data shard for data storage, thereby optimizing the storage distribution and access path of the data. This data shard selection method reduces data conflicts and write bottlenecks, and improves the concurrent processing capability of data writing and system throughput.

[0150] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0151] An embodiment of the present invention further provides an electronic device, including:

[0152] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data storage method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0153] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned data storage method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0154] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0155] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0156] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0159] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0160] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0161] The data storage method, device, equipment and medium provided by the present invention are described in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data storage method, characterized in that: The method comprises: Obtain data sharding strategy, storage node strategy, data sharding selection strategy and data to be processed; Determine at least one data shard according to the data sharding strategy; Determine a storage node for each data shard in the at least one data shard according to the at least one data shard, the storage node strategy and at least one preset storage node; Determining at least one data component of the data to be processed according to the data to be processed; According to the data shard selection strategy, the at least one data component is written into the corresponding data shard.

2. The method according to claim 1, characterized in that The storage node strategy includes a polling strategy; and determining the storage node of each data shard in the at least one data shard according to the at least one data shard and the storage node strategy includes: According to the polling strategy and the at least one data shard, a storage node for each data shard in the at least one data shard is determined.

3. The method according to claim 1, characterized in that The storage node strategy includes a weight strategy; and determining the storage node of each data shard in the at least one data shard according to the at least one data shard and the storage node strategy includes: Obtaining a weight corresponding to the at least one preset storage node; A storage node for each data shard in the at least one data shard is determined according to the at least one preset storage node, a weight corresponding to the at least one preset storage node, and the at least one data shard.

4. The method according to claim 1, characterized in that: The storage node strategy includes a resource strategy; and determining the storage node of each data shard in the at least one data shard according to the at least one data shard and the storage node strategy includes: Obtaining resource utilization information corresponding to the at least one preset storage node; A storage node for each data slice in the at least one data slice is determined according to resource utilization information corresponding to the at least one preset storage node, the at least one preset storage node, and the at least one data slice.

5. The method according to any one of claims 2 to 4, characterized in that: The determining, according to the data to be processed and the data sharding strategy, at least one data component of the data to be processed includes: Obtain data processing strategies; At least one data component of the data to be processed is determined according to the data processing strategy, the data sharding strategy and the data to be processed.

6. The method according to claim 5, characterized in that The data processing strategy includes a data prefix processing strategy; the determining at least one data component of the data to be processed according to the data processing strategy, the data sharding strategy and the data to be processed includes: Determining preliminary processing data according to the data prefix processing strategy and the data to be processed; At least one data component of the data to be processed is determined according to the preliminary processed data, the data processing strategy, the data sharding strategy and the data to be processed.

7. The method according to claim 6, characterized in that The method further comprises: Obtain at least one data shard copy corresponding to the at least one data shard; After writing the at least one data component into the corresponding data shard according to the data shard selection strategy, the data component in the at least one data shard is synchronized to the at least one data shard replica according to a preset synchronization algorithm.

8. A data storage device, characterized in that: The device comprises: A data sharding strategy determination module is used to obtain data sharding strategy, storage node strategy, data sharding selection strategy and data to be processed; A data shard determination module, configured to determine at least one data shard according to the data shard strategy; A storage node allocation module, configured to determine a storage node for each data shard in the at least one data shard according to the at least one data shard, the storage node policy and at least one preset storage node; A data component determination module, used to determine at least one data component of the data to be processed according to the data to be processed; A data storage module is used to write the at least one data component into the corresponding data slice according to the data slice selection strategy.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the data storage method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data storage method according to any one of claims 1 to 7 are implemented.

Images