Data distributed storage method and device

Abstract

The invention discloses a data distributed storage method and device. The method comprises the steps that a to-be-stored relational database table is obtained, multi-dimensional data features of the relational database table are determined, and the dimensions of the data features comprise at least one of the table size, the field type, the field incidence relation and the access state; according to the multi-dimensional data features, a fragmentation strategy for the relational database table is determined, the relational database table is divided into a plurality of data fragments according to the fragmentation strategy, and the fragmentation strategy comprises at least one of range fragmentation, hash fragmentation and list fragmentation; and storing the plurality of data fragments to a plurality of storage nodes in a distributed manner, and recording a mapping relationship between each data fragment and each storage node. The technical problem that the data storage and use efficiency is easily influenced by unreasonable data fragmentation in the existing distributed storage technology is solved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method and device for distributed data storage. Background Art

[0002] In a stand-alone environment, the data storage capacity of traditional relational databases is limited by the physical limitations of server hardware, such as disk capacity and memory size. With the explosive growth of data volume, stand-alone database systems quickly encounter performance bottlenecks, which are reflected in the increase of query response time and the increase of write operation delay. To solve these problems, early distributed relational databases introduced data sharding strategies to distribute data to multiple computing nodes. Such methods usually rely on simple rules, such as allocating data according to the value range of a specific field or hash results. However, these sharding schemes based on fixed rules are not flexible enough. Once the data is unevenly distributed - that is, the amount of data on some nodes far exceeds that of other nodes - some nodes will be overloaded, while the resource utilization rate of the remaining nodes will be low, thus affecting the efficiency and performance of the overall system.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a data distributed storage method and device to at least solve the technical problem that the existing distributed storage technology is prone to affect the efficiency of data storage and use due to unreasonable data fragmentation.

[0005] According to one aspect of an embodiment of the present application, a data distributed storage method is provided, comprising: obtaining a relational database table to be stored, and determining multi-dimensional data features of the relational database table, wherein the dimensions of the data features include at least one of the following: table size, field type, field association relationship, and access status; determining a sharding strategy for the relational database table based on the multi-dimensional data features, and dividing the relational database table into multiple data shards based on the sharding strategy, wherein the sharding strategy includes at least one of the following: range sharding, hash sharding, and list sharding; distributively storing the multiple data shards to multiple storage nodes, and recording a mapping relationship between each data shard and each storage node.

[0006] Optionally, determining multi-dimensional data features of a relational database table includes: determining a first data feature of a table size dimension of the relational database table through metadata analysis and partial sample data scanning, wherein the first data feature includes at least one of the following: the number of records in the table and the storage space occupied; and / or, analyzing the field data type and distribution, determining a second data feature of a field type dimension of the relational database table, wherein the second data feature includes at least one of the following: text fields, numeric fields, date fields, and the proportion of each type of field in the relational database table; and / or, identifying the association between different fields, determining a third data feature of a field association relationship dimension of the relational database table, wherein the third data feature includes at least one of the following: primary key constraints, foreign key constraints, and query combinations between fields; and / or, analyzing query logs, determining a fourth data feature of an access status dimension of the relational database table, wherein the fourth data feature includes at least one of the following: access frequency, access time, and access mode.

[0007] Optionally, a sharding strategy for a relational database table is determined based on multi-dimensional data features, including: when fields exceeding a first preset proportion in the relational database table are text fields, and query fields are concentrated on text fields, determining to perform hash sharding on the relational database table; when fields exceeding a second preset proportion in the relational database table are continuous numeric fields or date fields, and query fields are concentrated on numeric ranges or date ranges, determining to perform range sharding on the relational database table; when the size of the relational database table exceeds a preset threshold, determining to perform list sharding on the relational database table.

[0008] Optionally, the method also includes: when determining the sharding strategy, obtaining the number of storage nodes, and determining the granularity of each data shard to be divided according to the size of the relational database table and the number of storage nodes; determining the degree of association between each field in the relational database table, and determining to divide multiple fields with a degree of association higher than a preset association threshold into the same data shard.

[0009] Optionally, multiple data shards are distributedly stored in multiple storage nodes, including: respectively determining the average access frequency of multiple fields in each data shard; obtaining node status information of each storage node, wherein the node status information includes at least one of the following: node capacity, node load, node read and write speed, and node network bandwidth; based on the load balancing principle, determining the storage node corresponding to each data shard according to the average access frequency and the node status information, wherein the storage node corresponding to the data shard with a higher average access frequency has a greater node read and write speed and / or node network bandwidth.

[0010] Optionally, the mapping relationship between each data shard and each storage node is recorded, including: recording the mapping relationship between each data shard and each storage node in the form of a key-value pair, and storing each mapping relationship in a target cache, wherein the key in the key-value pair is the shard identifier of the data shard, and the value in the key-value pair is the node identifier of the storage node; periodically determining the access frequency of each data shard, and updating each mapping relationship in the target cache using an algorithm using a minimum frequency, wherein the mapping relationship corresponding to the data shard with the lowest access frequency is deleted first.

[0011] Optionally, the method also includes: in response to a query instruction of the target object, determining a target data shard corresponding to the query instruction; determining a target storage node corresponding to the target data shard based on a mapping relationship stored in a target cache, and sending a data read instruction corresponding to the query instruction to the target storage node; and feeding back the data fed back by the target storage node to the target object.

[0012] According to another aspect of an embodiment of the present application, a data distributed storage device is also provided, including: an acquisition module, used to acquire a relational database table to be stored, and determine multi-dimensional data features of the relational database table, wherein the dimensions of the data features include at least one of the following: table size, field type, field association relationship, access status; a sharding module, used to determine a sharding strategy for the relational database table based on the multi-dimensional data features, and divide the relational database table into multiple data shards based on the sharding strategy, wherein the sharding strategy includes at least one of the following: range sharding, hash sharding, list sharding; a storage module, used to distribute and store multiple data shards to multiple storage nodes, and record the mapping relationship between each data shard and each storage node.

[0013] According to another aspect of an embodiment of the present application, a computer program product is further provided, the computer program product comprising: a computer program, wherein when the computer program is executed by a processor, the above-mentioned data distributed storage method is implemented.

[0014] According to another aspect of an embodiment of the present application, an electronic device is further provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned data distributed storage method through the computer program.

[0015] In the embodiment of the present application, by collecting multi-dimensional data features such as table size, field type, field association and access status of the relational database table, the characteristics and access mode of the data can be fully understood, which provides an accurate data basis for the subsequent sharding strategy selection, ensuring that the sharding strategy takes into account both the characteristics of the data structure and the actual needs of data access, thereby improving the efficiency of data storage and query; the sharding strategy determined based on multi-dimensional data features, such as range sharding, hash sharding or list sharding, can solve the problem of data skew in a targeted manner, optimize data distribution, improve access speed, and reduce the overhead of cross-node queries; the data shards are distributed and stored in multiple storage nodes, which not only utilizes the hardware resources of multiple nodes, improves storage capacity and input / output performance, but also realizes fast data positioning and access by recording the mapping relationship between shards and nodes, thereby solving the technical problem that the existing distributed storage technology is prone to affect data storage and utilization efficiency due to unreasonable data sharding. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a flow chart of an optional data distributed storage method according to an embodiment of the present application;

[0018] Figure 2 is a schematic structural diagram of an optional data distributed storage device according to an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] Example 1

[0023] According to an embodiment of the present application, a data distributed storage method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0024] Figure 1 is a flow chart of a data distributed storage method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0025] Step S102, obtaining a relational database table to be stored, and determining multi-dimensional data features of the relational database table, wherein the dimensions of the data features include at least one of the following: table size, field type, field association, and access status;

[0026] Step S104, determining a sharding strategy for the relational database table according to the multi-dimensional data features, and dividing the relational database table into a plurality of data shards according to the sharding strategy, wherein the sharding strategy includes at least one of the following: range sharding, hash sharding, and list sharding;

[0027] Step S106, distributing and storing the multiple data shards to the multiple storage nodes, and recording the mapping relationship between each data shard and each storage node.

[0028] The following describes each step of the data distributed storage method in conjunction with a specific implementation process.

[0029] First, a relational database table to be stored is obtained, and multi-dimensional data features of the relational database table are determined, wherein the dimensions of the data features include at least one of the following: table size, field type, field association relationship, and access status.

[0030] A relational database table is the basic structure for storing data in a relational database management system. It organizes data in the form of a two-dimensional table, where each row is called a record or tuple, each column is called a field or attribute, and each field has a specific data type, such as integer, floating point, character, date and time, etc., which is used to define the format and nature of the data in the column.

[0031] As an optional implementation, the following method may be used to determine the multi-dimensional data features of a relational database table:

[0032] The first data feature of the table size dimension of the relational database table is determined by metadata parsing and scanning of some sample data, wherein the first data feature includes at least one of the following: the number of records in the table and the storage space occupied. This method takes into account that under normal circumstances, the database management system will provide metadata, which contains the structural information of the table, such as field name, data type, and table size, etc. Through metadata parsing, the basic information of the table can be quickly obtained. At the same time, in order to more accurately understand the actual storage occupancy of the data, some sample data of the table is scanned to analyze the size of the data in the actual storage, including considering additional factors such as data compression ratio and index occupancy. For example, for the user order information table, its field information can be obtained through metadata, and then the average storage size of each record can be estimated by scanning some order records, thereby calculating the storage occupancy of the entire table.

[0033] Analyze the field data type and distribution, and determine the second data feature of the field type dimension of the relational database table, where the second data feature includes at least one of the following: text field, numeric field, date field, and the proportion of each type of field in the relational database table. What needs to be analyzed here is the data type of the field in the table and the distribution of different types of data. For example, the order table may contain fields such as order number (text type), order amount (numeric type), and order date (date type). Analyzing the type of each field and its distribution in the data helps to understand the structure of the data and provide information for the subsequent selection of sharding strategies. For example, it is found that the numerical distribution of the order amount field is relatively uniform, while the order date shows obvious seasonal fluctuations. This information is crucial for choosing a range-based or hash-based sharding strategy.

[0034] Identify the associations between different fields, and determine the third data feature of the field association dimension of the relational database table, wherein the third data feature includes at least one of the following: primary key constraint, foreign key constraint, and query combination between fields. In a relational database, the associations between fields (such as primary key and foreign key constraints) reflect the logical connection between data, which is very important for efficient query and maintaining data consistency. By identifying these associations, the dependencies between data tables can be understood, so that the integrity of these relationships can be considered during sharding, avoiding the complexity of querying cross-shard data after sharding. For example, the order table is associated with the user information table through the user ID (foreign key), which indicates that hash sharding of the user ID should be considered during sharding in order to efficiently query all orders for a specific user.

[0035] Analyze the query log to determine the fourth data feature of the access status dimension of the relational database table, wherein the fourth data feature includes at least one of the following: access frequency, access time, and access mode. The query log records the history of database operations, including the frequency, time point, and query mode of queries. By analyzing the query log, you can find the hot spots of data access, that is, which fields or which query modes are most frequently accessed by users. This will help prioritize these hot fields when sharding, ensure that frequently accessed data can be quickly located and accessed, and improve the overall performance of the database. For example, queries by order date range in the order table are the most frequent, which indicates that sharding based on order date (such as sharding by month) will significantly improve the efficiency of such queries.

[0036] It should be noted that the above method for determining the multi-dimensional data features of a relational database table may adopt any one of the methods or may adopt multiple methods simultaneously.

[0037] After determining the multi-dimensional data features of the relational database table, a sharding strategy for the relational database table is determined based on the multi-dimensional data features, and the relational database table is divided into multiple data shards based on the sharding strategy, wherein the sharding strategy includes at least one of the following: range sharding, hash sharding, and list sharding.

[0038] As an optional implementation, the process of determining a sharding strategy for a relational database table according to multi-dimensional data features may include the following steps:

[0039] In the case where more than a first preset ratio of fields in the relational database table are text fields and the query fields are concentrated on the text fields, it is determined to perform hash sharding on the relational database table. For example, if a table contains a large number of user IDs (text type) and most queries are based on user IDs, then using hash sharding can ensure that each user ID is evenly distributed to different data shards, thereby avoiding the generation of hot data areas and improving the uniformity and efficiency of data access.

[0040] In the case where the fields exceeding the second preset ratio in the relational database table are continuous numeric fields or date fields, and the query fields are concentrated in the numeric range or date range, it is determined to perform range sharding on the relational database table. For example, for an order table, if the query mainly involves the order date range, the order data can be divided into different shards by month or quarter according to the date, and each shard contains data within a time range, so that the data within a specific time range can be quickly located, thereby improving query efficiency.

[0041] When the size of the relational database table exceeds a preset threshold, it is determined to perform list sharding on the relational database table. List sharding is a list-based segmentation method, which is usually used for very large data sets. The data can be segmented according to the order in the list, and each part of the data becomes a shard. For example, for a very large user information table, users can be segmented into multiple shards in order of ID. Each shard contains user information within a certain range, which is convenient for management and query.

[0042] In addition, when determining the sharding strategy, the number of storage nodes can be obtained, and the granularity of each data shard to be divided can be determined based on the size of the relational database table and the number of storage nodes, so as to ensure that the data can be evenly distributed to each node. For example, if the total size of the database table is 50GB, and there are 10 storage nodes, the size of each shard should be approximately 5GB, which can avoid the situation where the storage resources of some nodes are overused and the resources of other nodes are idle.

[0043] Furthermore, considering the degree of association between fields, the degree of association between fields in the relational database table can also be determined, and multiple fields with a degree of association higher than a preset association threshold can be divided into the same data shard. For example, the order table and the user information table are associated through the user ID. In order to ensure query efficiency and integrity, the order data and user information data involving the user ID can be divided into the same shard.

[0044] It should be noted that all sharding strategies can be used individually or in combination. In addition, the following sharding strategies can also be used:

[0045] Composite sharding (also called multi-dimensional sharding) combines two or more sharding strategies to distribute data. For example, you can first perform hash sharding based on user ID, and then perform range sharding based on order date within each shard. This approach can simultaneously meet the needs of different query patterns and improve the execution efficiency of complex queries.

[0046] Alternatively, data shards can be divided based on the value range of a specific field (usually the primary key); based on transaction processing requirements, related data can be stored on the same or a group of shards to simplify transaction processing; data can be distributed based on the user's geographic location to ensure that the user can obtain data from the nearest data center; the imbalance of data access can be considered when sharding, and hot data and cold data can be stored separately to avoid concentrating resources on a few shards; the size and number of shards can be dynamically adjusted based on the growth of data volume and changes in query load.

[0047] After obtaining multiple data shards, the multiple data shards are distributed and stored in multiple storage nodes, and the mapping relationship between each data shard and each storage node is recorded.

[0048] As an optional implementation, the process of distributing and storing multiple data shards to multiple storage nodes may take the following steps:

[0049] The average access frequency of multiple fields in each data shard is determined separately. This information is crucial for the subsequent intelligent allocation of data shards, as it reflects the popularity and importance of different data shards. Then, the node status information of each storage node is obtained, wherein the node status information includes at least one of the following: node capacity, node load, node read and write speed, and node network bandwidth. This information is the key basis for determining the storage location of the data shard, ensuring that the data can be stored on the node with the best performance at the moment. Subsequently, based on the load balancing principle, the storage node corresponding to each data shard is determined according to the average access frequency and the node status information, wherein the storage node corresponding to the data shard with a higher average access frequency has a greater node read and write speed and / or node network bandwidth. In addition, the node allocation can be periodically updated and the mapping relationship can be adaptively adjusted.

[0050] For example, suppose there are 5 storage nodes, the storage capacity of node 1 is 10TB, the disk I / O (input / output) speed is 500MB / s, and the network bandwidth is 1Gbps; the storage capacity of node 2 is 8TB, the disk I / O speed is 400MB / s, and the network bandwidth is 1Gbps; the storage capacity of node 3 is 6TB, the disk I / O speed is 300MB / s, and the network bandwidth is 500Mbps; the storage capacity of node 4 is 12TB, the disk I / O speed is 600MB / s, and the network bandwidth is 2Gbps; the storage capacity of node 5 is 5TB, the disk I / O speed is 200MB / s, and the network bandwidth is 500Mbps.

[0051] Assign shards with frequent read and write operations (such as the current month shard of the order table, which has frequent read and write operations) to node 4 because it has a higher disk I / O speed and network bandwidth, and assign historical order shards with lower access frequency (such as order data from 3 years ago) to node 5 to utilize its storage capacity. At the same time, by real-time monitoring of node load conditions, such as CPU (Central Processing Unit) utilization, memory utilization, disk I / O utilization, etc., avoid overloading a node and ensure load balance of each node.

[0052] In addition, the method of distributing and storing multiple data shards to multiple storage nodes is not limited to dynamic allocation based on access frequency and performance parameters, and other methods can also be used, such as:

[0053] Based on the geographic location information of data access, data is reasonably allocated to geographically close nodes to reduce network latency; based on the affinity between data or the access pattern of the application, data that is frequently accessed together is allocated to the same or adjacent nodes; based on the cost and performance of the storage node, data shards are allocated to the most cost-effective nodes while ensuring performance requirements; historical data and machine learning algorithms are used to predict future data access patterns and allocate data shards accordingly; algorithms (such as binary allocation, random weighted selection, etc.) are used to ensure that the amount of data on each storage node is roughly the same to avoid excessive load on a single point; nodes are organized into a hierarchical structure, and data is stored in layers on nodes at different levels based on the frequency and importance of data access.

[0054] Among them, as an optional implementation method, the process of recording the mapping relationship between each data shard and each storage node can take the following steps: record the mapping relationship between each data shard and each storage node in the form of a key-value pair, and store each mapping relationship in the target cache, wherein the key in the key-value pair is the shard identifier of the data shard, and the value in the key-value pair is the node identifier of the storage node; in addition, the access frequency of each data shard will be periodically calculated, and the minimum frequency usage algorithm will be used to update each mapping relationship in the target cache, wherein the mapping relationship corresponding to the data shard with the lowest access frequency is deleted first, ensuring that the cache stores the mapping information of the most active data shard at present, further improving query efficiency and response speed.

[0055] When a query instruction of a target object (such as an application or a database client) is received, the following steps may be performed: in response to the query instruction of the target object, the target data shard corresponding to the query instruction is determined; the target storage node corresponding to the target data shard is determined based on the mapping relationship stored in the target cache, and a data reading instruction corresponding to the query instruction is sent to the target storage node; and the data fed back by the target storage node is fed back to the target object.

[0056] For example, if the query instruction requires the retrieval of all order information for a certain month, the system will identify the shard ID corresponding to the order data for that month. If the shard ID of the query is P001, the system will find the node ID N004 related to P001 from the cache, indicating that the data is stored on node N004, and then send a data read instruction to the determined storage node, requesting to retrieve the data related to the query instruction. After the storage node executes the read operation, it will feed back the query results to the system, and the system will then feed back the data to the target object that originally issued the query instruction, completing the entire process of data query. This direct positioning and data reading mechanism based on mapping relationships avoids broadcast queries across the entire network and significantly improves the efficiency and response speed of data access.

[0057] In the embodiment of the present application, by collecting multi-dimensional data features such as table size, field type, field association and access status of the relational database table, the characteristics and access mode of the data can be fully understood, which provides an accurate data basis for the subsequent sharding strategy selection, ensuring that the sharding strategy takes into account both the characteristics of the data structure and the actual needs of data access, thereby improving the efficiency of data storage and query; the sharding strategy determined based on multi-dimensional data features, such as range sharding, hash sharding or list sharding, can solve the problem of data skew in a targeted manner, optimize data distribution, improve access speed, and reduce the overhead of cross-node queries; the data shards are distributed and stored in multiple storage nodes, which not only utilizes the hardware resources of multiple nodes, improves storage capacity and input / output performance, but also realizes fast data positioning and access by recording the mapping relationship between shards and nodes, thereby solving the technical problem that the existing distributed storage technology is prone to affect data storage and utilization efficiency due to unreasonable data sharding.

[0058] Example 2

[0059] According to an embodiment of the present application, a data distributed storage device for implementing the data distributed storage method in Embodiment 1 is also provided. Figure 2 As shown, the data distributed storage device at least includes: an acquisition module 21, a sharding module 22 and a storage module 23, wherein:

[0060] The acquisition module 21 is used to acquire the relational database table to be stored and determine the multi-dimensional data features of the relational database table, wherein the dimensions of the data features include at least one of the following: table size, field type, field association relationship, and access status;

[0061] A sharding module 22, used to determine a sharding strategy for a relational database table according to multi-dimensional data features, and divide the relational database table into a plurality of data shards according to the sharding strategy, wherein the sharding strategy includes at least one of the following: range sharding, hash sharding, and list sharding;

[0062] The storage module 23 is used to distribute and store multiple data slices to multiple storage nodes, and record the mapping relationship between each data slice and each storage node.

[0063] The functions of each module of the data distributed storage device are described below in conjunction with a specific implementation process.

[0064] The acquisition module acquires the relational database table to be stored, and determines multi-dimensional data features of the relational database table, wherein the dimensions of the data features include at least one of the following: table size, field type, field association relationship, and access status.

[0065] As an optional implementation, the multi-dimensional data features of the relational database table may be determined by the following method:

[0066] Determine a first data feature of a table size dimension of a relational database table by metadata analysis and partial sample data scanning, wherein the first data feature includes at least one of the following: the number of records in the table and the occupied storage space;

[0067] Analyze the field data type and distribution to determine the second data feature of the field type dimension of the relational database table, wherein the second data feature includes at least one of the following: a text field, a numeric field, a date field, and the proportion of each type of field in the relational database table;

[0068] Identify the association relationship between different fields, and determine the third data feature of the field association relationship dimension of the relational database table, wherein the third data feature includes at least one of the following: primary key constraint, foreign key constraint, and query combination between fields;

[0069] The query log is analyzed to determine a fourth data feature of an access status dimension of the relational database table, wherein the fourth data feature includes at least one of the following: access frequency, access time, and access mode.

[0070] After determining the multi-dimensional data features of the relational database table, the sharding module determines a sharding strategy for the relational database table based on the multi-dimensional data features, and divides the relational database table into multiple data shards based on the sharding strategy, wherein the sharding strategy includes at least one of the following: range sharding, hash sharding, and list sharding.

[0071] Among them, as an optional implementation method, the sharding strategy for the relational database table is determined according to multi-dimensional data features. The process may include the following steps: when the fields exceeding a first preset proportion in the relational database table are text fields, and the query fields are concentrated on the text fields, it is determined to perform hash sharding on the relational database table; when the fields exceeding a second preset proportion in the relational database table are continuous numeric fields or date fields, and the query fields are concentrated in numeric ranges or date ranges, it is determined to perform range sharding on the relational database table; when the size of the relational database table exceeds a preset threshold, it is determined to perform list sharding on the relational database table.

[0072] Among them, when determining the sharding strategy, the number of storage nodes is obtained, and the granularity of each data shard to be divided is determined based on the size of the relational database table and the number of storage nodes. In addition, the degree of association between each field in the relational database table is determined, and it is determined that multiple fields with a degree of association higher than a preset association threshold will be divided into the same data shard.

[0073] After obtaining multiple data shards, the storage module stores the multiple data shards in a distributed manner to multiple storage nodes, and records the mapping relationship between each data shard and each storage node.

[0074] Among them, as an optional implementation, the process of distributing and storing multiple data shards to multiple storage nodes can take the following steps: respectively determine the average access frequency of multiple fields in each data shard; obtain the node status information of each storage node, wherein the node status information includes at least one of the following: node capacity, node load, node read and write speed, node network bandwidth; based on the load balancing principle, determine the storage node corresponding to each data shard according to the average access frequency and the node status information, wherein the storage node corresponding to the data shard with a higher average access frequency has a greater node read and write speed and / or node network bandwidth.

[0075] Among them, as an optional implementation method, the process of recording the mapping relationship between each data shard and each storage node can take the following steps: record the mapping relationship between each data shard and each storage node in the form of a key-value pair, and store each mapping relationship in the target cache, wherein the key in the key-value pair is the shard identifier of the data shard, and the value in the key-value pair is the node identifier of the storage node; periodically determine the access frequency of each data shard, and use the minimum frequency usage algorithm to update each mapping relationship in the target cache, wherein the mapping relationship corresponding to the data shard with the lowest access frequency is deleted first.

[0076] When a query instruction of a target object (such as an application or a database client) is received, the following steps may be performed: in response to the query instruction of the target object, the target data shard corresponding to the query instruction is determined; the target storage node corresponding to the target data shard is determined based on the mapping relationship stored in the target cache, and a data reading instruction corresponding to the query instruction is sent to the target storage node; and the data fed back by the target storage node is fed back to the target object.

[0077] It should be noted that each module in the data distributed storage device in the embodiment of the present application corresponds one by one to each implementation step of the data distributed storage method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be elaborated here.

[0078] Example 3

[0079] According to an embodiment of the present application, a computer program product is also provided. The computer program product includes a computer program, wherein when the computer program is executed by a processor, the data distributed storage method in Embodiment 1 is implemented.

[0080] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the data distributed storage method in Example 1 by running the computer program.

[0081] According to an embodiment of the present application, a processor is also provided, which is used to run a computer program, wherein the data distributed storage method in Example 1 is executed when the computer program is running.

[0082] According to an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the data distributed storage method in Example 1 through the computer program.

[0083] Specifically, when the computer program is running, the following steps are executed: obtaining a relational database table to be stored, and determining multi-dimensional data features of the relational database table, wherein the dimensions of the data features include at least one of the following: table size, field type, field association relationship, and access status; determining a sharding strategy for the relational database table based on the multi-dimensional data features, and dividing the relational database table into multiple data shards based on the sharding strategy, wherein the sharding strategy includes at least one of the following: range sharding, hash sharding, and list sharding; distributively storing the multiple data shards to multiple storage nodes, and recording the mapping relationship between each data shard and each storage node.

[0084] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 3 FIG. 1 shows a hardware structure block diagram of an electronic device for implementing a data distributed storage method. Figure 3 As shown, the electronic device 30 may include one or more (302a, 302b, ..., 302n are used to illustrate) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations shown.

[0085] It should be noted that the one or more processors 302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 30. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0086] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data distributed storage method in the embodiment of the present application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, realizing the vulnerability detection method of the above-mentioned application. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the electronic device 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0087] The transmission device 306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 30. In one example, the transmission device 306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0088] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 30 .

[0089] The serial numbers of the above embodiments are only for description and do not represent the advantages or disadvantages of the embodiments.

[0090] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0093] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0095] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data distributed storage method, characterized in that: include: Acquire a relational database table to be stored, and determine multi-dimensional data features of the relational database table, wherein the dimensions of the data features include at least one of the following: table size, field type, field association relationship, and access status; Determine a sharding strategy for the relational database table according to the multi-dimensional data features, and divide the relational database table into a plurality of data shards according to the sharding strategy, wherein the sharding strategy includes at least one of the following: range sharding, hash sharding, and list sharding; The multiple data shards are distributed and stored in multiple storage nodes, and the mapping relationship between each data shard and each storage node is recorded.

2. The method according to claim 1, characterized in that Determining the multi-dimensional data features of the relational database table includes: Determine a first data feature of a table size dimension of the relational database table by metadata analysis and partial sample data scanning, wherein the first data feature includes at least one of the following: the number of records in the table and the occupied storage space; and / or, Analyze the field data type and distribution to determine the second data feature of the field type dimension of the relational database table, wherein the second data feature includes at least one of the following: a text field, a numeric field, a date field, and the proportion of each type of field in the relational database table; and / or, Identify the association relationship between different fields, and determine the third data feature of the field association relationship dimension of the relational database table, wherein the third data feature includes at least one of the following: primary key constraint, foreign key constraint, query combination between fields; and / or, Analyze the query log to determine a fourth data feature of the access status dimension of the relational database table, wherein the fourth data feature includes at least one of the following: access frequency, access time, and access mode.

3. The method according to claim 1, characterized in that Determining a sharding strategy for the relational database table according to the multi-dimensional data features includes: In a case where fields exceeding a first preset ratio in the relational database table are text fields, and query fields are concentrated on text fields, determining to perform hash sharding on the relational database table; In a case where the fields exceeding the second preset ratio in the relational database table are continuous numeric fields or date fields, and the query fields are concentrated in the numeric range or the date range, determining to perform range sharding on the relational database table; When the size of the relational database table exceeds a preset threshold, it is determined to perform list fragmentation on the relational database table.

4. The method according to claim 3, characterized in that The method further comprises: When determining the sharding strategy, the number of the storage nodes is obtained, and the granularity of each data shard to be divided is determined according to the size of the relational database table and the number of the storage nodes; Determine the degree of association between the fields in the relational database table, and determine to divide multiple fields whose association degrees are higher than a preset association threshold into the same data shard.

5. The method according to claim 1, characterized in that Distributing and storing the multiple data shards to multiple storage nodes includes: Determine the average access frequency of multiple fields in each data shard separately; Obtaining node status information of each storage node, wherein the node status information includes at least one of the following: node capacity, node load, node read / write speed, and node network bandwidth; Based on the load balancing principle, the storage node corresponding to each data shard is determined according to the average access frequency and the node status information, wherein the storage node corresponding to the data shard with a higher average access frequency has a greater node read / write speed and / or node network bandwidth.

6. The method according to claim 1, characterized in that Record the mapping relationship between each data shard and each storage node, including: Record the mapping relationship between each data shard and each storage node in the form of a key-value pair, and store each of the mapping relationships in the target cache, wherein the key in the key-value pair is the shard identifier of the data shard, and the value in the key-value pair is the node identifier of the storage node; The access frequency of each of the data slices is periodically determined, and each of the mapping relationships in the target cache is updated using a minimum frequency usage algorithm, wherein the mapping relationship corresponding to the data slice with the lowest access frequency is deleted first.

7. The method according to claim 6, characterized in that The method further comprises: In response to a query instruction of a target object, determining a target data shard corresponding to the query instruction; Determine a target storage node corresponding to the target data shard according to the mapping relationship stored in the target cache, and send a data read instruction corresponding to the query instruction to the target storage node; The data fed back by the target storage node is fed back to the target object.

8. A data distributed storage device, characterized in that: include: An acquisition module, used to acquire a relational database table to be stored, and determine multi-dimensional data features of the relational database table, wherein the dimensions of the data features include at least one of the following: table size, field type, field association relationship, and access status; A sharding module, configured to determine a sharding strategy for the relational database table according to the multi-dimensional data features, and divide the relational database table into a plurality of data shards according to the sharding strategy, wherein the sharding strategy includes at least one of the following: range sharding, hash sharding, and list sharding; The storage module is used to distribute and store the multiple data shards to multiple storage nodes, and record the mapping relationship between each data shard and each storage node.

9. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the data distributed storage method described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the data distributed storage method according to any one of claims 1 to 7 through the computer program.

Images