A data storage method and device, electronic equipment and storage medium

Abstract

The application discloses a data storage method and device, electronic equipment and storage medium. The method comprises the following steps: in response to a storage request, a first target node in a candidate node is determined; target node information of each second target node in the candidate node is obtained through the first target node, and a target node weight of each second target node is calculated according to the target node information; wherein the second target node contains a storage function; according to the target node weight and table information of to-be-stored data tables, a target storage node in the second target node associated with each to-be-stored data table is determined, and the to-be-stored data table is stored to the target storage node. Through the technical scheme provided by the embodiment of the application, the problem that a single server cannot support the storage and calculation requirements with the increase of data volume and the increase of the number of data tables can be solved, and the storable amount of data is expanded.

Description

Technical Field

[0001] This invention relates to computer technology, and more particularly to a data storage method, apparatus, electronic device, and storage medium. Background Technology

[0002] In fields such as privacy computing, data is stored in the form of data tables, and the number of data tables increases as the amount of data increases.

[0003] In existing technologies, all data tables are usually stored directly on a single server. As the amount of data to be stored increases, a single server may not be able to support the storage and computing needs. Summary of the Invention

[0004] This invention provides a data storage method, apparatus, electronic device, and storage medium to expand the storable amount of data.

[0005] According to one aspect of the present invention, a data storage method is provided, the method comprising:

[0006] In response to a storage request, determine the first target node among the candidate nodes;

[0007] The target node information of each second target node in the candidate nodes is obtained through the first target node, and the target node weight of each second target node is calculated based on the target node information; wherein, the second target node includes storage function;

[0008] Based on the target node weight and the table information of the data table to be stored, the target storage node among the second target nodes associated with each of the data tables to be stored is determined, and the data table to be stored is stored in the target storage node.

[0009] According to another aspect of the present invention, a data storage device is provided, the device comprising:

[0010] The first node determination module is used to determine the first target node among the candidate nodes in response to a storage request;

[0011] The first weight calculation module is used to obtain target node information of each second target node in the candidate nodes through the first target node, and calculate the target node weight of each second target node based on the target node information; wherein, the second target node includes a storage function;

[0012] The first data table storage module is used to determine the target storage node among the second target nodes associated with each of the data tables to be stored, based on the target node weight and the table information of the data tables to be stored, and to store the data tables to be stored in the target storage node.

[0013] According to another aspect of the present invention, a data storage method is provided, the method comprising:

[0014] Store the data to be stored into the data table to be stored, and generate the data table identifier and the number of data tables;

[0015] Based on the number of data tables and the preset threshold for the number of directory tables, a target directory associated with each of the data tables to be stored is created on the current server, and the data tables to be stored are stored in the target directory.

[0016] According to another aspect of the present invention, a data storage device is provided, the device comprising:

[0017] The data storage module is used to store the data to be stored into the data table to be stored, and to generate the data table identifier and the number of data tables;

[0018] The second data table storage module is used to create a target directory associated with each of the data tables to be stored on the current server according to the number of data tables and a preset threshold for the number of directory tables, and to store the data tables to be stored in the target directory.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0020] At least one processor; and

[0021] A memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data storage method described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data storage method described in any embodiment of the present invention.

[0024] The technical solution provided in this embodiment obtains target node information for each second target node among the candidate nodes through a first target node, calculates the target node weight for each second target node based on the target node information, and determines the nodes among the second target nodes that can store the data tables to be stored based on the target node weights. This solves the problem that as the amount of data and the number of data tables increase, a single server cannot support the storage and computing needs, thus expanding the storable amount of data. Furthermore, by determining the target storage nodes among the second target nodes associated with each data table to be stored based on the target node weights and the table information of the data tables to be stored, the target storage nodes are determined, and the data tables to be stored are stored in the target storage nodes. By combining the characteristics of the data tables to be stored to determine the target storage nodes, a storage method based on data tables is implemented, improving the targeting of data storage.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] Figure 1 A flowchart of a data storage method provided in Embodiment 1 of the present invention;

[0027] Figure 2 A flowchart illustrating a data storage method provided in Embodiment 2 of the present invention;

[0028] Figure 3 A flowchart of a data storage method provided in Embodiment 3 of the present invention;

[0029] Figure 4 A flowchart of a data storage method provided in Embodiment 4 of the present invention;

[0030] Figure 5 This is a schematic diagram of the structure of a data storage device provided in Embodiment 5 of the present invention. Figure 1 ;

[0031] Figure 6 This is a schematic diagram of the structure of a data storage device provided in Embodiment 5 of the present invention. Figure 2 ;

[0032] Figure 7 A schematic diagram of the structure of an electronic device used to implement the data storage method in Embodiments 1 to 3 or 4 of the present invention is shown. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] Example 1

[0036] Figure 1 This is a flowchart illustrating a data storage method according to Embodiment 1 of the present invention. This embodiment is applicable to situations where data tables are stored across multiple servers. The method can be executed by the data storage device provided in this embodiment, which can be implemented in software and / or hardware. See also... Figure 1 The data storage method provided in this embodiment includes:

[0037] Step 110: In response to the storage request, determine the first target node among the candidate nodes.

[0038] The storage request is used to request the storage of data to be stored. It is issued by the storage requester, which can be a terminal such as a computer or mobile phone that can connect to the candidate node. This embodiment does not impose any restrictions on this.

[0039] Candidate nodes are one or more online servers, each server being a node. The first target node is the node that acts as the primary service node during data storage operations. When multiple candidate nodes are included, a cluster is formed, and there are connections between each node. Any candidate node can be determined as the first target node.

[0040] After the first target node is determined, the storage requester transmits the data to be stored to the first target node.

[0041] Step 120: Obtain the target node information of each second target node in the candidate nodes through the first target node, and calculate the target node weight of each second target node based on the target node information; wherein, the second target node includes storage function.

[0042] Candidate nodes can be categorized into two types: Type 1 nodes possess computational capabilities but do not perform storage functions; these are denoted as computational type nodes. If these nodes require data for computation, they can obtain it from other nodes with storage capabilities. Type 2 nodes possess both computational and storage functions; these are denoted as computational-storage type nodes. The first target node can be either a computational type node or a computational-storage type node, and the second target node is a computational-storage type node. The first target node can also be the second target node; that is, if the first target node includes storage functionality, it can participate in the calculation of the target node weights as a second target node.

[0043] If there are multiple second target nodes, the target node information of all second target nodes is obtained, that is, the target node information of all nodes with storage functions is obtained. The target node information can be obtained from the target node information management party, such as distributed coordination services like ZooKeeper, and may include the number of CPU cores, the total memory of the node, etc. This embodiment does not impose any restrictions on this.

[0044] For example, the method of managing the target node information of each node through the distributed coordination service may include: each node registers the target node information with the distributed coordination service, and the software on each node periodically reports the network status of the node to the distributed coordination service in real time through a heartbeat mechanism or other means to ensure that the node is online; if it is determined that the network communication between the node and the distributed coordination service is disconnected, the target node information registered on the distributed coordination service is cleared to ensure the validity of the target node information managed by the distributed coordination service.

[0045] The target node weights of each second target node are calculated based on the target node information. The target node weights are the node weights calculated based on the target node information. The weights of each second target node can be determined based on the target node information and a preset calculation formula. For example, the second target node with more CPU cores and a larger total node memory has a higher weight.

[0046] The target node weight W of each second target node can be determined according to the following calculation formula. i ′:

[0047] W i ′=(C i ×W ci +M i ×W mi )×W i

[0048] Among them, C i W ci M i W mi and W ci All of these are target node information, W ci The CPU weight pre-configured for the second target node i, W mi The memory weight pre-configured for the second target node i, W i The node weights pre-configured for the second target node i; C i M represents the actual number of CPU cores in the second target node i. i Let be the actual total memory of the second target node i. By combining the above target node information, the determination of weights incorporates multiple pieces of information related to the storage capacity of the second target node, thereby improving the effectiveness of target node weight determination.

[0049] Step 130: Based on the target node weight and the table information of the data table to be stored, determine the target storage node among the second target nodes associated with each data table to be stored, and store the data table to be stored in the target storage node.

[0050] The data table to be stored is a table containing data to be stored. In specific application scenarios, the data to be stored needs to be stored in the form of a table. The table information is the information of the data table to be stored, such as the table name information, etc. This embodiment does not impose any restrictions on this.

[0051] Based on the target node weights and the table information of the data tables to be stored, the target storage node among the second target nodes associated with each data table to be stored is determined. If the table name information is in numeric form, the table name and the weights of each target node can be calculated according to a preset formula to obtain the calculation result. The second target node associated with the largest calculation result is determined as the target storage node.

[0052] The data to be stored can be stored in the data table to be stored through the first target node. After the target storage node is determined, the first target node notifies the target storage node to download the corresponding data table to be stored from the first target node to store the data table. The first target node can delete the data table that has been stored by the target storage node to avoid occupying storage space.

[0053] In this embodiment, optionally, the target storage node among the second target nodes associated with each data table to be stored is determined based on the target node weight and the table information of the data table to be stored, including:

[0054] Based on the data table identifier in the table information and the node identifier of each second target node, determine the identifier calculation result;

[0055] The candidate node calculation results are determined based on the identification calculation results and the target node weights, and the target node calculation results are determined from the candidate node calculation results according to the preset node calculation result determination method.

[0056] Based on the preset association between the target node's calculation results and the node identifier, the target storage node associated with the target node's calculation results is determined.

[0057] Typically, each data table has a unique data table identifier, which can be determined based on the data stored in the data table; it is a fixed-length string of numbers. Each node corresponds to a unique node identifier, such as a pre-determined node number.

[0058] The data table identifier in the table information and the node identifier of each second target node can be ANDed to determine the identifier operation result corresponding to each second target node.

[0059] The candidate node calculation result is determined based on the identifier operation result and the target node weight. This can be achieved by multiplying the identifier operation result corresponding to the second target node and the target node weight corresponding to the second target node to obtain the node calculation result, and then using the node calculation results of all second target nodes as the candidate node calculation results.

[0060] The target node calculation result is determined from the candidate node calculation results according to the preset node calculation result determination method. This can be done by taking the maximum value in the candidate node calculation results as the target node calculation result. This embodiment does not limit this.

[0061] For example, the target node calculation result N′ can be determined by the following formula:

[0062] N′=max{(R∧N i )×W i ′}

[0063] Where R is the data table identifier, N i W is the node identifier of the second target node i. i ′ represents the target node weight.

[0064] The pre-defined association between the target node's calculation result and the node identifier can be a mapping relationship between the target node's calculation result and the range of the node's settlement result corresponding to the node identifier. For example, if the target node's calculation result is within the range A of the node's calculation result, it is mapped to node identifier one; if the target node's calculation result is within the range B of the node's calculation result, it is mapped to node identifier two, and so on. The second target node that has a pre-defined association with the target node's settlement result is determined as the target storage node.

[0065] Since the table identifiers corresponding to different tables to be stored are usually different, the results of the identifier calculation for different tables to be stored are usually different, which increases the randomness of the determination of the target node calculation result. Since the candidate node calculation result is determined by the identifier calculation result and the target node weight, the second target node with a larger target node weight is more likely to be determined as the target storage node, which improves the accuracy of the target storage node determination and makes the tables to be stored relatively evenly distributed among the second target nodes, thus improving the balance of data storage.

[0066] The technical solution provided in this embodiment obtains target node information for each second target node among the candidate nodes through a first target node, calculates the target node weight for each second target node based on the target node information, and determines the nodes among the second target nodes that can store the data tables to be stored based on the target node weights. This solves the problem that as the amount of data and the number of data tables increase, a single server cannot support the storage and computing needs, thus expanding the storable amount of data. Furthermore, by determining the target storage nodes among the second target nodes associated with each data table to be stored based on the target node weights and the table information of the data tables to be stored, the target storage nodes are determined, and the data tables to be stored are stored in the target storage nodes. By combining the characteristics of the data tables to be stored to determine the target storage nodes, a storage method based on data tables is implemented, improving the targeting of data storage.

[0067] Example 2

[0068] Figure 2 This is a flowchart illustrating a data storage method according to Embodiment 2 of the present invention. This technical solution provides supplementary explanation of the process after storing the data table to be stored to the target storage node. Compared with the above solution, this solution is specifically optimized as follows: after storing the data table to be stored to the target storage node, it further includes:

[0069] In response to a query request from the querying end, a third target node is determined from the candidate nodes; wherein, the query request includes the first key parameter generated by the querying end, and the data information of the data to be queried;

[0070] The target storage node is determined based on the data information and the target node weight, and the data table to be queried containing the data to be queried in the target storage node is transferred to the third target node.

[0071] The query data is obtained from the query data table through the third target node, and encrypted using the first key parameter to obtain encrypted query data.

[0072] The encrypted query data is sent to the query end via a third target node, allowing the query end to decrypt the encrypted query data using a second key parameter generated by the query end. Specifically, the flowchart of the data storage method is as follows: Figure 2 As shown:

[0073] Step 210: In response to the storage request, determine the first target node among the candidate nodes.

[0074] Step 220: Obtain the target node information of each second target node in the candidate nodes through the first target node, and calculate the target node weight of each second target node based on the target node information; wherein, the second target node includes storage function.

[0075] Step 230: Based on the target node weight and the table information of the data table to be stored, determine the target storage node among the second target nodes associated with each data table to be stored, and store the data table to be stored in the target storage node.

[0076] Step 240: In response to the query request from the querying end, determine the third target node among the candidate nodes; wherein, the query request includes the first key parameter generated by the querying end, and the data information of the data to be queried.

[0077] A query request is used to request data that has been stored in a node. It is issued by the querying end, which can be a computer, mobile phone, or other terminal that can connect to the candidate node. This embodiment does not impose any restrictions on this.

[0078] The third target node is the node that acts as the primary service node when performing data query operations. Any node among the candidate nodes can be determined as the third target node.

[0079] The query request includes a first key parameter generated by the querying end, such as a temporary public key. This embodiment does not impose any limitations on this. The data to be queried is the data that needs to be retrieved corresponding to the query request. The data information may include table information of the data table to which the data to be queried belongs, as well as the specific location information of the data in the table, etc.

[0080] Step 250: Determine the target storage node based on the data information and the target node weight, and transfer the data table containing the data to be queried in the target storage node to the third target node.

[0081] If the data information contains table information of the data table to which the data to be queried belongs, the target storage node can be determined according to the data information and the target node weight in step 230.

[0082] The target storage node passes the data table containing the data to be queried to the third target node. If the third target node already contains the data table to be queried, this passing step can be omitted.

[0083] If a transmission process exists, the transmission method can be that the target storage node obtains a pre-generated public key from the third target node, encrypts the data table to be queried, sends the encrypted data table to the third target node, and then decrypts it using the private key at the third target node, thereby improving the security of the data during transmission.

[0084] Step 260: Obtain the data to be queried from the data table to be queried through the third target node, and encrypt the data to be queried through the first key parameter to obtain encrypted query data.

[0085] The third target node can extract the data to be queried from the data table based on the data location information in the data information. Since the data is stored in a non-plaintext manner, the data to be queried can be extracted from the data table through preset cryptographic processing to ensure that the data to be queried before and after extraction is still non-plaintext data relative to the third target node, thereby improving the security of data query.

[0086] The extracted query data is encrypted using the first key parameter. For example, the query data can be encrypted using a temporary public key to obtain encrypted query data.

[0087] Step 270: Send the encrypted query data to the query end through the third target node, so that the query end can decrypt the encrypted query data using the second key parameter generated by the query end.

[0088] The second key parameter generated by the query terminal corresponds to the first key parameter, for example, a temporary private key; the third target node sends the encrypted query data to the query terminal so that the query terminal can decrypt the encrypted query data using the second key parameter to obtain the decrypted query data.

[0089] In this embodiment, optionally, it also includes:

[0090] If a node addition event occurs for a second target node, the target node information is updated to obtain the current node information;

[0091] The weight of each second target node is calculated based on the current node information.

[0092] The "Add Second Target Node" event indicates the addition of a second target node, signifying the addition of a node with storage functionality. Therefore, the node information of the newly added second target node can be added by the target node information manager to update the overall target node information and obtain the current node information.

[0093] The current node weight of each second target node is calculated based on the current node information. Then, the node weight of all second target nodes can be recalculated according to the node weight calculation method based on the current node information to obtain the current node weight.

[0094] After adding a second target node, the target storage node corresponding to the newly stored data table can be determined according to the current node weight; similarly, a new query request can determine the target storage node through the current node weight.

[0095] By updating the target node information promptly after adding a second target node, the new node can participate in data storage in a timely manner, avoiding resource waste and improving the accuracy of target storage node determination.

[0096] In this embodiment, optionally, it also includes:

[0097] If a node deletion event exists for a second target node, update the target node information and determine the deletion node associated with the node deletion event;

[0098] Migrate the storage data associated with the deleted node to the fourth target node among the remaining nodes of the second target node;

[0099] The migration relationship is generated based on the removed node and the fourth target node, and the migration relationship is stored in the fourth target node in the form of a linked list.

[0100] The node deletion event for the second target node indicates a reduction in the number of nodes containing storage functionality. Therefore, the node information of the reduced second target node can be deleted by the target node information manager to update the overall target node information. The deleted node associated with the node deletion event is the actual second target node that was removed.

[0101] The stored data associated with the deleted node is migrated to the fourth target node among the remaining nodes of the second target node. The remaining nodes are the nodes in the second target node that are still in operation after the deletion of the node, and the fourth target node is the node to which the stored data is actually migrated. The fourth target node can be determined by: migrating the stored data evenly and then determining the node to which it is migrated as the fourth target node; or by re-determining the target node weights according to the updated target node information and calculating the fourth target node corresponding to the migrated stored data based on the re-determined target node weights. This embodiment does not impose any restrictions on this method. This avoids data loss after node deletion and improves the security of data storage.

[0102] A migration relationship is generated based on the deleted node and the fourth target node. The migration relationship includes the original deleted node where the migrated storage data was located and the fourth target node after migration. The migration relationship is stored in the fourth target node in the form of a linked list. If the fourth target node changes and the storage data needs to be moved, the new position change relationship can be added to the linked list. This makes it easy to track the position change process of the storage data, avoid the inability to retrieve the data due to the change of position in data query, and improve the orderliness of data storage.

[0103] This invention determines the target storage node based on data information and target node weights, avoiding the need to traverse all nodes when querying data and improving data query efficiency. The data to be queried is encrypted using a first key parameter to obtain encrypted query data. The querying end then decrypts the encrypted query data using a second key parameter generated by the querying end, preventing data leakage during the transmission of the data to be queried to the querying end and thus improving data transmission security.

[0104] Example 3

[0105] Figure 3 This is a flowchart illustrating a data storage method according to Embodiment 3 of the present invention. This technical solution provides supplementary explanation of the process of storing a data table to be stored to a target storage node. Compared with the above solution, this solution is specifically optimized as follows: storing the data table to be stored to the target storage node includes:

[0106] Obtain the table identifier and number of tables corresponding to the target storage node;

[0107] Based on the number of data tables and a preset threshold for the number of directory tables, target directories associated with each data table to be stored are created on the target storage node, and the data tables to be stored are then stored in the target directories. Specifically, the flowchart for the data storage method is as follows: Figure 3 As shown:

[0108] Step 310: In response to the storage request, determine the first target node among the candidate nodes.

[0109] Step 320: Obtain the target node information of each second target node in the candidate nodes through the first target node, and calculate the target node weight of each second target node based on the target node information; wherein, the second target node includes storage function.

[0110] Step 330: Based on the target node weight and the table information of the data tables to be stored, determine the target storage node among the second target nodes associated with each data table to be stored.

[0111] Step 340: Obtain the data table identifier and the number of data tables corresponding to the target storage node.

[0112] Among them, the data tables to be stored corresponding to the target storage node are the data tables that are determined to be stored on the target storage node. The target storage node can be determined for all the data tables to be stored, and the data table identifier and the number of data tables in each target storage node can be obtained. For example, there are 10,000 data tables to be stored, of which data tables 1-1000 are assigned to target storage node one. Then the number of data tables in target storage node one is 1,000, and the data table identifier is the identifier of these 1,000 data tables.

[0113] Step 350: Based on the number of data tables and the preset threshold for the number of directory tables, create a target directory associated with each data table to be stored on the target storage node, and store the data tables to be stored in the target directory.

[0114] Create target directories on the target storage node. Each target directory can contain a corresponding data table.

[0115] The preset threshold for the number of directory tables is the maximum number of tables that a single directory can hold, for example, 10,000. This embodiment does not impose any limit on this.

[0116] The number of directories n1 can be determined based on the number of data tables and the preset threshold for the number of directory tables, where n1 = S1 / M1, and S1 is the number of data tables; M1 is the preset threshold for the number of directory tables, which can be adjusted according to the disk read / write performance of the node.

[0117] The directory identifier for each directory can be determined based on the number of directories and the data table identifiers. The directory identifier contains a portion of the data table identifiers. The target directory is then determined based on the data table identifiers and the directory identifiers. For example, the data table identifiers are 101878, 101567, 110678, 110598, and 110908. If the number of directories is 2, it indicates that 5 data tables need to be stored in two directories. To balance storage, tables 101878 and 101567 can be stored in directory one, with the directory identifier set to 101; tables 110678, 110598, and 110908 can be stored in directory two, with the directory identifier set to 110. Therefore, the target directory corresponding to table 101878 is directory 101.

[0118] It should be noted that, in order to balance storage, the number of bits in the directory identifier may be different depending on the number of directories and the number of data table identifiers. For example, if the number of directories is 1, the directory identifier can be 1. This embodiment does not impose any restrictions on this.

[0119] Once the target directory is determined, it can be created on the target storage node to store the data tables to be stored in the corresponding target directories.

[0120] The target storage node's storage database can be of LMDB type to improve the efficiency of accessing stored data. Correspondingly, the target directory can be built based on LMDB, and this embodiment does not impose any restrictions on this.

[0121] This invention improves the orderliness of storage and enhances the data storage capacity of a single node by creating target directories associated with each data table to be stored on the target storage node based on the number of data tables and a preset threshold for the number of directory tables. This avoids directly storing all data tables to be stored on the target storage node, which would reduce data access efficiency and potentially cause the database to crash if the large number of tables were queried sequentially. Furthermore, subsequent access to data tables can be performed by first accessing the directory identifier, and then further access can be performed from the accessed directory, thus improving data access efficiency.

[0122] Example 4

[0123] Figure 4 This is a flowchart illustrating a data storage method according to Embodiment 4 of the present invention. This embodiment is applicable to situations where data tables are stored on a single server. The method can be executed by the data storage device provided in this embodiment, which can be implemented in software and / or hardware. See also... Figure 4 The data storage method provided in this embodiment includes:

[0124] Step 410: Store the data to be stored in the data table to be stored, and generate the data table identifier and the number of data tables.

[0125] The data to be stored is the data before it is stored in the table. After the data to be stored is stored in the data table through a preset storage method, different data tables can generate corresponding data table identifiers according to the specific data stored in them and the preset identifier calculation method. The preset storage method can be to store data of the same type of data in the same table, etc., and this embodiment does not limit this.

[0126] After storing all the data to be stored in the table, the amount of data in the table can be obtained.

[0127] Step 420: Based on the number of data tables and the preset threshold for the number of directory tables, create target directories associated with each data table to be stored on the current server, and store the data tables to be stored in the target directories.

[0128] The current server is the server where the data is currently stored. Target directories are created on the current server, and each target directory can contain a corresponding data table.

[0129] The preset threshold for the number of directory tables is the maximum number of tables that a single directory can hold, for example, 10,000. This embodiment does not impose any limit on this.

[0130] The number of directories n2 can be determined based on the number of data tables and the preset threshold for the number of directory tables, where n2 = S2 / M2, and S2 is the number of data tables; M2 is the preset threshold for the number of directory tables, which can be adjusted according to the current disk read / write performance of the server.

[0131] The directory identifier for each directory can be determined based on the number of directories and the data table identifiers. The directory identifier contains a portion of the data table identifiers. The target directory is then determined based on the data table identifiers and the directory identifiers. For example, the data table identifiers are 101878, 101567, 110678, 110598, and 110908. If the number of directories is 2, it indicates that 5 data tables need to be stored in two directories. To balance storage, tables 101878 and 101567 can be stored in directory one, with the directory identifier set to 101; tables 110678, 110598, and 110908 can be stored in directory two, with the directory identifier set to 110. Therefore, the target directory corresponding to table 101878 is directory 101.

[0132] It should be noted that, in order to balance storage, the number of bits in the directory identifier may be different depending on the number of directories and the number of data table identifiers. For example, if the number of directories is 1, the directory identifier can be 1. This embodiment does not impose any restrictions on this.

[0133] Once the target directory is determined, it can be created on the current server to store the data tables to be stored in the corresponding target directories.

[0134] The storage database of the current server can be of type LMDB to improve the efficiency of accessing stored data. Correspondingly, the target directory can be built based on LMDB. This embodiment does not impose any restrictions on this.

[0135] Optionally, if the number of data tables exceeds a preset threshold, subdirectories can be created in the directory for further storage.

[0136] In existing technologies, general-purpose relational databases and non-relational database systems, such as MySQL and MongoDB, are typically used on servers for direct data storage. However, when there is a large volume of data or numerous tables, data access efficiency decreases, and it may even lead to database crashes. The technical solution provided in this embodiment writes data using a table-based approach and stores data in a table format to meet the needs of business scenarios. When the number of data tables is large, target directories associated with each data table to be stored are created on the current server based on the number of data tables and a preset threshold for the number of directory tables. The data tables to be stored are then stored in the target directories, improving the orderliness of storage, solving the aforementioned problems in existing technologies, and increasing the data storage capacity of a single server. Subsequent access to data tables can be performed by first accessing the directory identifier, and then further access can be performed from the accessed directory, improving data access efficiency.

[0137] Example 5

[0138] Figure 5 This is a schematic diagram of the structure of a data storage device provided in Embodiment 5 of the present invention. Figure 1 This device can be implemented in hardware and / or software, and can execute the data storage method provided in embodiments one to three of this invention, possessing the corresponding functional modules and beneficial effects of the method. For example... Figure 5 As shown, the device includes:

[0139] The first node determination module 510 is used to determine the first target node among the candidate nodes in response to a storage request;

[0140] The first weight calculation module 520 is used to obtain target node information of each second target node in the candidate nodes through the first target node, and calculate the target node weight of each second target node based on the target node information; wherein, the second target node includes a storage function;

[0141] The first data table storage module 530 is used to determine the target storage node among the second target nodes associated with each of the data tables to be stored, based on the target node weight and the table information of the data tables to be stored, and to store the data tables to be stored in the target storage node.

[0142] Based on the above technical solutions, optionally, the first data table storage module includes:

[0143] The calculation result determination unit is used to determine the identification calculation result based on the data table identifier in the table information and the node identifier of each second target node;

[0144] The target node calculation result determination unit is used to determine the candidate node calculation result based on the identifier calculation result and the target node weight, and to determine the target node calculation result from the candidate node calculation result according to a preset node calculation result determination method;

[0145] The target storage node determination unit is used to determine the target storage node associated with the target node calculation result based on a preset association relationship between the target node calculation result and the node identifier.

[0146] Based on the above technical solutions, optionally, the device further includes:

[0147] The third node determination module is used after the data table storage module to determine the third target node among the candidate nodes in response to a query request from the query end; wherein, the query request includes the first key parameter generated by the query end and the data information of the data to be queried;

[0148] The data transmission module is used to determine the target storage node based on the data information and the target node weight, and to transmit the query data table containing the query data in the target storage node to the third target node;

[0149] The data encryption module is used to obtain the data to be queried from the data table to be queried through the third target node, and to encrypt the data to be queried through the first key parameter to obtain encrypted query data.

[0150] The data decryption module is used to send the encrypted query data to the query terminal through the third target node, so that the query terminal can decrypt the encrypted query data using the second key parameter generated by the query terminal.

[0151] Based on the above technical solutions, optionally, the device further includes:

[0152] The node information update module is used to update the target node information to obtain the current node information if there is a node addition event for the second target node.

[0153] The second weight calculation module is used to calculate the current node weight of each of the second target nodes based on the current node information.

[0154] Based on the above technical solutions, optionally, the device further includes:

[0155] The node deletion determination module is used to update the target node information and determine the deletion node associated with the node deletion event if there is a node deletion event for the second target node.

[0156] The data migration module is used to migrate the stored data associated with the deleted node to the fourth target node among the remaining nodes of the second target node;

[0157] The relation storage module is used to generate a migration relationship based on the deleted node and the fourth target node, and store the migration relationship in the form of a linked list to the fourth target node.

[0158] Based on the above technical solutions, optionally, the first data table storage module includes:

[0159] The identifier quantity determination unit is used to obtain the data table identifier and the number of data tables of the data table to be stored corresponding to the target storage node;

[0160] The data table storage unit is used to create a target directory associated with each of the data tables to be stored on the target storage node according to the number of data tables and a preset threshold for the number of directory tables, and to store the data tables to be stored in the target directory.

[0161] Figure 6 This is a schematic diagram of the structure of a data storage device provided in Embodiment 5 of the present invention. Figure 2 This device can be implemented in hardware and / or software, and can execute the data storage method provided in Embodiment 4 of the present invention, possessing the corresponding functional modules and beneficial effects of the method execution. For example... Figure 6 As shown, the device includes:

[0162] Data storage module 610 is used to store data to be stored into a data table to be stored, and to generate a data table identifier and the number of data tables;

[0163] The second data table storage module 620 is used to create a target directory associated with each of the data tables to be stored on the current server according to the number of data tables and a preset directory table number threshold, and to store the data tables to be stored in the target directory.

[0164] Example 6

[0165] Figure 7A schematic diagram of an electronic device is shown that can be used to implement the data storage method in Embodiments 1 to 3 or 4 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0166] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0167] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0168] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the data storage methods in Embodiments 1 to 3 or the data storage method in Embodiment 4.

[0169] In some embodiments, the data storage method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data storage method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the data storage method by any other suitable means (e.g., by means of firmware).

[0170] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0171] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0172] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0174] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0175] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0176] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0177] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data storage method, characterized in that, include: In response to a storage request, determine the first target node among the candidate nodes; The target node information of each second target node in the candidate nodes is obtained through the first target node, and the target node weight of each second target node is calculated based on the target node information; wherein, the second target node includes storage function; Based on the target node weight and the table information of the data table to be stored, determine the target storage node among the second target nodes associated with each of the data tables to be stored, and store the data table to be stored in the target storage node; Specifically, determining the target storage node among the second target nodes associated with each of the data tables to be stored, based on the target node weights and table information of the data tables to be stored, includes: The identifier operation result is determined based on the data table identifier in the table information and the node identifier of each second target node; wherein, the data table identifier and the node identifier are ANDed. The candidate node calculation result is determined based on the identifier calculation result and the target node weight, and the target node calculation result is determined from the candidate node calculation result according to the preset node calculation result determination method; wherein, the candidate node calculation refers to the node calculation result of all the second target nodes; Based on the preset association relationship between the target node calculation result and the node identifier, the target storage node associated with the target node calculation result is determined; wherein, the preset association relationship is a mapping relationship between the target node calculation result and the range of node calculation results corresponding to the node identifier; Storing the data table to be stored to the target storage node includes: Obtain the data table identifier and number of data tables corresponding to the target storage node; Based on the number of data tables and the preset threshold for the number of directory tables, a target directory associated with each of the data tables to be stored is created on the target storage node, and the data tables to be stored are stored in the target directory.

2. The method according to claim 1, characterized in that, After storing the data table to be stored to the target storage node, the process further includes: In response to a query request from the querying end, a third target node is determined from the candidate nodes; wherein, the query request includes a first key parameter generated by the querying end, and data information of the data to be queried; The target storage node is determined based on the data information and the target node weight, and the query data table containing the query data in the target storage node is transferred to the third target node; The query data is obtained from the query data table through the third target node, and the query data is encrypted using the first key parameter to obtain encrypted query data. The encrypted query data is sent to the query terminal through the third target node, so that the query terminal can decrypt the encrypted query data using the second key parameter generated by the query terminal.

3. The method according to claim 1, characterized in that, Also includes: If a node addition event occurs for the second target node, the target node information is updated to obtain the current node information; The current node weight of each second target node is calculated based on the current node information.

4. The method according to claim 1, characterized in that, Also includes: If a node deletion event exists for the second target node, then update the target node information and determine the deletion node associated with the node deletion event; The stored data associated with the deleted node is migrated to the fourth target node among the remaining nodes of the second target node; A migration relationship is generated based on the deleted node and the fourth target node, and the migration relationship is stored in the fourth target node in the form of a linked list.

5. A data storage device, characterized in that, include: The first node determination module is used to determine the first target node among the candidate nodes in response to a storage request; The first weight calculation module is used to obtain target node information of each second target node in the candidate nodes through the first target node, and calculate the target node weight of each second target node based on the target node information; wherein, the second target node includes a storage function; The first data table storage module is used to determine the target storage node among the second target nodes associated with each of the data tables to be stored, based on the target node weight and the table information of the data tables to be stored, and to store the data tables to be stored in the target storage node. The first data table storage module includes: The calculation result determination unit is used to determine the identification calculation result based on the data table identifier in the table information and the node identifier of each second target node; wherein the data table identifier and the node identifier are ANDed. The target node calculation result determination unit is used to determine the candidate node calculation result based on the identifier calculation result and the target node weight, and to determine the target node calculation result from the candidate node calculation result according to a preset node calculation result determination method; wherein, the candidate node calculation refers to the node calculation result of all the second target nodes; A target storage node determination unit is configured to determine the target storage node associated with the target node calculation result based on a preset association relationship between the target node calculation result and the node identifier; wherein, the preset association relationship is a mapping relationship between the target node calculation result and the range of node calculation results corresponding to the node identifier; The first data table storage module includes: The identifier quantity determination unit is used to obtain the data table identifier and the number of data tables of the data table to be stored corresponding to the target storage node; The data table storage unit is used to create a target directory associated with each of the data tables to be stored on the target storage node according to the number of data tables and a preset threshold for the number of directory tables, and to store the data tables to be stored in the target directory.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data storage method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the data storage method according to any one of claims 1-4.

Images