Data storage methods, devices, electronic equipment and storage media

Abstract

This application discloses a data storage method, apparatus, electronic device, and storage medium. The data storage method includes: generating a first field value for a first field of a first record; the first field value represents a field value determined based on a first numerical value; the first numerical value is generated based on the field value of a second field of a second record and is used to indicate a first node storing the second record; and storing the processed first record to the first node based on the first field value. The first record represents a record that belongs to a different data table and is associated with the second record.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data storage method, apparatus, electronic device, and storage medium. Background Technology

[0002] As the amount of data involved in the system increases, the system usually adopts a distributed storage approach, storing the data on multiple electronic devices and determining the distribution of the data on multiple electronic devices through data sharding.

[0003] In related technologies, in order to achieve a single read operation, concurrent requests need to be sent to all shards that conform to the routing rules, and the results returned by all requests need to be merged, which increases the number of distributed operations and results in low system read efficiency. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data storage method, apparatus, electronic device, and storage medium to at least solve the problem of low read efficiency in related technical systems.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a data storage method, the method comprising:

[0007] Generate a first field value for a first field of a first record; the first field value represents a field value determined based on a first numerical value; the first numerical value is generated based on the field value of a second field of a second record and is used to indicate the first node storing the second record;

[0008] Based on the value of the first field, the processed first record is stored in the first node; wherein,

[0009] The first record represents a record that belongs to a different data table than the second record but is related to it.

[0010] In the above scheme, generating the first field value of the first field for the first record includes:

[0011] The first value is embedded into the field value of the third field of the first record to obtain the first field value of the first field of the first record.

[0012] In the above scheme, embedding the first value into the field value of the third field of the first record includes:

[0013] The value of the third field of the first record is concatenated with the first numerical value.

[0014] In the above scheme, storing the processed first record to the first node based on the first field value includes:

[0015] Based on the first value corresponding to the first field value, determine the first slot corresponding to the first record;

[0016] According to the established mapping relationship, the first record is stored in the first node corresponding to the first slot; the mapping relationship represents the relationship between each node and the corresponding slot.

[0017] The method in the above scheme further includes:

[0018] Update the mapping relationship for at least one node;

[0019] Based on the updated mapping, records from at least one slot are migrated to the corresponding node.

[0020] In the above scheme, before generating the first field value of the first field for the first record, the method further includes:

[0021] Based on at least one field of the first data table to which the first record belongs, the second field is determined in the fields of the second data table;

[0022] The first value is generated based on the field value of the second field of the second record.

[0023] In the above scheme, generating the first value based on the field value of the second field of the second record includes:

[0024] A first hash value is generated based on the field value of the second field of the second record;

[0025] The first value is obtained by taking the remainder of the first hash value.

[0026] This application also provides a data storage device, including:

[0027] The first generation unit is used to generate a first field value for a first field of a first record; the first field value represents a field value determined based on a first numerical value; the first numerical value is generated based on the field value of a second field of a second record and is used to indicate the first node storing the second record;

[0028] A storage unit is used to store the processed first record to the first node based on the value of the first field; wherein,

[0029] The first record represents a record that belongs to a different data table than the second record but is related to it.

[0030] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor.

[0031] When the processor runs the computer program, it executes the steps of the above-described data storage method.

[0032] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described data storage method.

[0033] In this embodiment, a first field value is generated for a first record based on a first numerical value. The processed first record is then stored in a first node containing a second record based on this first field value. The first numerical value is generated based on the field value of the second field of the second record. This first numerical value indicates the first node storing the second record. The first record represents a record belonging to a different data table but related to the second record. By generating field values ​​for records, records can be stored in nodes containing related records, improving the centralization of related record storage. Based on this data storage scheme, the number of read operations on different nodes can be reduced when reading data, thereby improving the read efficiency. Attached Figure Description

[0034] Figure 1 A diagram illustrating cross-shard queries provided for related technologies;

[0035] Figure 2 A flowchart illustrating the data storage method provided in an embodiment of this application;

[0036] Figure 3 A schematic diagram illustrating the determination of a second field of a second data table based on a first data table, provided as an embodiment of this application;

[0037] Figure 4 A schematic diagram illustrating the storage hierarchy provided in the embodiments of this application;

[0038] Figure 5 A schematic diagram illustrating the storage correspondence provided in an embodiment of this application;

[0039] Figure 6 A schematic diagram illustrating the slot and node mapping relationship provided in an embodiment of this application;

[0040] Figure 7 A schematic diagram of the gene-based method for generating fragmentation key values ​​is provided for an application embodiment of this application;

[0041] Figure 8 A schematic diagram illustrating the generation of fragment key values ​​provided in the application embodiments of this application;

[0042] Figure 9 A schematic diagram of fragmented storage data provided for an application embodiment of this application;

[0043] Figure 10 A schematic diagram of the fragmentation key value provided in the application embodiments of this application;

[0044] Figure 11 A schematic diagram of fragmented storage data provided for an application embodiment of this application;

[0045] Figure 12 A schematic diagram illustrating data copying for an application embodiment of this application;

[0046] Figure 13 A schematic diagram illustrating the data mapping relationship before and after capacity expansion provided for an application embodiment of this application;

[0047] Figure 14 This is a schematic diagram of the structure of a data storage device provided in an embodiment of this application;

[0048] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0049] As the amount of data involved in a system increases, distributed storage is typically adopted, storing data across multiple electronic devices to improve system availability and reduce database response time. To store data across multiple electronic devices, data sharding is required to determine the distribution of data across these devices. This ensures even data distribution, load balancing, and minimizes data migration during scaling up and down.

[0050] Common data sharding methods include number segmentation, modulo, lookup tables, and consistent hashing. Number segmentation involves storing the same number segment range on each electronic device, suitable for integer field values, but suffers from uneven data distribution. Modulo methods calculate the hash value of the field value and then take the modulo of the number of database shards to determine the electronic device number where the data resides; this results in significant data migration when scaling up or down the system. Lookup tables store the mapping relationship between field values ​​and electronic devices, determining data distribution by searching the lookup table; however, lookup tables require substantial storage space. Consistent hashing maps key values ​​to numbers between 0 and 2. 32 The same method is used to obtain the hash value of the key name of the stored data in the ring hash space, and it is also mapped onto the ring hash space. Then, starting from the position where the data is mapped, the search proceeds clockwise and the data is saved to the first server found. When scaling up or down, the nodes need to be recalculated and a large amount of data migration is generated.

[0051] In related technologies, such as Figure 1The diagram illustrates a cross-shard query. Related data is distributed across different shards. To achieve a single operation, concurrent requests need to be sent to all shards that match the routing rules, and all results need to be merged, generating a large number of cross-shard queries. The table joins in cross-shard queries require the transmission of a significant amount of data over the network. This increases the number of read operations on different nodes during data retrieval, leading to low system read efficiency and reduced performance of the distributed storage system. Furthermore, scaling up or down the system also presents the problem of large data migration, requiring lengthy maintenance time and reducing the availability of electronic devices.

[0052] Based on this, in this embodiment, a first field value is generated for the first record based on a first numerical value. The processed first record is then stored in the first node containing the second record based on the first field value. The first numerical value is generated based on the field value of the second field of the second record. This first numerical value indicates the first node storing the second record. The first record represents a record belonging to a different data table and related to the second record. In this way, by generating field values ​​for the records, the records can be stored in the nodes containing related records, improving the centralization of related record storage. Based on this data storage scheme, the number of read operations on different nodes can be reduced when reading data, thereby improving the read efficiency.

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] Figure 2 This is a schematic diagram illustrating the implementation process of the data storage method provided in this application embodiment. The execution subject of the process is an electronic device that performs data storage, including but not limited to electronic devices such as terminals and servers.

[0055] like Figure 2 The data storage methods shown include:

[0056] Step 201: Generate the first field value for the first field of the first record.

[0057] Wherein, the first field value represents the field value determined based on the first numerical value; the first numerical value is generated based on the field value of the second field of the second record and is used to indicate the first node storing the second record.

[0058] Here, the value of the second field represents the value of a designated field in the data table to which the second record belongs. This designated field can be determined based on the characteristics of the data table in the given scenario, and is usually selected from fields that can uniquely identify the record, including but not limited to phone numbers, ID card numbers, etc.

[0059] The first value generated based on the field value of the second field can also be used to instruct the second record to be stored in the corresponding first node. In this way, the first value can instruct the first record to be stored in the first node that contains the second record.

[0060] The first field represents a defined field in the data table to which the first record belongs. Typically, the value of the first field is the identity document (ID) value.

[0061] Step 202: Based on the value of the first field, store the processed first record in the first node.

[0062] The first record represents a record that belongs to a different data table than the second record but is related to it.

[0063] Here, a first field value is generated for the first record based on the first numerical value, enabling the first record to be stored in the first node indicated by the first numerical value. Alternatively, the second record can be stored in the first node simultaneously with the first record, or the first node can be stored before the first record is stored.

[0064] The first field value is generated for the first record based on the first numerical value, and the first record can be stored in the first node indicated by the first numerical value. The second record is stored on the first node, which realizes the storage of the associated first record and second record in the same node. This improves the centralization of the storage of associated records. Based on the above data storage scheme, the number of read operations on different nodes can be reduced when reading data, thereby improving the reading efficiency of the read operation.

[0065] In one embodiment, before generating the first field value of the first field for the first record, the method further includes:

[0066] Based on at least one field of the first data table to which the first record belongs, the second field is determined in the fields of the second data table;

[0067] The first value is generated based on the field value of the second field of the second record.

[0068] Here, based on at least one field of the first data table, a second field is determined from the fields of the second data table. A first value is generated based on the field value of the second field determined from the second record. The field values ​​of at least one field of the first data table and the field value of the second field are strongly correlated under the given scenario. The first data table and the second data table represent different data tables, and under the given scenario, the first data table and the second data table are related.

[0069] The second field is typically the most valuable field in the second data table. This field value can be understood as the ability to distinguish different second records based on its value. The field value can be evaluated using at least one of the following methods: the proportion of missing values ​​for this field in each second record, the frequency of use of this field's value in the corresponding second record, and whether the field's value can be used to distinguish between different second records. Based on at least one field from the first data table, a second field is determined from the fields of the second data table. A first value is generated based on the field value of the second field in each second record. This ensures that a unique first value is obtained for each second record, establishing a correspondence between the first value and the second record, thus enabling the associated first and second records to be stored on the same node.

[0070] For example, such as Figure 3 The diagram illustrates how a second field in a second data table is determined based on a first data table. The first data table is an order table, including three fields: order ID, product name, and price. Each record in the first data table represents the information of one order. The second data table is a user table, including three fields: phone number, address, and remarks. Each record in the second data table represents the information of one user. Because users have significantly different consumption habits, different users purchase different types of goods, and even when purchasing the same goods, the prices can vary. It is evident that the field values ​​of the product name and price fields in the order table are strongly correlated with the corresponding users, while the phone number in the second data table can effectively distinguish users and achieve the effect of identifying user identity. Therefore, based on at least one field in the first data table, the second field is determined from the fields in the second table.

[0071] In one embodiment, generating the first value based on the field value of the second field of the second record includes:

[0072] A first hash value is generated based on the field value of the second field of the second record;

[0073] The first value is obtained by taking the remainder of the first hash value.

[0074] The first value can be calculated using the following formula 1:

[0075] a = crc16(key) mod n (1)

[0076] Here, 'a' represents the first value, 'key' represents the value of the second field, and 'n' can be set to the maximum number of nodes that the system can expand to.

[0077] Here, the hash value of the second field is calculated using CRC16 in the Cyclic Redundancy Check (CRC) algorithm as an example. In practical applications, hash algorithms such as CRC8 and CRC16 can also be used.

[0078] The first value is generated by a hash algorithm, which ensures that the first value can be evenly distributed within the range of possible values. In this way, when the processed first record is stored in the first node indicated by the first value, the record can be evenly stored in each node, thus improving the uniformity of the record distribution.

[0079] In one embodiment, generating a first field value for a first field of a first record includes:

[0080] The first value is embedded into the field value of the third field of the first record to obtain the first field value of the first field of the first record.

[0081] The value of the first field of the first record is obtained by embedding the first value into the field value of the third field of the first record.

[0082] Here, the third field and the first field can be the same field. In other words, the first value can be embedded in the value of the first field before processing to obtain the value of the first field after processing. The value of the first field before processing can be a long integer value generated according to a distributed ID generation algorithm, which includes, but is not limited to, the Snowflake algorithm.

[0083] Here, the third field and the first field may not be the same field. That is, the first field is used to record the field value of the third field and the first value, and the processed field value of the first field is used to store the field value of the third field and the first value.

[0084] When embedding the first numerical value into the field value of the third field, the first text corresponding to the first numerical value is embedded into the second text corresponding to the field value of the third field, and then the processed text is converted to obtain the first field value. For example, based on the first numerical value 1234, the corresponding first text is determined to be "2222", the second text corresponding to the field value of the third field is "1111222233334444555", the processed text is "11112222333344441234555", and the processed text is converted to obtain the first field value 11112222333344441234555.

[0085] In this way, based on the information of the first value contained in the field value of the first field, the first record can be stored in the first node where the second record indicated by the first value is located, so that the related records are stored on the same node, which improves the centralization of the related record storage. Based on the above data storage scheme, when reading data, the number of read operations on different nodes can be reduced, thereby improving the reading efficiency of the read operation.

[0086] In one embodiment, embedding the first numerical value in the field value of the third field of the first record includes:

[0087] The value of the third field of the first record is concatenated with the first numerical value.

[0088] Here, the value of the first field of the first record is obtained by concatenating the value of the third field of the first record with the first numerical value. During concatenation, the first text corresponding to the first numerical value can be concatenated before the second text corresponding to the value of the third field, or the first text corresponding to the first numerical value can be concatenated after the second text corresponding to the value of the third field.

[0089] In this way, based on the information of the first value contained in the field value of the first field, the first record can be stored in the first node where the second record indicated by the first value is located, so that the related records are stored on the same node, which improves the centralization of the related record storage. Based on the above data storage scheme, when reading data, the number of read operations on different nodes can be reduced, thereby improving the reading efficiency of the read operation.

[0090] In one embodiment, a first field can be defined in a first data table. All first records in the first data table have an empty value in this first field. The generated first value is then written into the first field of each first record. In other words, the first field is primarily used to store the first value. Thus, based on the value of the first field (i.e., the first numerical value), the first node can be directly determined when storing a first record.

[0091] In one embodiment, storing the processed first record to the first node based on the first field value includes:

[0092] Based on the first value corresponding to the first field value, determine the first slot corresponding to the first record;

[0093] According to the established mapping relationship, the first record is stored in the first node corresponding to the first slot; the mapping relationship represents the relationship between each node and the corresponding slot.

[0094] When the first record needs to be stored on a node, the first slot corresponding to the first record is determined based on the first value corresponding to the first field value of the processed first record. Then, according to the set mapping relationship between nodes and corresponding slots, the first record is stored in the first node corresponding to the first slot. Here, the slot is a virtual slot partition, and the number of slots is generally set much larger than the number of nodes to facilitate data splitting and system cluster expansion.

[0095] like Figure 4 The diagram illustrates the storage hierarchy. For a cluster in the system, a first value is generated based on the second field of the second record. The first field value of the first record is generated based on the generated first value. The slot corresponding to the first record is determined according to the first value corresponding to the first field value. The first record is then stored in the corresponding first node according to the set mapping relationship.

[0096] The second record is stored in the first node based on the first value. When the first record needs to be stored in a node, it is stored in the first node containing the second record based on the first value corresponding to the first field value. This allows records to be stored on nodes containing related records, improving the centralization of related record storage. Based on this data storage scheme, the number of read operations on different nodes can be reduced when reading data, thereby improving read efficiency. Simultaneously, by utilizing the correspondence between nodes and slots, unrelated records are evenly distributed across nodes, while related records are stored on the same node, minimizing data migration during expansion or contraction.

[0097] In practical applications, such as Figure 5 The diagram illustrates the storage mapping, introducing slots between records and nodes in the data table. There is a many-to-one correspondence between records and slots, and a many-to-one relationship between slots and nodes. The number of slots can be set according to the maximum number of nodes that can be expanded in the distributed system. For example, setting n=2048 slots means the maximum number of nodes that can be expanded in the distributed system is 2048. A first value for sharding is calculated based on the selected second field. This first value from the first field of the associated data table is mapped to a slot. Each first value maps to a specific slot, and associated records with the same first value are mapped to the same slot. The mapping between nodes and slots in the system can be configured and allocated as needed. The nodes corresponding to slots are configured according to the actual equipment of the nodes; nodes with high-configuration equipment can be allocated more slots. Based on the configured slot-node mapping relationship, the record corresponding to the slot is stored on the corresponding actual node.

[0098] like Figure 6The diagram illustrates the slot and node mapping relationship. The maximum number of expansion nodes is set to n = 2048. The slot 'a' to which the first record is mapped is calculated using Formula 1. The slot-to-node mapping rule is that the slots are evenly distributed across 4 nodes. The slot range for node 1 is 0 to 511, for node 2 it's 512 to 1023, for node 3 it's 1024 to 1535, and for node 4 it's 2048. Because the first field value of associated records corresponds to the same first value, the records will be mapped to the same slot and ultimately stored on the same node.

[0099] Because distributed systems involve varying amounts of data, nodes need to be added or removed. Scaling up or down the system also presents the problem of large data migration volumes, requiring lengthy maintenance times and reducing the availability of electronic devices. In one embodiment, the method further includes:

[0100] Update the mapping relationship for at least one node;

[0101] Based on the updated mapping, records from at least one slot are migrated to the corresponding node.

[0102] Here, the mapping relationship between the defined nodes and their corresponding slots is updated. Based on the updated mapping relationship, at least one record corresponding to a slot is migrated to the corresponding node. In this way, when expanding or shrinking, record migration is mapped to slot migration. During expansion or shrinking, only associated records need to be migrated, while unassociated records remain unaffected. By handling associated and unassociated records differently, the impact of expansion or shrinking on records is reduced, thereby achieving efficient record migration.

[0103] The present application will now be described in further detail with reference to application examples.

[0104] In related technologies, data is mapped to different shards based on type keys, resulting in related data being scattered across different shards. To achieve a single operation, concurrent requests need to be sent to all shards that conform to the routing rules, and all results need to be merged, generating a large number of cross-shard queries. Table joins for cross-shard queries require the transmission of large amounts of data over the network. This increases the number of read operations on different nodes during data retrieval, leading to low system read efficiency and reduced performance of the distributed storage system. Furthermore, because related data is distributed across shards, scaling up or down the system involves significant data migration, requiring lengthy maintenance times, potentially necessitating service downtime for migration, reducing the availability of electronic devices, and possibly even causing malfunctions.

[0105] Based on this, this application provides a gene-based data sharding method. It uses the common correlation attributes of related data as the basis for generating data genes, divides the data into sharding nodes according to these genes, and stores related records on the same node. This improves the centralization of related data storage, increases the probability of reading operations on a single node when reading data, reduces the number of read operations on different nodes, and improves reading efficiency. Simultaneously, based on the mapping relationship between nodes and corresponding slots, data migration is mapped to slot migration, achieving smooth expansion or contraction.

[0106] Combination Figure 7 The corresponding flowchart for generating fragmentation key values ​​using the gene method includes the following steps:

[0107] Step 701: Determine the genetic field.

[0108] Typically, the field with the highest value in the second data table is determined based on the corresponding scenario, and this field is used as the gene field for generating data genes.

[0109] Here, factors affecting the value of a field include, but are not limited to, the percentage of missing field values, the frequency of use of field values, and whether field values ​​can be used to distinguish record data.

[0110] Step 702: Calculate the data genes.

[0111] Based on the gene field, calculate the value of the data gene according to Formula 1.

[0112] Here, the data gene is obtained by taking the modulus according to the CRC16 algorithm, and the slots allocated to the data are mapped through the data gene.

[0113] Step 703: Determine the sharding key field.

[0114] Determine the relationships between the data tables and identify the sharding key field of the first data table that is associated with the second data table.

[0115] Step 704: Generate sharding key values.

[0116] The sharding key value of the first data table is generated based on the associated data genes. Usually, the sharding key value is the ID value, and the generated sharding key value can be of a set number of bits.

[0117] In practical applications, such as Figure 8 The diagram shown is a sharding key value diagram. The first 19 bits are long integer values ​​generated by the Snowflake algorithm. If the value is less than 19 bits, it is padded with 0s at the beginning. The last 4 bits are the associated data genes. If the value is less than four bits, it is padded with 0s at the beginning.

[0118] Step 705: Determine data sharding.

[0119] The data in the gene field is split into a shard, and the associated data generated based on the gene data shard key values ​​are also split into the same shard, so that related data are stored on the same shard.

[0120] Here, when performing sharding operations based on the sharding key value, it is only necessary to use the last four digits of the sharding key value to represent the data gene, and then route to the shard containing the corresponding gene field data.

[0121] In this way, the data-based sharding storage method shards data based on data relationships, storing related data in the same shard node. This improves the centralization of related data storage, reduces cross-shard query operations for related data in the scenario, thereby reducing cross-shard operation overhead, reducing the number of read operations, and improving the performance of the distributed system.

[0122] In practical applications, such as Figure 9 The diagram shown illustrates sharded data storage. The phone number field in the user table serves as the gene field, and the order ID in the order table and the family ID in the family table serve as the sharding keys. Figure 10 The diagram shown illustrates the sharding key value, which embeds a data gene generated based on the corresponding phone number field. The first 19 bits are long integer values ​​generated by the Snowflake algorithm, and the last 4 bits are the associated data gene.

[0123] In this way, based on the family table data and order table data corresponding to users on the same shard, it is possible to perform data association queries. This method of generating data genes based on association relationships can reduce cross-shard query operations in the scenario, thereby reducing cross-shard operation overhead and improving the performance of distributed storage systems.

[0124] Combination Figure 11 The corresponding system expansion method includes the following steps:

[0125] Step 1101: Assign the corresponding slot range to the newly added node, and modify the original node slot range accordingly.

[0126] Step 1102: Determine the data that needs to be migrated to the new node based on the slot range, and copy the data of the corresponding slot range in the original node to the new node.

[0127] At this point, the original node still retains and stores this part of the data.

[0128] Step 1103: Update the mapping relationship between nodes and corresponding slots, and modify the updated mapping relationship into the configuration to take effect.

[0129] At this point, new requests within the slot range of the newly added node will arrive at the newly added node, and the data in the corresponding slot range of the original node will become invalid.

[0130] Step 1104: Delete the data in the corresponding slot range of the original node.

[0131] In practical applications, such as Figure 6 The diagram illustrates the slot and node mapping relationship. The system originally had four nodes with evenly distributed data. Now, a new node 5 is added, and the data from the original nodes is evenly distributed. The slot range for node 4 is changed to 1536 to 1971, and the slot range for node 5 is 1972 to 2048. Therefore, it is only necessary to copy the data corresponding to slots 1972 to 2048 from node 4 to node 5. Figure 12 The diagram shows a data copy process. After completion, the mapping relationship between nodes and corresponding slots is updated. At this point, the slots based on data gene mapping will not change, but the slots will be mapped to the new node 5. Then, the data corresponding to slots 1972 to 2048 on node 4 can be deleted. For example, as shown... Figure 13 The diagram shown illustrates the data mapping relationship before and after the expansion. Data in slots 1984 and 2005 is copied from node 4 to node 5.

[0132] In this way, by converting the mapping relationship from data to nodes into a mapping relationship from slots, only associated records need to be migrated during scaling up or down, while unassociated records remain unaffected. By handling associated and unassociated records differently, the impact of data migration on the system during scaling up or down is reduced, thereby avoiding system blockage during scaling up or down and achieving efficient record migration.

[0133] In practical applications, with tens of millions of users, the average database response time is only 1ms, which can improve the performance of distributed storage systems.

[0134] To implement the method of the embodiments of this application, the embodiments of this application also provide a data storage device, disposed on a first electronic device, such as... Figure 14 As shown, the device includes:

[0135] The first generation unit 1401 is used to generate a first field value for a first field of a first record; the first field value represents a field value determined based on a first numerical value; the first numerical value is generated based on the field value of a second field of a second record and is used to indicate the first node storing the second record;

[0136] Storage unit 1402 is used to store the processed first record to the first node based on the first field value; wherein,

[0137] The first record represents a record that belongs to a different data table than the second record but is related to it.

[0138] In one embodiment, the first generation unit 1401 is configured to:

[0139] The first value is embedded into the field value of the third field of the first record to obtain the first field value of the first field of the first record.

[0140] In one embodiment, the first generation unit 1401 is configured to:

[0141] The value of the third field of the first record is concatenated with the first numerical value.

[0142] In one embodiment, the storage unit 1402 is used for:

[0143] Based on the first value corresponding to the first field value, determine the first slot corresponding to the first record;

[0144] According to the established mapping relationship, the first record is stored in the first node corresponding to the first slot; the mapping relationship represents the relationship between each node and the corresponding slot.

[0145] In one embodiment, the device further includes:

[0146] An update unit is used to update the set mapping relationship corresponding to at least one node;

[0147] A migration unit is used to migrate records from at least one slot to the corresponding node based on the updated mapping relationship.

[0148] In one embodiment, the device further includes:

[0149] The determining unit is configured to determine a second field in a second data table based on at least one field of a first data table to which the first record belongs;

[0150] The second generation unit is used to generate the first value based on the field value of the second field of the second record.

[0151] In one embodiment, the second generating unit is configured to:

[0152] A first hash value is generated based on the field value of the second field of the second record;

[0153] The first value is obtained by taking the remainder of the first hash value.

[0154] In practical applications, the first generation unit 1401, the storage unit 1402, the update unit, the migration unit, the determination unit, and the second generation unit can be implemented by processors in the data storage device, such as central processing units (CPUs), digital signal processors (DSPs), microcontroller units (MCUs), or field-programmable gate arrays (FPGAs).

[0155] It should be noted that the data storage device provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the data storage device and the data storage method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0156] Based on the hardware implementation of the above program modules, and in order to implement the data storage method of the embodiments of this application, the embodiments of this application also provide an electronic device, such as... Figure 15 As shown, the electronic device 1500 includes:

[0157] The communication interface 1510 enables information exchange with other devices, such as network devices.

[0158] The processor 1520 is connected to the communication interface 1510 to enable information interaction with other devices and to execute the methods provided by one or more of the above-described technical solutions when running a computer program. The computer program is stored in the memory 1530.

[0159] Specifically, the processor 1520 is used for:

[0160] Generate a first field value for a first field of a first record; the first field value represents a field value determined based on a first numerical value; the first numerical value is generated based on the field value of a second field of a second record and is used to indicate the first node storing the second record;

[0161] Based on the value of the first field, the processed first record is stored in the first node; wherein,

[0162] The first record represents a record that belongs to a different data table than the second record but is related to it.

[0163] In one embodiment, the processor 1520 is configured to:

[0164] The first value is embedded into the field value of the third field of the first record to obtain the first field value of the first field of the first record.

[0165] In one embodiment, the processor 1520 is configured to:

[0166] The value of the third field of the first record is concatenated with the first numerical value.

[0167] In one embodiment, the processor 1520 is configured to:

[0168] Based on the first value corresponding to the first field value, determine the first slot corresponding to the first record;

[0169] According to the established mapping relationship, the first record is stored in the first node corresponding to the first slot; the mapping relationship represents the relationship between each node and the corresponding slot.

[0170] In one embodiment, the processor 1520 is configured to:

[0171] Update the mapping relationship for at least one node;

[0172] Based on the updated mapping, records from at least one slot are migrated to the corresponding node.

[0173] In one embodiment, the processor 1520 is configured to:

[0174] Based on at least one field of the first data table to which the first record belongs, the second field is determined in the fields of the second data table;

[0175] The first value is generated based on the field value of the second field of the second record.

[0176] In one embodiment, the processor 1520 is configured to:

[0177] A first hash value is generated based on the field value of the second field of the second record;

[0178] The first value is obtained by taking the remainder of the first hash value.

[0179] Of course, in practical applications, the various components in electronic device 1500 are coupled together through bus system 1540. It can be understood that bus system 1540 is used to realize the connection and communication between these components. In addition to a data bus, bus system 1540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 15The general labeled all buses as Bus System 1540.

[0180] The memory 1530 in this embodiment is used to store various types of data to support the operation of the electronic device 1500. Examples of such data include any computer program used to operate on the electronic device 1500.

[0181] It is understood that memory 1530 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 1530 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0182] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1520. The processor 1520 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 1520 or by instructions in the form of software. The processor 1520 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1520 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 1530. The processor 1520 reads the program in the memory 1530 and completes the steps of the aforementioned method in conjunction with its hardware.

[0183] Optionally, when the processor 1520 executes the program, it implements the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0184] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 1530 storing a computer program, which can be executed by a processor 1520 of an electronic device to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, electronic devices, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0186] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0188] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0189] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0190] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict. Unless otherwise stated and limited, the term "connection" should be interpreted broadly. For example, it can refer to an electrical connection, or the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above term according to the specific circumstances.

[0191] Furthermore, in the examples of this application, "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the objects distinguished by "first," "second," and "third" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than those illustrated or described herein.

[0192] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0193] The specific technical features described in the various embodiments in the detailed implementation can be combined in various ways without contradiction. For example, different implementation methods can be formed by combining different specific technical features. In order to avoid unnecessary repetition, the various possible combinations of the specific technical features in this application will not be described separately.

Claims

1. A data storage method, characterized in that, The method includes: A first field value is generated for a first record; the first field value represents a field value determined based on a first numerical value; the first numerical value is generated based on the field value of a second field of a second record and is used to indicate the first node storing the second record; the first field value includes the field value of a third field and the first numerical value, the field value of the third field is generated based on a distributed identity identifier (ID) generation algorithm; generating the first field value for the first field of the first record includes: embedding the first numerical value into the field value of the third field of the first record to obtain the first field value of the first field of the first record; Based on the first value corresponding to the first field value, determine the first slot corresponding to the first record; According to the established mapping relationship, the first record is stored in the first node corresponding to the first slot; the mapping relationship represents the relationship between each node and the corresponding slot; the first record represents a record that belongs to a different data table and is associated with the second record.

2. The data storage method according to claim 1, characterized in that, Embedding the first value in the field value of the third field of the first record includes: The value of the third field of the first record is concatenated with the first numerical value.

3. The data storage method according to claim 1, characterized in that, The method further includes: Update the mapping relationship for at least one node; Based on the updated mapping, records from at least one slot are migrated to the corresponding node.

4. The data storage method according to claim 1, characterized in that, Before generating the first field value for the first field of the first record, the method further includes: Based on at least one field of the first data table to which the first record belongs, the second field is determined in the fields of the second data table; The first value is generated based on the field value of the second field of the second record.

5. The data storage method according to claim 4, characterized in that, Generating the first value based on the field value of the second field of the second record includes: A first hash value is generated based on the field value of the second field of the second record; The first value is obtained by taking the remainder of the first hash value.

6. A data storage device, characterized in that, include: The first generation unit is used to generate the first field value of the first field for the first record; The first field value represents the field value determined based on the first numerical value; The first value is generated based on the field value of the second field of the second record and is used to indicate the first node storing the second record; the first field value includes the field value of the third field and the first value, and the field value of the third field is generated based on the distributed identity number ID generation algorithm; generating the first field value of the first field of the first record includes: embedding the first value into the field value of the third field of the first record to obtain the first field value of the first field of the first record; The storage unit is used to determine the first slot corresponding to the first record based on the first value corresponding to the first field value; and to store the first record in the first node corresponding to the first slot according to the set mapping relationship; the mapping relationship represents the relationship between each node and the corresponding slot. The first record represents a record that belongs to a different data table than the second record but is related to it.

7. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the data storage method according to any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data storage method as described in any one of claims 1 to 5.

Images