A data storage method, distribution device, and storage medium for data entry.

Abstract

This application discloses a data allocation method, allocation device, and storage medium for data storage technology. The method includes: acquiring multiple data entries from a database; sorting the multiple data entries according to their entry time to obtain a sorting sequence number; obtaining a marker information for each data entry to be stored in a preset data table corresponding to different total number of tables based on the sorting sequence number; determining the total number of tables in the database after expansion when the database tables are expanded; matching the expanded total number of tables with the marker information of each data entry to determine the target data table for each data entry; and storing each data entry in the corresponding target data table. This method allows for the allocation of data entries without limiting the number of expanded data tables, reducing the amount of data operations required for data deletion.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data storage allocation method, allocation device, and storage medium. Background Technology

[0002] In the current context of big data, massive amounts of data need to be stored in databases. Databases are composed of different data tables. Storing a large amount of data in one data table can cause operational lag. Sharding-JDBC (a database sharding framework) can be used to perform horizontal partitioning of the database. The core of horizontal partitioning is to divide a large data table into multiple smaller data tables. The smaller data tables have the same structure as the large data tables, except that different data is stored in different data tables in a distributed manner.

[0003] When the number of data tables obtained from the initially defined horizontal partitioning cannot meet the ever-increasing data changes, dynamic expansion of the data tables is required through a registry center and orchestration. Configuration files are updated in real time to achieve the system's intelligent expansion capability. For example, the `t_score` master table becomes the base data tables `t_score_0` and `t_score_1` after initial horizontal partitioning. As the data volume continues to increase, `t_score_0` and `t_score_1` can no longer handle the large amount of data, requiring table expansion to ensure that subsequent data can be continuously stored in other expanded tables, guaranteeing normal system operation. For instance, if data is entered into the database with primary keys of 1, 2, 3, 4, 5, and 6 (auto-incrementing), using preset modulo rules such as 1%2=1, 2%2=0, 3%2=1, etc., 1, 3, and 5 will be stored in table `t_score_1`, while 2, 4, and 6 will be stored in table `t_score_0`. When the data volume in tables t_score_0 and t_score_1 became very large, leading to performance bottlenecks, horizontal dynamic scaling was performed, splitting the tables into t_score_0, t_score_1, and t_score_2. New data with primary keys 7, 8, 9, 10, 11, and 12 was initially added to the database. Following the modulo rule (7%3 = 1, 8%3 = 2, 9%3 = 0...), this data was distributed across the t_score_0, t_score_1, and t_score_2 tables. However, horizontal partitioning and horizontal dynamic scaling only solved the data distribution problem for data not yet added to the database; the data initially in the base tables could not be evenly distributed. After horizontal dynamic scaling, data would still be stored in the original base tables according to the modulo rule, putting pressure on the existing base tables.

[0004] Currently, the general method for data balancing after horizontal table expansion is to replicate the data in the original base table in the new expanded table, with both primary and backup tables appearing in multiples. Then, data is cross-deleted between the new expanded table and the base table to achieve data redistribution. For example... Figure 1 As shown, t_score_0 and t_score_1 are the base data tables, and t_score_2 and t_score_3 are the newly expanded tables. The cross-deletion method is as follows: for table t_score_0, remove data where mod 4! = 0 (i.e., in table t_score_0, only data whose primary key leaves a remainder of 0 when divided by 4); for table t_score_1, remove data where mod 4! = 1; for table t_score_2, remove data where mod 4! = 2; and for table t_score_3, remove data where mod 4! = 3. It is evident that this processing method requires table replication, and data deletion after replication requires modulo operation on the primary key, as well as cross-deletion of data in the newly expanded tables and the base data tables. The newly expanded tables need to be expanded in pairs, limiting the number of data tables to an even number, and requiring a significant amount of data manipulation for deletion. Summary of the Invention

[0005] This application provides a data allocation method, allocation device, and storage medium for data entry, which can allocate data without limiting the number of data tables, thereby reducing the amount of data operations required for data deletion.

[0006] This application provides a data storage allocation method, including:

[0007] Retrieve multiple data entries from the database;

[0008] Sort the multiple data entries according to their entry time to obtain a sorting number for the multiple data entries;

[0009] Based on the sorting sequence number of the multiple data entries, the marking information of the preset data table to which each data entry is stored is obtained when the total number of tables is different. The total number of tables is the total number of data tables in the database used to store data.

[0010] When the data tables in the database are expanded, determine the total number of tables in the database after the expansion;

[0011] Match the expanded total number of tables with the tag information of each of the inbound data to determine the target data table where each of the inbound data is stored.

[0012] Each piece of data to be entered into the database is stored in the corresponding target data table.

[0013] Furthermore, the sorting sequence number includes: primary key;

[0014] The step of sorting the multiple inbound data according to their inbound time to obtain a sorting sequence number for the multiple inbound data includes:

[0015] Sort the multiple inbound data according to their inbound time, and determine the key value of the primary key of the multiple inbound data.

[0016] Furthermore, the step of obtaining the tagging information of the preset data table to which each piece of data is stored when there are different total number of tables, based on the sorting sequence number of the multiple pieces of data, includes:

[0017] Take the remainder of the primary key value of each of the imported data with respect to different total number of tables, and use the remainder as the table number of the preset data table to which each of the imported data is stored when there are different total number of tables.

[0018] Each piece of data entering the database is bound to a preset field in the database. The table number of the preset data table to which the data is stored is marked in the bound preset field when the total number of tables is different, thus obtaining the marking information corresponding to multiple preset fields.

[0019] Furthermore, when the data tables in the database are expanded multiple times, the data to be added includes: the target data in the data tables before and after the expansion;

[0020] The step of obtaining the tagging information of the preset data table to which each piece of data is stored when there are different total number of tables, based on the sorting sequence number of the multiple pieces of data, includes:

[0021] Based on the key value of the primary key of multiple target data, the tag information of the preset data table to be stored for each target data when the total number of tables is different is obtained.

[0022] Furthermore, the step of matching the expanded total number of tables with the tag information of each piece of data entering the database to determine the target data table to which each piece of data entering the database is stored includes:

[0023] Match the expanded total number of tables with the tag information of each target data to determine the target data table where each target data is stored.

[0024] The step of storing each piece of data into the corresponding target data table includes:

[0025] Based on the target data table into which each target data is stored, determine whether there is any mismatched target data stored.

[0026] If so, the mismatched target data will be migrated to the corresponding target data table.

[0027] Furthermore, when the data tables in the database are expanded only once, the data to be added to the database includes: the data in the first data table before the expansion;

[0028] The step of obtaining the tagging information of the preset data table to which each piece of data is stored when there are different total number of tables, based on the sorting sequence number of the multiple pieces of data, includes:

[0029] Based on the key values ​​of the primary keys of data in multiple first data tables, the tagging information of the preset data table to which each piece of data in the first data table is stored is obtained when the total number of tables is different.

[0030] Furthermore, the step of matching the expanded total number of tables with the tag information of each piece of data entering the database to determine the target data table to which each piece of data entering the database is stored includes:

[0031] The total number of tables after expansion is matched with the tagging information of the data in each of the first data tables to determine the target data table where the data in each of the first data tables is stored.

[0032] The step of storing each piece of data into the corresponding target data table includes:

[0033] When the target data table into which the data in the first data table is stored is the expanded data table, the data in the first data table is migrated to the corresponding expanded data table, wherein the expanded data table is the data table added after the expansion of the data table before expansion.

[0034] This application provides a data storage and allocation device, including:

[0035] The retrieval unit is used to retrieve multiple data entries from the database.

[0036] A sorting unit is used to sort multiple data entries according to their entry time to obtain a sorting sequence number for the multiple data entries.

[0037] An execution unit is used to obtain the tag information of a preset data table to be stored for each piece of data when the total number of tables is different, based on the sorting sequence number of the multiple pieces of data. The total number of tables is the total number of data tables in the database used to store data.

[0038] The first determining unit is used to determine the total number of tables in the database after the expansion of the database tables.

[0039] The second determining unit is used to match the expanded total number of tables with the tag information of each of the inbound data to determine the target data table where each of the inbound data is stored.

[0040] The storage unit is used to store each type of data into the corresponding target data table.

[0041] This application embodiment also provides a data storage and allocation device, including:

[0042] Central processing unit, memory, and input / output interfaces;

[0043] The memory is either a short-term storage memory or a persistent storage memory;

[0044] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the methods described above.

[0045] This application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the above-described method.

[0046] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0047] The method in this application includes: acquiring multiple inbound data from a database; sorting the multiple inbound data according to their inbound time to obtain a sorting sequence number; obtaining a marker information for each inbound data corresponding to a preset data table at different total number of tables based on the sorting sequence number; determining the total number of tables in the database after expansion when the database tables are expanded; matching the expanded total number of tables with the marker information of each inbound data to determine the target data table to which each inbound data is stored; and storing each inbound data in the corresponding target data table. In this application embodiment, by storing the data in the expanded data table into the corresponding target data table based on the marker information of the inbound data, the number of data tables is not limited when allocating inbound data, and it is not necessary to copy the data of the original data table multiple times, thus reducing the amount of data operations for deleting data. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0049] Figure 1 This is a schematic diagram illustrating the expansion and allocation of a data table disclosed in an embodiment of this application;

[0050] Figure 2 This is a flowchart illustrating the data entry and allocation process disclosed in an embodiment of this application.

[0051] Figure 3This is another data storage allocation flowchart disclosed in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram illustrating the expansion allocation of another data table disclosed in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram illustrating the transfer of data disclosed in an embodiment of this application;

[0054] Figure 6 This is a diagram of a data storage and allocation device disclosed in an embodiment of this application;

[0055] Figure 7 This is a diagram of another data storage and allocation device disclosed in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] In the description of the embodiments of this application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0058] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0059] Currently, when the capacity of the original base data table in a database reaches its limit, it is necessary to horizontally partition and expand the base data table. The expected data distribution after expansion is that the original data in the base data table and the newly added data are evenly distributed across all expanded data tables (base data table and expansion table) according to a modulo rule. Existing methods for balancing data after horizontal partitioning expansion typically involve table copying and cross-deletion to adjust table data. This involves using table replication to evenly distribute all data across the base data table, and then replicating the base data table to create a new expansion table. This method results in pairs of data tables, with each base data table corresponding to a new expansion table, and requires extensive deletion operations on both the base data table and the new expansion table. This not only limits the number of data tables to an even number but also requires a significant amount of data deletion operations. Therefore, this application provides a data allocation method that does not limit the number of data tables when allocating data, reducing the amount of data deletion operations. Figure 2 The specific steps are as follows:

[0060] 201. Retrieve multiple data entries from the database.

[0061] The data distribution device can acquire multiple data entries from a database. This data is input into the database, which contains multiple tables for storing the data. Specifically, the distribution device can connect to the database. When data is input into the database, the distribution device can record information corresponding to that data, including the input time, and can distribute the data stored in the tables.

[0062] 202. Obtain the sorting sequence number of multiple inbound data based on their inbound time.

[0063] The allocation device can obtain a sorting sequence number for multiple incoming data entries based on their entry time. Generally, the data that enters the database first (data entered into the database earlier in terms of time) is stored in the database table first. The data is then sorted according to its entry time to obtain a sorting sequence number. This sorting sequence number is generally ordered from smallest to largest based on the entry time; that is, the sorting sequence number of earlier-entered data is generally smaller than the sorting sequence number of later-entered data. Furthermore, to facilitate data allocation, this sorting sequence number is a consecutive positive integer, i.e., 1, 2, 3, 4... based on the entry time. It can be understood that this sorting sequence number generally refers to a sorting identifier for the incoming data and does not typically include the actual data content.

[0064] 203. Based on the sorting sequence number of multiple inbound data, obtain the tag information of the preset data table to which each inbound data is stored when the total number of tables is different.

[0065] The allocation device can obtain the marking information of the preset data table to be stored for each piece of data when the number of total tables is different, based on the sorting sequence number of multiple inbound data. The total number of tables is the total number of all data tables in the database used to store data, and the total table is all the data tables in the database used to store data. The preset data table is a data table in the total table. That is, the preset data table to be stored for inbound data when the number of total tables is different refers to a specific data table in the total table that the inbound data should be stored in when the number of total tables is different. Only then is the allocation of the inbound data more reasonable. This marker information can be understood as the modulo information obtained according to the modulo rule, corresponding to the preset data table where the incoming data is stored when the total number of tables is different. This modulo rule can be a remainder. For example, when the sorting number of the incoming data is 3 and the total number of tables is 1, the rule is 3mod1 = 0, indicating that the incoming data with sorting number 3 is stored in table 0; when the total number of tables is 2, 3mod2 = 1, indicating that the incoming data with sorting number 3 is stored in table 1; when the total number of tables is 3, 3mod3 = 0, indicating that the incoming data with sorting number 3 is stored in table 0; when the total number of tables is 4, 5, or 6, 3mod4 = 3, 3mod5 = 3, 3mod6 = 3, indicating that the incoming data with sorting number 3 is stored in table 3. It can be understood that when arranging data tables in the database, the first data table is generally designated as table 0, the second data table as table 1, and so on. The allocation device can obtain the tagging information of the preset data tables to which the incoming data is stored when the total number of tables is different. For example, when the sorting number of the incoming data is 3, the corresponding tagging information is {0@1@0@3@3.....}. In this tagging information, the symbol @ is a separator, with one tag on each side. The number of digits in the tag corresponds one-to-one with the total number of tables, and the specific value of each tag digit represents the data table to which the incoming data is stored. For example, 0 in the first digit represents that when the total number of tables is 1, the incoming data is stored in table 0. In this embodiment, 100 or 200 data tables can be pre-tagged for one incoming data, and the specific number is not limited here.

[0066] It is understandable that steps 201, 202, and 203 can be executed when the data is entered into the database. That is, when the data is entered into the database, the data can be marked according to the sorting sequence number of the data.

[0067] 204. Determine the total number of tables in the database after expansion.

[0068] When the capacity of a database table reaches its limit and can no longer store newly added data, the database table needs to be expanded. After expansion, the existing data in the original tables continues to be stored there. New data is allocated to both the original tables and the newly added tables according to a set of rules, leading to an imbalance in data distribution. To ensure balanced distribution of new data after expansion, the total number of tables in the database after expansion needs to be determined. This total number includes the original tables plus the number of newly added tables. Each table in the database has an identifier, and the allocation device can determine the total number of tables after expansion based on the number of identifiers.

[0069] 205. Match the total number of tables after expansion with the tag information of each incoming data to determine the target data table to which each incoming data is stored.

[0070] The allocation device can match the expanded total number of tables with the tag information of each incoming data to determine the target data table where each incoming data should be stored. This target data table refers to a specific data table within the expanded total table, which includes the data tables from before the expansion and the newly added data tables. In other words, matching the expanded total number of tables with the tag information of each incoming data means determining the number of tags corresponding to the tag information of the incoming data based on the expanded total number of tables. The specific value recorded on the tag is the target data table where the incoming data should be stored. For example, if the sorting sequence number of the incoming data is 3, and the tag information is {0@1@0@3@3.....}, then when the expanded total number of tables is 3, the incoming data with sorting sequence number 3 should be stored in data table 0.

[0071] 206. Store each piece of data into the corresponding target data table.

[0072] Once the target data table for each incoming data item is determined, the allocation device can store each incoming data item into the corresponding target data table. This storage can be understood as redistributing all incoming data in the existing data table to the corresponding target data table, or as migrating incoming data that does not conform to the expected distribution to the corresponding target data table; the specific method is not limited here. This redistribution can store all incoming data into the corresponding target data table based on the target data table for each incoming data item. The expected distribution refers to the data table where the incoming data is stored after the data table is expanded, which is the corresponding target data table. The device checks one by one whether the incoming data in each data table of the expanded master table is already in the target data table. If so, no migration is performed; otherwise, the incoming data is migrated to the corresponding target data table. It is understood that, in this embodiment, after determining the target data table for the incoming data, the incoming data can be stored in batches into the corresponding target data table.

[0073] In this embodiment, multiple inbound data are obtained from the database; the multiple inbound data are sorted according to their inbound time to obtain a sorting sequence number; based on the sorting sequence number, the tag information of the preset data table to which each inbound data is stored at different total number of tables is obtained; when the data tables in the database are expanded, the total number of tables in the database after expansion is determined; the total number of tables after expansion is matched with the tag information of each inbound data to determine the target data table to which each inbound data is stored; each inbound data is stored in the corresponding target data table. In this embodiment, by storing the data in the expanded data tables into the corresponding target data tables based on the tag information of the inbound data, the number of data tables is not limited when allocating inbound data, and it is not necessary to copy the data of the original data tables (basic data tables) many times, thus reducing the amount of data operations for deleting data. In one feasible approach, existing methods often involve replicating data from existing tables after expansion. This requires script-based deletion of data across all tables, prioritizing paired data entries. Asymmetric methods would be cumbersome for this data replication. This embodiment, however, uses data tagging for batch data transfer from existing tables (base tables), allowing for flexible expansion without strict pairing requirements. The even distribution of data from the original tables to the newly added tables after expansion significantly reduces the workload.

[0074] Furthermore, to more clearly describe the data storage allocation method in the embodiments of this application, the following will be combined with... Figure 3 A detailed description is provided below, with the specific steps as follows:

[0075] 301. Retrieve multiple data entries from the database.

[0076] It is understandable that step 301 is similar to the steps described above, and will not be repeated here.

[0077] 302. Sort multiple inbound data according to their inbound time and determine the primary key value of the multiple inbound data.

[0078] When sorting multiple inbound data based on their inbound time, the sorting sequence includes the primary key. The allocation device can sort the multiple inbound data based on their inbound time to determine the key value of the primary key for each inbound data. Specifically, the earlier the inbound time, the smaller the key value of the primary key, similar to step 202 above, and will not be elaborated further here.

[0079] 303. Based on the key values ​​of the primary keys of multiple target data, obtain the tag information of the preset data table to be stored for each target data when the total number of tables is different.

[0080] Understandably, when a database table needs to be expanded multiple times, all incoming data needs to be marked. Incoming data is generally stored in a data table. In this case, the incoming data includes: the target data in the data table before and after the expansion; the data table before and after the expansion includes the original data table (base table) before the expansion and the newly added data table after the expansion.

[0081] The allocation device can obtain the tagging information of the preset data table to be stored for each target data at different total number of tables based on the key values ​​of the primary keys of multiple target data. Specifically, the modulo of the key value of the primary key of each target data with different total number of tables can be used as the table number of the preset data table to be stored for each data entry at different total number of tables; each data entry can be bound to a preset field in the database, which can be a JSON field; the table number of the preset data table to be stored for the corresponding data entry at different total number of tables can be marked in the bound preset field to obtain the tagging information corresponding to multiple preset fields. For example, if the key value of the primary key of a data entry is 3, the JSON field tag {0@1@0@3@3@3@3@3......} can be obtained; each JSON field is bound to different target data, and when the tagging information is needed later, it can be obtained from the JSON field corresponding to the target data.

[0082] 304. Match the total number of tables after expansion with the tag information of each target data to determine the target data table where each target data is stored.

[0083] The allocation device can match the tagging information of each target data with the total number of tables after expansion to determine the target data table where each target data is stored. Specifically, it can obtain the tagging information from the JSON fields corresponding to the target data, and determine the tagging bit based on the total number of tables after expansion, which is the number of tagging bits in the tagging information. The specific value recorded on this tagging bit is the table number of the target data table. For example, for data with a primary key value of 3, when the total number of tables after expansion is 3, the third tagging bit will be 0, meaning the target data table number for data with a primary key value of 3 is 0. This allows us to determine the target data table corresponding to the data entered in all tables after expansion.

[0084] 305. Migrate the mismatched target data to the corresponding target data table.

[0085] The allocation device can migrate mismatched target data to the corresponding target data table. Specifically, the allocation device can determine whether mismatched target data exists based on the target data table where each target data is stored. If, after expansion, the target data is not located in the target data table, it is determined that the target data is mismatched; if the target data is located in the target data table, it is determined that the target data is matched. When the target data is determined to be mismatched, it is migrated to the corresponding target data table.

[0086] 306. Based on the key values ​​of the primary keys of the data in multiple first data tables, obtain the tag information of the preset data table to which the data in each first data table is stored when the total number of tables is different.

[0087] It's understandable that database tables may expand multiple times, or perhaps only once. In this case, simply migrating the data from the original table (base table) before the expansion will balance the data distribution. Specifically, when the database tables expand only once, the data being imported includes: data from the first table before the expansion; and the data from the original table before the expansion. For example... Figure 4 As shown, the original database tables were t_score_0 and t_score_1. The primary keys of the newly added data (IDs) were 1, 2, 3, and 4, with the following entry status: 1 mod 2 = table 1; 2 mod 2 = table 0; 3 mod 2 = table 1; 4 mod 2 = table 0. After expansion to four tables: t_score_0, t_score_1, t_score_2, and t_score_3, the primary keys of the newly added data (IDs) were 5, 6, 7, and 8, with the following entry status: 5 mod 4 = table 1; 6 mod 4 = table 2; 7 mod 4 = table 3; 8 mod 4 = table 0.

[0088] The allocation device can obtain the tag information of the preset data table to be stored when the data in each first data table is stored in different total number of tables, based on the key value of the primary key of the data in multiple first data tables. The specific details are similar to step 303 above, and will not be repeated here.

[0089] It should be noted that the order in which steps 303 and 304 are executed is not limited.

[0090] 307. Match the total number of tables after expansion with the tag information of the data in each first data table to determine the target data table where the data in each first data table is stored.

[0091] The allocation device can match the expanded total number of tables with the tag information of the data in each first data table to determine the target data table where the data in each first data table is stored. The specific steps are similar to those in step 304 above, and will not be repeated here.

[0092] 308. When the target data table is an expanded data table, migrate the data in the first data table to the corresponding expanded data table.

[0093] When the target data table into which the data in the first data table is stored is the expanded data table, the allocation device can migrate the data in the first data table to the corresponding expanded data table, wherein the expanded data table is the data table before expansion and the data table added after expansion. Figure 5 As shown, the expanded data tables are t_score_0, t_score_1, t_score_2, and t_score_3. t_score_0 and t_score_1 are the first data tables, and t_score_2 and t_score_3 are the expanded data tables, for a total of 4 tables. Therefore, in the tagging information, the target data table corresponding to the data with primary key 1 in the first data table (the original data table) is 1 mod 4 = table 1; the target data table corresponding to the data with primary key 2 is 2 mod 4 = table 2; the primary key is... The target data table corresponding to the data with primary key 3 is table 3mod4=3; the target data table corresponding to the data with primary key 4 is table 4mod4=0. Therefore, the target data tables corresponding to the data with primary keys 2 and 3 are the expanded data tables t_score_2 and t_score_3, respectively. Based on the batch migration method, the data with primary key 2 stored in t_score_0 is migrated to table t_score_2, and the data with primary key 3 stored in t_score_1 is migrated to table t_score_3. The data distribution after migration is relatively balanced. Furthermore, the migration can be performed in batches, that is, simultaneously migrating the data with primary key 2 stored in t_score_0 to table t_score_2, and migrating the data with primary key 3 stored in t_score_1 to table t_score_3.

[0094] In this embodiment, after horizontal scaling of the table using the Sharding-JDBC middleware, all data already in the table and data about to be in the table are marked with a JSON format. Finally, the data already in the original data table (base table) is marked and judged, and those meeting the conditions are batch-transferred to the newly added data table after scaling, achieving data balance between the original data table and the newly added data table after scaling. By using data marking, a modulo flag is preset for each inbound data, fully utilizing the data allocation mechanism of Sharding-JDBC, the inbound data before scaling is transferred to the newly added data table after scaling according to the flag, achieving data balance. This method does not require paired scaling of the base table, can expand by any number of tables, and significantly reduces the amount of data operations by batch-transferring data from the original data table.

[0095] This application also provides a data storage and distribution device, such as... Figure 6 As shown, it includes:

[0096] The acquisition unit 601 is used to acquire multiple data entries from the database.

[0097] The sorting unit 602 is used to sort the multiple data entries according to their entry time to obtain a sorting sequence number for the multiple data entries.

[0098] The execution unit 603 is used to obtain the tag information of the preset data table to be stored for each piece of data when the total number of tables is different, based on the sorting sequence number of the multiple pieces of data. The total number of tables is the total number of data tables in the database used to store data.

[0099] The first determining unit 604 is used to determine the total number of tables in the database after the expansion of the data tables in the database.

[0100] The second determining unit 605 is used to match the expanded total number of tables with the tag information of each of the inbound data to determine the target data table where each of the inbound data is stored.

[0101] The storage unit 606 is used to store each of the data to be stored into the corresponding target data table.

[0102] This application embodiment also provides a data storage and distribution device 700, such as... Figure 7 As shown, it includes:

[0103] Central processing unit 701, memory 702, and input / output interface 703;

[0104] The memory 702 is a short-term storage memory or a persistent storage memory;

[0105] The central processing unit 701 is configured to communicate with the memory 702 and execute instructions in the memory 702 to perform the allocation method described above.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0108] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0110] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method of allocating data warehousing, characterized by, The application comprises the following steps: acquiring a plurality of database data; sorting the plurality of database data according to the storage time of the plurality of database data to obtain the sorting sequence number of the plurality of database data; obtaining the marking information of the preset data table corresponding to the storage of each database data at different total table numbers according to the sorting sequence number of the plurality of database data, wherein the total table number is the total number of data tables in the database for storing data; determining the total table number after the expansion of the data table in the database; matching the total table number after the expansion with the marking information of each database data to determine the target data table for the storage of each database data; storing each database data in the corresponding target data table; the sorting sequence number comprises a primary key; the marking information of the preset data table corresponding to the storage of each database data at different total table numbers is obtained according to the sorting sequence number of the plurality of database data, which comprises the following steps: taking the remainder of the key value of the primary key of each database data with respect to the different total table numbers, and taking the remainder as the table number of the preset data table corresponding to the storage of each database data at different total table numbers; binding each database data with a preset field in the database, and marking the table number of the preset data table corresponding to the storage of the corresponding database data at different total table numbers in the bound preset field to obtain the marking information corresponding to the plurality of preset fields.

2. The dispensing method of claim 1, wherein, the sorting sequence number of the plurality of database data is obtained by sorting the plurality of database data according to the storage time of the plurality of database data, which comprises the following steps: sorting the plurality of database data according to the storage time of the plurality of database data to determine the key value of the primary key of the plurality of database data.

3. The dispensing method of claim 2, wherein, when the data table in the database is expanded for multiple times, the database data comprises target data in the data table before and after the expansion; the marking information of the preset data table corresponding to the storage of each database data at different total table numbers is obtained according to the sorting sequence number of the plurality of database data, which comprises the following steps: obtaining the marking information of the preset data table corresponding to the storage of each target data at different total table numbers according to the key value of the primary key of the plurality of target data.

4. The dispensing method of claim 3, wherein, the target data table for the storage of each database data is determined by matching the total table number after the expansion with the marking information of each database data, which comprises the following steps: matching the total table number after the expansion with the marking information of each target data to determine the target data table for the storage of each target data; the target data table for the storage of each database data is determined by matching the total table number after the expansion with the marking information of each database data, which comprises the following steps: determining whether there is storage mismatched target data according to the target data table for the storage of each target data; if yes, then transplanting the storage mismatched target data to the corresponding target data table.

5. The dispensing method of claim 2, wherein, when the data table in the database is expanded only once, the database data comprises the first data table before the expansion; the marking information of the preset data table corresponding to the storage of each database data at different total table numbers is obtained according to the sorting sequence number of the plurality of database data, which comprises the following steps: According to key values of primary keys of the first data table, the marking information of the preset data table corresponding to the storage of each of the first data table is obtained.

6. The dispensing method of claim 5, wherein, The matching of the expanded total table number with the marking information of each of the first data table is used to determine the target data table for the storage of each of the first data table. The matching of the expanded total table number with the marking information of each of the first data table is used to determine the target data table for the storage of each of the first data table. The storage of each of the first data table in the corresponding target data table includes: When the target data table for the storage of the first data table is an expanded data table, the first data table is transplanted to the corresponding expanded data table, wherein the expanded data table is a data table added after the expansion of the data table before the expansion.

7. An apparatus for allocating data to a repository, characterized by The method comprises the following steps: An acquisition unit is configured to acquire a plurality of data to be stored in a database; An ordering unit is configured to order the plurality of data to be stored according to the storage time of the plurality of data to be stored, and obtain the ordering sequence number of the plurality of data to be stored; An execution unit is configured to obtain the marking information of a preset data table corresponding to the storage of each of the plurality of data to be stored according to the ordering sequence number of the plurality of data to be stored, wherein the total number of data tables is the total number of data tables in the database for storing data; A first determination unit is configured to determine the total number of data tables after the expansion of the data tables in the database; A second determination unit is configured to match the total number of data tables after the expansion with the marking information of each of the data to be stored, and determine the target data table for the storage of each of the data to be stored; A storage unit is configured to store each of the data to be stored in the corresponding target data table; The ordering sequence number comprises a primary key. The execution unit is specifically configured to take the remainder of the key value of the primary key of each of the data to be stored with respect to the different total number of data tables, and use the remainder as the table number of the preset data table corresponding to the storage of each of the data to be stored with respect to the different total number of data tables; bind each of the data to be stored with a preset field in the database, and mark the table number of the preset data table corresponding to the storage of the corresponding data to be stored with respect to the different total number of data tables in the bound preset field, to obtain the marking information corresponding to the plurality of preset fields.

8. An apparatus for allocating data to a repository, characterized by The method comprises the following steps: A central processing unit, a memory, and an input / output interface; The memory is a volatile memory or a persistent memory; The central processing unit is configured to communicate with the memory, and execute the instruction operation in the memory to perform the method in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The instructions, when executed on a computer, cause the computer to perform the method in any one of claims 1 to 6.

Images