A method and system for dynamically partitioning massive data based on fields

Abstract

The present invention relates to the field of computer communication technology, and in particular to a method and system for dynamically sharding massive data based on fields. The method comprises the following steps: step S1, preparing n databases, using n as a key and a database link string as a value, and storing information of the database link string in a configuration center; step S2, when a new user is added, reading a globally unique and incremental user number id from a Redis cache and obtaining a shard number i, then converting the shard number i into an M-digit string, where M is the number of digits of the shard quantity value, splicing the string after the user number id to form a new user number id, and then inserting the newly added user into the corresponding database; the present invention can efficiently manage and access large-scale data, and simplifies the maintenance and configuration management process of database shards and tables.

Description

Technical Field

[0001] The present invention relates to the field of computer communication technology, and in particular to a method and system for dynamically partitioning massive data into fields. Background Art

[0002] In the current technological landscape, database sharding is a common solution to address the challenges of large-scale data storage and querying. However, this approach also has some pain points and drawbacks. For example, when adding or removing database shards, data migration is required, which can lead to system downtime or data consistency issues. Another example is the high cost of configuration management and maintenance, which increases with the number of database shards. For example, when troubleshooting a problem with a specific user ID, it's impossible to intuitively identify the shard the user is in. A program must run specific rules to determine the shard the user is in, and only then can the problem be resolved. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention provides a method and system for dynamically partitioning massive data based on fields, which can efficiently manage and access large-scale data and simplify the maintenance and configuration management process of database partitioning and table partitioning.

[0004] The present invention is achieved by adopting the following technical solutions:

[0005] A method for dynamically partitioning massive data based on fields, the method comprising the following steps:

[0006] Step S1: prepare n databases, use n as the key and the database link string as the value, and store the database link string information in the configuration center;

[0007] Step S2: When a new user is added, the globally unique and incremented user ID is read from the Redis cache and the shard ID i is obtained. The shard ID i is then converted into an M-digit string, where M is the number of digits in the shard value. The string is appended to the user ID to form a new user ID, and the new user is then inserted into the corresponding database.

[0008] Step S3: When user information needs to be read, the sub-library number i is obtained according to the user's new user number id, and then the database link string con of the sub-library number i is read from the configuration center, and the user data is read from the sub-library;

[0009] Step S4: When the database with sub-library number i among the n databases is migrated to a new database, the new database link string is set to conx. The configuration center updates the database link string con corresponding to the database with sub-library number i to conx, records the original database link string as con, and marks the sub-library number i with a migration identifier. The latest database link string information is then pushed to the program.

[0010] Step S5: When user information needs to be read, the migrated sub-library number ix is ​​obtained based on the user's new user ID, and then the database link string conx of the sub-library number ix is ​​read from the configuration center, and the user data is read from the sub-library;

[0011] Step S6: If the database with sub-library number ix becomes unstable or experiences other abnormalities, the database link string information of the sub-library number ix in the configuration center is promptly modified; the database link string conx of the sub-library is changed to the database link string con, and the migration flag is removed;

[0012] If the database with sub-database number ix is ​​normal, remove the migration flag directly so that the new database completely replaces the original database and starts working.

[0013] Preferably, step S2 is further specified as follows: when a new user is added, the most recently generated user number id is read from the Redis cache through the GET lastid command, and then the value of the user number id is increased by 1, and then the obtained user number id is modulo operation with n to obtain the sub-library number i.

[0014] Preferably, the step S2 is further specified as follows: after forming a new user number id, reading the database link string con corresponding to the sub-library number i from the configuration center, and then inserting the new user into the corresponding sub-library.

[0015] Preferably, the step S3 is further specified as follows: when it is necessary to read the user information, the last M digits of the user's new user number id are intercepted to obtain the character string in step S2, and the corresponding sub-library number i is obtained according to the character string.

[0016] Preferably, the step S5 is further specified as follows: when it is necessary to read the user information, the last M digits of the user's new user number id are intercepted to obtain the character string in step S2, and the corresponding sub-library number ix is ​​obtained according to the character string.

[0017] Preferably, step S5 is further specifically as follows: when a new user registers, the most recently generated user ID is read from the Redis cache through the GET lastid command, and then the value of the user ID is increased by 1, and then the obtained user ID is modulo-operated with n to obtain the sub-library number ix;

[0018] Then the sub-library number ix is ​​spliced ​​onto the user number id to form a new user number id, and then the migration identifier of the sub-library numbered ix is ​​obtained from the configuration center, and then the new database link string conx and database link string con are obtained, and then the user data is written into these two databases at the same time.

[0019] A system for dynamically partitioning massive data into different databases based on fields, comprising a preparation module, a new addition module, a read module A, a migration module, a read module B, and a modification module;

[0020] Prepare modules and n databases, using n as the key and the database link string as the value, and store the database link string information in the configuration center;

[0021] New module, when adding a new user, read the globally unique incremental user number id from the Redis cache and obtain the shard number i, then convert the shard number i into an M-digit string, where M is the number of digits in the shard value, concatenate the string to the user number id to form a new user number id, and then insert the new user into the corresponding database;

[0022] Reading module A, when it needs to read user information, obtains the sub-library number i according to the user's new user number id, then reads the database link string con of the sub-library number i from the configuration center, and reads the user data from the sub-library;

[0023] Migration module: When the database with sub-library number i among n databases is migrated to a new database, the new database link string is set to conx. The configuration center updates the database link string con corresponding to the database with sub-library number i to conx, records the original database link string as con, and marks the sub-library number i with a migration identifier. Then, the latest database link string information is pushed to the program.

[0024] Reading module B, when it needs to read user information, obtains the migrated sub-library number ix based on the user's new user ID, then reads the database link string conx of the sub-library number ix from the configuration center, and reads the user data from the sub-library;

[0025] Modify the module. If the database with sub-library number ix becomes unstable or experiences other abnormalities, promptly modify the database link string information for the sub-library number ix in the configuration center; change the database link string conx of the sub-library to the database link string con, and remove the migration flag.

[0026] If the database with sub-database number ix is ​​normal, remove the migration flag directly so that the new database completely replaces the original database and starts working.

[0027] Preferably, when a new user is added, the most recently generated user number id is read from the Redis cache through the GET lastid command, and then the value of the user number id is increased by 1, and then the obtained user number id is modulo operation with n to obtain the sub-library number i.

[0028] Preferably, the new module is further specified as follows: after forming a new user number id, reading the database link string con corresponding to the sub-library number i from the configuration center, and then inserting the new user into the corresponding sub-library.

[0029] Preferably, the reading module A is further specified as follows: when it is necessary to read user information, the last M digits of the user's new user number id are intercepted as a character string in the newly added module, and the corresponding sub-library number i is obtained according to the character string.

[0030] Preferably, the reading module B is further specified as follows: when it is necessary to read user information, the last M digits of the user's new user number id are intercepted as a character string in the newly added module, and the corresponding sub-library number ix is ​​obtained according to the character string.

[0031] Preferably, the reading module B is further specifically configured to: when a new user registers, read the most recently generated user ID from the Redis cache through the GET lastid command, then add 1 to the value of the user ID, and then perform a modulo operation on the obtained user ID and n to obtain the sub-library number ix;

[0032] Then the sub-library number ix is ​​spliced ​​onto the user number id to form a new user number id, and then the migration identifier of the sub-library numbered ix is ​​obtained from the configuration center, and then the new database link string conx and database link string con are obtained, and then the user data is written into these two databases at the same time.

[0033] Beneficial effects of the present invention:

[0034] The present invention provides a method and system for dynamically sharding massive data based on fields, which can efficiently manage and access large-scale data and simplify the maintenance and configuration management process of database sharding and table sharding. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic flow chart of the method of the present invention.

[0036] Figure 2 It is a system principle block diagram of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below with reference to the accompanying drawings.

[0038] See also Figure 1 The present invention provides a method for dynamically partitioning massive data based on fields, the method comprising the following steps:

[0039] Step S1: prepare n databases, use n as the key and the database link string as the value, and store the database link string information in the configuration center;

[0040] Step S2: When a new user is added, the globally unique and incremented user ID is read from the Redis cache and the shard ID i is obtained. The shard ID i is then converted into an M-digit string, where M is the number of digits in the shard value. The string is appended to the user ID to form a new user ID, and the new user is then inserted into the corresponding database.

[0041] Step S3: When user information needs to be read, the sub-library number i is obtained according to the user's new user number id, and then the database link string con of the sub-library number i is read from the configuration center, and the user data is read from the sub-library;

[0042] Step S4: When the database with sub-library number i among the n databases is migrated to a new database, the new database link string is set to conx. The configuration center updates the database link string con corresponding to the database with sub-library number i to conx, records the original database link string as con, and marks the sub-library number i with a migration identifier. The latest database link string information is then pushed to the program.

[0043] Step S5: When user information needs to be read, the migrated sub-library number ix is ​​obtained based on the user's new user ID, and then the database link string conx of the sub-library number ix is ​​read from the configuration center, and the user data is read from the sub-library;

[0044] Step S6: If the database with sub-library number ix becomes unstable or experiences other abnormalities, the database link string information of the sub-library number ix in the configuration center is promptly modified; the database link string conx of the sub-library is changed to the database link string con, and the migration flag is removed;

[0045] If the database with sub-database number ix is ​​normal, remove the migration flag directly so that the new database completely replaces the original database and starts working.

[0046] The present invention will be further described below in conjunction with a specific embodiment:

[0047] Step 1: Prepare n databases, using n as the key and the database link string as the value, and store the database link string information in the configuration center.

[0048] Based on the number of databases you want to divide into for your business, for example, if you want to distribute the data volume across 10 databases, then prepare 10 databases, use the database number n as the key, and the database link string corresponding to the database as the value, and store the database link strings of the 10 databases in the configuration center. Assume the data is as follows:

[0049] key Database link string 0 con0 1 con1 2 con2 ... ...

[0050] Step 2: When a new user is added, the globally unique and incremented user ID is read from the Redis cache and the shard ID i is obtained. The shard ID i is then converted into an M-digit string, where M is the number of digits in the shard value. The string is appended to the user ID to form a new user ID, and the new user is then inserted into the corresponding database.

[0051] When a new user is added, the GET lastid command is used to read the most recently generated user ID from the Redis cache (lastid is the key of Redis, and the value is the user ID of the last user registered. GET is a Redis command, which is to read the most recently generated user ID from Redis). Then, the value of the user ID is added by 1, and then the obtained user ID is modulo-operated with n to obtain the partition number i.

[0052] After forming a new user number id, read the database link string con corresponding to the sub-database number i from the configuration center, and then insert the new user into the corresponding sub-database.

[0053] To ensure globally increasing user IDs, Redis is used as a generator for globally unique, increasing user IDs. When adding a new user, execute the GET lastid command to retrieve the most recently generated user ID from Redis. For example, if ID = 5000, add 1 to it, which equals 5001, which is the new user ID. Then, use the INCR lastid command to increment the globally unique, increasing user ID in Redis by 1.

[0054] Perform a modulo operation on the obtained user number id = 5001 and the number of sub-libraries n, 5001%10=1, to obtain the sub-library number i = 1, then convert the sub-library number into a two-digit string to obtain 01, and fill the value 0 before the sub-library number (the value is not necessarily 0, it can also be 1 or 2, etc., and the value 0 can also be filled after the sub-library number, that is, 10), and splice 01 after the user number id to obtain 500101. Since the upper limit of the number of sub-libraries is 1-99, then M is a 2-digit number, so a 2-digit string is sufficient. If the number of sub-libraries needs to be 1-999, then M is a 3-digit number, and the string needs to be 3 digits (for example: convert the sub-library number into a three-digit string to obtain 001).

[0055] Then, the database link string of the sub-library number i=1 is read from the configuration center to obtain con1, and the new user userid=500101 is inserted into the database.

[0056] Step 3: When user information needs to be read, obtain the sub-library number i according to the user's new user ID, then read the database link string con of the sub-library number i from the configuration center, and read the user data from the sub-library;

[0057] When user information needs to be read, the last M digits of the user's new user number id are intercepted to obtain the character string in step S2, and the corresponding sub-library number i is obtained according to the character string.

[0058] When user information needs to be read, based on the user's userid = 500101, the last two digits are intercepted to obtain the sub-library number i = 01, and the database link string of the sub-library number i = 1 is read from the configuration center to obtain con1, and the database is linked to read the user data of the user number = 500101.

[0059] Step 4. When the database with sub-library number i among n databases is migrated to a new database, the new database link string is set to conx. The configuration center updates the database link string con corresponding to the database with sub-library number i to conx, records the original database link string as con, and marks the sub-library number i with a migration identifier, and then pushes the latest database link string information to the program.

[0060] When the database with shard number i=1 among the 10 shards needs to be migrated to a new database, assuming the new database link string is conx, the operation and maintenance team will complete the initial work of copying and migrating the data from the original database con to the new database conx. Then, they will change the database link string configuration with the configuration center key 1 to conx, record the original database link string con in this configuration, add a migration identifier to it, and then push the latest sharding rules to the program. The modified configuration center data is as follows:

[0061] key Database link string Original link string Migration ID 0 con0 1 conx con1 true 2 con2 ...

[0062] Step 5: When user information needs to be read, obtain the migrated sub-library number ix based on the user's new user ID, then read the database link string conx of the sub-library number ix from the configuration center, and read the user data from the sub-library;

[0063] When user information needs to be read, the last M digits of the user's new user number id are intercepted to obtain the character string in step S2, and the corresponding sub-library number ix is ​​obtained according to the character string.

[0064] When a new user registers, the GET lastid command is used to read the most recently generated user ID from the Redis cache. The value of the user ID is then increased by 1. The resulting user ID is then modulo-operated with n to obtain the partition ID ix.

[0065] Then the sub-library number ix is ​​spliced ​​onto the user number id to form a new user number id, and then the migration identifier of the sub-library numbered ix is ​​obtained from the configuration center, and then the new database link string conx and database link string con are obtained, and then the user data is written into these two databases at the same time.

[0066] When the user data in the sub-database with number ix=1 needs to be read again, the last two digits of the user number are parsed in the same way as in step 3 to obtain the corresponding sub-database number ix, and then the data link string conx is obtained for reading. When a new user is registered, the globally unique and incrementing user number id is read from Redis in the same way as in step 2. The user number id is modulo-operated with n to obtain the sub-database number ix, and then the sub-database number ix is ​​concatenated to the end of the auto-incrementing user number id to form a new auto-incrementing number id. When the configuration center link string data of ix=1 is the migration identifier = true, then the new and old database link strings conx and con1 are obtained, and the user data is written to both databases at the same time.

[0067] Step 6: If the database with sub-database number ix becomes unstable or experiences other abnormalities, promptly modify the database link string information for the sub-database number ix in the configuration center. Change the database link string conx to con and remove the migration flag.

[0068] If the database with sub-database number ix is ​​normal, remove the migration flag directly so that the new database completely replaces the original database and starts working.

[0069] As the system runs, if the new database conx corresponding to the shard numbered ix becomes unstable or experiences other anomalies, requiring restoration to the original database con1, the link string configuration numbered ix=1 in the configuration center should be promptly modified to change conx to the original link string con1 and remove the migration flag, i.e., set it to false. If the new database conx is operating normally, the migration flag should be removed, allowing the new database conx to completely replace the original database con and begin operations.

[0070] Please continue reading Figure 2 , the present invention also provides a system for dynamically partitioning massive data based on fields, the system comprising a preparation module, a new addition module, a reading module A, a migration module, a reading module B and a modification module;

[0071] Prepare modules and n databases, using n as the key and the database link string as the value, and store the database link string information in the configuration center;

[0072] New module, when adding a new user, read the globally unique incremental user number id from the Redis cache and obtain the shard number i, then convert the shard number i into an M-digit string, where M is the number of digits in the shard value, concatenate the string to the user number id to form a new user number id, and then insert the new user into the corresponding database;

[0073] Reading module A, when it needs to read user information, obtains the sub-library number i according to the user's new user number id, then reads the database link string con of the sub-library number i from the configuration center, and reads the user data from the sub-library;

[0074] Migration module: When the database with sub-library number i among n databases is migrated to a new database, the new database link string is set to conx. The configuration center updates the database link string con corresponding to the database with sub-library number i to conx, records the original database link string as con, and marks the sub-library number i with a migration identifier. Then, the latest database link string information is pushed to the program.

[0075] Reading module B, when it needs to read user information, obtains the migrated sub-library number ix based on the user's new user ID, then reads the database link string conx of the sub-library number ix from the configuration center, and reads the user data from the sub-library;

[0076] Modify the module. If the database with sub-library number ix becomes unstable or experiences other abnormalities, promptly modify the database link string information for the sub-library number ix in the configuration center; change the database link string conx of the sub-library to the database link string con, and remove the migration flag.

[0077] If the database with sub-database number ix is ​​normal, remove the migration flag directly so that the new database completely replaces the original database and starts working.

[0078] When a new user is added, the most recently generated user ID is read from the Redis cache through the GET lastid command, and then the value of the user ID is increased by 1. The obtained user ID is then modulo-operated with n to obtain the partition number i.

[0079] The new module is further specified as follows: after forming a new user number id, reading the database link string con corresponding to the sub-library number i from the configuration center, and then inserting the new user into the corresponding sub-library.

[0080] The reading module A is further specifically as follows: when it is necessary to read user information, the last M digits of the user's new user number id are intercepted as the character string in the newly added module, and the corresponding sub-library number i is obtained according to the character string.

[0081] The reading module B is further specifically as follows: when it is necessary to read user information, the last M digits of the user's new user number id are intercepted as the character string in the newly added module, and the corresponding sub-library number ix is ​​obtained according to the character string.

[0082] The reading module B is further specifically configured to: when a new user registers, read the most recently generated user ID from the Redis cache through the GET lastid command, then add 1 to the value of the user ID, and then perform a modulo operation on the obtained user ID and n to obtain the sub-library number ix;

[0083] Then the sub-library number ix is ​​spliced ​​onto the user number id to form a new user number id, and then the migration identifier of the sub-library numbered ix is ​​obtained from the configuration center, and then the new database link string conx and database link string con are obtained, and then the user data is written into these two databases at the same time.

[0084] In summary, the present invention can efficiently manage and access large-scale data and simplifies the maintenance and configuration management process of database sharding and table sharding.

[0085] The above description is only a preferred embodiment of the present invention and should not be understood as limiting the present application. All equivalent changes and modifications made within the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A method for dynamically partitioning massive data based on fields, characterized by: The method comprises the following steps: Step S1: prepare n databases, use n as the key and the database link string as the value, and store the database link string information in the configuration center; Step S2: When a new user is added, the globally unique and incremented user ID is read from the Redis cache and the shard ID i is obtained. The shard ID i is then converted into an M-digit string, where M is the number of digits in the shard value. The string is appended to the user ID to form a new user ID, and the new user is then inserted into the corresponding database. Step S3: When user information needs to be read, the sub-library number i is obtained according to the user's new user number id, and then the database link string con of the sub-library number i is read from the configuration center, and the user data is read from the sub-library; Step S4: When the database with sub-library number i among the n databases is migrated to a new database, the new database link string is set to conx. The configuration center updates the database link string con corresponding to the database with sub-library number i to conx, records the original database link string as con, and marks the sub-library number i with a migration identifier. The latest database link string information is then pushed to the program. Step S5: When user information needs to be read, the migrated sub-library number ix is ​​obtained based on the user's new user ID, and then the database link string conx of the sub-library number ix is ​​read from the configuration center, and the user data is read from the sub-library; Step S6: If the database with sub-library number ix becomes unstable or experiences other abnormalities, the database link string information of the sub-library number ix in the configuration center is promptly modified; the database link string conx of the sub-library is changed to the database link string con, and the migration flag is removed; If the database with sub-database number ix is ​​normal, remove the migration flag directly so that the new database completely replaces the original database and starts working.

2. The method for dynamically partitioning massive data based on fields according to claim 1, characterized in that: The step S2 is further specified as follows: when a new user is added, the most recently generated user number id is read from the Redis cache through the GETlastid command, and then the value of the user number id is increased by 1, and then the obtained user number id is modulo operation with n to obtain the sub-library number i.

3. The method for dynamically partitioning massive data based on fields according to claim 1, characterized in that: The step S2 is further specifically as follows: after forming a new user number id, reading the database link string con corresponding to the sub-library number i from the configuration center, and then inserting the new user into the corresponding sub-library.

4. The method for dynamically partitioning massive data based on fields according to claim 1, characterized in that: The step S3 is further specified as follows: when user information needs to be read, the last M digits of the user's new user number id are intercepted to obtain the character string in step S2, and the corresponding sub-library number i is obtained according to the character string.

5. The method for dynamically partitioning massive data based on fields according to claim 1, characterized in that: The step S5 is further specified as follows: when user information needs to be read, the last M digits of the user's new user number id are intercepted to obtain the character string in step S2, and the corresponding sub-library number ix is ​​obtained according to the character string.

6. The method for dynamically partitioning massive data based on fields according to claim 1, characterized in that: The step S5 is further specifically as follows: when a new user registers, the most recently generated user ID is read from the Redis cache through the GET lastid command, the value of the user ID is increased by 1, and the obtained user ID is modulo-operated with n to obtain the sub-library number ix; Then the sub-library number ix is ​​spliced ​​onto the user number id to form a new user number id, and then the migration identifier of the sub-library numbered ix is ​​obtained from the configuration center, and then the new database link string conx and database link string con are obtained, and then the user data is written into these two databases at the same time.

7. A system for dynamically partitioning massive amounts of data based on fields, characterized by: The system includes a preparation module, a new addition module, a reading module A, a migration module, a reading module B, and a modification module; Prepare modules and n databases, using n as the key and the database link string as the value, and store the database link string information in the configuration center; New module, when adding a new user, read the globally unique incremental user number id from the Redis cache and obtain the shard number i, then convert the shard number i into an M-digit string, where M is the number of digits in the shard value, concatenate the string to the user number id to form a new user number id, and then insert the new user into the corresponding database; Reading module A, when it needs to read user information, obtains the sub-library number i according to the user's new user number id, then reads the database link string con of the sub-library number i from the configuration center, and reads the user data from the sub-library; Migration module: When the database with sub-library number i among n databases is migrated to a new database, the new database link string is set to conx. The configuration center updates the database link string con corresponding to the database with sub-library number i to conx, records the original database link string as con, and marks the sub-library number i with a migration identifier. Then, the latest database link string information is pushed to the program. Reading module B, when it needs to read user information, obtains the migrated sub-library number ix based on the user's new user ID, then reads the database link string conx of the sub-library number ix from the configuration center, and reads the user data from the sub-library; Modify the module. If the database with sub-library number ix becomes unstable or experiences other abnormalities, promptly modify the database link string information for the sub-library number ix in the configuration center; change the database link string conx of the sub-library to the database link string con, and remove the migration flag. If the database with sub-database number ix is ​​normal, remove the migration flag directly so that the new database completely replaces the original database and starts working.

8. The system for dynamically partitioning massive data based on fields according to claim 7, characterized in that: When a new user is added, the most recently generated user ID is read from the Redis cache through the GET lastid command, and then the value of the user ID is increased by 1. The obtained user ID is then modulo-operated with n to obtain the partition number i.

9. The system for dynamically partitioning massive data based on fields according to claim 7, characterized in that: The new module is further specified as follows: after forming a new user number id, reading the database link string con corresponding to the sub-library number i from the configuration center, and then inserting the new user into the corresponding sub-library.

10. The system for dynamically partitioning massive data based on fields according to claim 7, characterized in that: The reading module A is further specifically as follows: when it is necessary to read user information, the last M digits of the user's new user number id are intercepted as the character string in the newly added module, and the corresponding sub-library number i is obtained according to the character string.

11. The system for dynamically partitioning massive data based on fields according to claim 7, characterized in that: The reading module B is further specifically as follows: when it is necessary to read user information, the last M digits of the user's new user number id are intercepted as the character string in the newly added module, and the corresponding sub-library number ix is ​​obtained according to the character string.

12. The system for dynamically partitioning massive data based on fields according to claim 7, characterized in that: The reading module B is further specifically configured to: when a new user registers, read the most recently generated user ID from the Redis cache using the GETlastid command, then add 1 to the value of the user ID, and then perform a modulo operation on the obtained user ID and n to obtain the sub-library number ix; Then the sub-library number ix is ​​spliced ​​onto the user number id to form a new user number id, and then the migration identifier of the sub-library numbered ix is ​​obtained from the configuration center, and then the new database link string conx and database link string con are obtained, and then the user data is written into these two databases at the same time.

Images