ID generator design method and device
By designing the application form in the database to record the maximum value of the used ID and claiming the ID when the system starts, creating an ID generator object, the performance bottleneck when dependent on the database to generate ID is solved, and efficient ID generation is achieved.
Patent Information
- Application Number
- CN202411426917.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-28
- Publication Date
- 2025-02-18
AI Technical Summary
The existing method of dependent databases to generate IDs has performance bottlenecks, and the numeric string generated by the snowflake ID generation scheme that does not depend on the database is too long to count from 1.
Design an ID generator to record the maximum value of the used ID in the database by designing the application form in the database, and obtaining the maximum value from the table when the system is started or when it is used for the first time, claiming a certain number of IDs, and after updating the database Create an ID generator object, and subsequent ID generation is only retrieved from that object.
It realizes that when recording the maximum value of the used ID using the database, it reduces the overhead of operating the database, and the performance is close to or reaches the level of directly generating IDs without relying on DB.
Smart Images
Figure CN119474079A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer technology and software technology, and in particular to a method and device for designing an ID generator. Background Art
[0002] Relying on a database (DB) to generate IDs, each time an ID is generated, the database must be updated once, and then the latest available ID must be obtained, which requires two database operations; if an ID counter is used in the program to record the ID to be issued by the database (that is, the ID number of the program is synchronized with the ID number of the database), the database can be updated successfully, and the ID is successfully distributed once. Although the operation of reading the ID number is reduced at this time, the database needs to be updated once each distribution. Operating the database has become the performance bottleneck of this generation method (relying on a database to generate IDs). The snowflake ID global unique ID generation solution that does not rely on a database generates a very long string of numbers, and it cannot be counted from 1 like the generation solution that does not rely on a database. Is there a solution that can use the database to record the maximum value of the used IDs while reducing the overhead of operating the database? Summary of the invention
[0003] The present invention provides an ID generator design method and device, aiming to solve at least one of the above technical problems.
[0004] To this end, one object of the present invention is to provide a method for solving the above-mentioned problem; another object of the present invention is to provide a device for solving the above-mentioned problem.
[0005] In order to achieve the above first object, the present invention proposes the following technical solution:
[0006] Solution 1:
[0007] Design a database claim table to record the maximum value of the ID that has been claimed for a certain identifier. The fields should at least include the id identifier, the maximum ID that has been claimed, and the number of IDs claimed this time. When the system starts or is used for the first time, find the maximum value of the ID that has been claimed for the specified identifier from the table; claim a certain number of IDs, and update the database. After the update is successful, use the three parameters of the ID identifier, the maximum value of the ID that has been claimed before the update, and the number of IDs claimed to create an ID generator object. The ID generator has three properties: the ID identifier, the ID counter that records the maximum ID number that has been distributed by the current object, and the number of remaining ID numbers. In the future, the ID of the identifier can only be obtained from the ID generator object, and the acquisition range is [the maximum value of the ID that has been claimed before the update + 1, the maximum value of the ID that has been claimed before the update + the number of claims]. When obtaining the ID, it will first check whether the claimed ID is almost used up. If so, a thread will be started to claim a batch of new IDs; and the generated ID will be returned.
[0008] Option 2:
[0009] Each time the system is started, the maximum value of a certain ID that has been used must be searched from the database and used as the initial value to initialize the generator. There is no need to synchronize with the DB in the future; this is equivalent to claiming all the numbers at once in the case of solution 1.
[0010] Option 3:
[0011] The maximum value of the claimable ID in Plan 1 is the maximum value of the type. If the Long type is used, it is the maximum value of the Long type. For the distributed situation, further improvements are made. The highest few digits are taken as the node code, and the rest are used as the maximum value of the claimable ID. The highest digits of the value must be added to each ID received, such as: 10*1000000000000000000+10000; you can also use binary and use the highest digits of the binary as the node code. A ds field should also be added to the claim form to save the data table node (data node) corresponding to the table, and the ID identifier should also add the ds part.
[0012] The above three solutions are related. Solution 2 is a simplified version of Solution 1; Solution 3 adds sharding function on the basis of Solution 1.
[0013] In addition, changing the claim table to a sharded broadcast table can prevent the single point problem of the claim table. A broadcast table refers to a table that exists in all sharded databases (the same broadcast table has one on each node, and the data is the same). The table structure and its data are completely consistent in each database. In addition, the ID generated by the ID generator has reached the maximum value and can be recycled or an error will be prompted.
[0014] In order to achieve the above second purpose, the present invention proposes the following technical solution:
[0015] The device includes a DB operation module, an ID generator module, and a generator factory module. The DB operation module includes operating the application form, obtaining the maximum value of the used ID, applying for a certain number of IDs (updating the application form), etc.; the ID generator module includes an ID generator, an application class, etc.; the generator factory module includes an ID generator factory, etc. Before generating an ID generator object, the ID generator factory must first know or define the ID identifier, and then obtain the maximum value of the used ID of the corresponding identifier through the DB operation module, and then apply for a certain number of IDs; next, the ID generator object is generated using three parameters: the ID identifier, the maximum value of the used ID before the update, and the number of IDs applied for. In the future, the ID of the identifier can be obtained only from the ID generator object; a more common practice is to remember the object in the ID generator factory, so that the next ID of the corresponding ID identifier can be obtained from the ID generator factory.
[0016] The generator factory module can call the DB operation module and the ID generator module, and the ID generator module can call the DB operation module.
[0017] The beneficial effects achieved by the present invention are:
[0018] The database can be used to record the maximum value of the used ID. In terms of performance, it is infinitely close to or even the same as the performance when it is directly generated by the program without relying on the DB. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 , the table structure and sample data of the claim form;
[0020] Figure 2 , update the record instance of the claim form and the updated record;
[0021] Figure 3 , flow chart of the main steps of generating ID;
[0022] Figure 4 , module call flow chart. DETAILED DESCRIPTION
[0023] The specific implementation of the present invention provides some ID generator design methods and device implementations.
[0024] The embodiments described below in conjunction with the accompanying drawings (when involving specific language descriptions, the computer languages used are Java and SQL) are only some implementation methods of the present invention and should not be construed as limitations of the present invention.
[0025] The design method of the present invention is as follows: generating ID is divided into two stages. First, batches of IDs are obtained from the database to the program, and then the user obtains the ID from the program. The database records the maximum value of the ID currently allowed to be used; the ID generator first obtains a batch of ID numbers from the database, and then allows the user to obtain the ID number from the ID generator object. Only the obtained IDs are allowed to be used. When they are used up or almost used up, a new batch is applied for.
[0026] Design a database claim table to record the maximum value of the ID that has been claimed for a certain ID. The fields should at least include the ID, the maximum ID that has been claimed, and the number of IDs claimed this time. Detailed table structure and sample data, such as Figure 1 When the system is started or used for the first time, the maximum ID value of the specified ID that has been taken is found from the table; a certain number of IDs are applied for, and the caller updates the database; the update instance and the updated record are as follows Figure 2As shown. After the update is successful, create an ID generator object using three parameters: ID identifier, maximum ID number that has been collected before the update, and number of IDs to apply for. The ID generator has three properties: ID identifier, ID counter that records the maximum ID number that has been distributed to the current object, and the number of remaining ID numbers. To obtain the ID of this identifier in the future, only obtain it from the ID generator object, and the acquisition range is [maximum ID number that has been collected before the update + 1, maximum ID number that has been collected before the update + number of IDs applied for]. The corresponding flow chart is as follows Figure 3 When obtaining IDs, it will first check whether the IDs that have been applied for are almost used up. If so, a thread will be started to apply for a batch of new IDs; the generated IDs will be returned.
[0027] Advantages: High efficiency, and the DB can record the maximum ID value that has been claimed. Disadvantages: In case of a sudden shutdown, the claimed IDs have not been used up. If the maximum ID value is found from the database claim table next time as the initial value of the generator, the claimed but unused ID number will be wasted, resulting in number skipping. The solution is to find the table and field used by the ID identifier, and find the maximum ID value actually used as the initial value. In scenarios where number skipping is not allowed, it is necessary to pay attention to "used" and whether it is claimed from the database and distributed by the program. And pay attention to the corresponding solution. In the description, the distinction is not very strict.
[0028] The device described in the present invention at least includes: a DB operation module, an ID generator module, and a generator factory module. The DB operation module includes operating the application form, obtaining the maximum value of the used ID, applying for a certain number of IDs (updating the application form), etc.; the ID generator module includes an ID generator, an application class, etc.; the generator factory module includes an ID generator factory, etc. Before generating an ID generator object, the ID generator factory must first know or define the ID identifier, and then obtain the maximum value of the used ID of the corresponding identifier through the DB operation module, and then apply for a certain number of IDs; next, the ID generator object is generated using three parameters: the ID identifier, the maximum value of the used ID before the update, and the number of IDs applied for. In the future, the ID of the identifier can be obtained only from the ID generator object; a more common practice is to remember the object in the ID generator factory, so that the next ID of the corresponding ID identifier can be obtained from the ID generator factory.
[0029] The generator factory module can call the DB operation module and the ID generator module, and the ID generator module can call the DB operation module. The module calling process is as follows Figure 4 shown.
[0030] The following is a further detailed explanation based on the source code and comments in the appendix.
[0031] Embodiment 1:
[0032] The source code SimpleEfficientId.java is a reference implementation of the ID generator (not the only one). It contains three properties: ID counter sequenceNumber, number of remaining ID numbers leftAmount, ID identifier idFlag, and size attribute for the number of ID numbers claimed. Among them, sequenceNumber records the maximum value of the ID numbers assigned to the current object; for the ID identifier, you can use "table name:: field name", so that you can easily know which field in which table uses the ID number. To create an ID generator, three parameters are required: the maximum value of the used ID before the update as the initial value initNum, the ID identifier, and the number of claims; the claim data can use the default value, and in this case, the two-parameter constructor can be used.
[0033] After creating the object, you can provide the service of obtaining ID. The method of obtaining a single ID is shown in the following code.
[0034]
[0035] First determine whether it is almost finished. If the remaining ID numbers are less than 20% of the claimed ID numbers, another thread will be started to claim another batch. You can also use a synchronous method without threads to claim a new batch first and then issue the numbers. After the number of remaining ID numbers is reduced by 1, the ID number is returned to the previous layer. Another thread is started to complete the claim, and it runs separately from the thread that obtains the ID, which will not affect the efficiency of the thread that obtains the ID. It can be considered that the performance of generating ID numbers is the same as that when it is generated directly by the program without relying on the DB.
[0036] addLeftAmount adds the claimed ID number to the remaining ID number.
[0037] The implementation reference of the thread for claiming a batch of ID numbers is shown in the ClaimedAmountThread.java source code in the appendix.
[0038] The main functions of the ClaimedAmountThread thread are as follows:
[0039] 1) Get a batch (claimedAmount) of ID numbers for idFlag from the database;
[0040] 2) After the database is updated successfully, the claimedAmount number of ID numbers obtained are added to the remaining ID number leftAmount of the idGen object.
[0041] The DB operation module includes operating the claim table, obtaining the maximum value of used IDs, and claiming a certain number of IDs (updating the claim table). To claim a certain number of IDs, you can refer to the claimedInDB() method of the ClaimedAmountThread class, the executed SQL example, and the effect after the data is modified, such as Figure 2 As shown; this method can be moved to the database module. You can query only a certain ID, or query the entire claim table, and traverse to generate multiple ID generator objects. The entity class Idclaim is a standard Javabean, which can be obtained from the table structure of the claim table and is not listed here.
[0042] The reference of the generator factory implementation of the generator factory module is shown in the GenIdFactory.java source code in the appendix. You can generate a corresponding ID generator for each ID identifier in the application form, and then use the addSimpleEfficientId method of GenIdFactory to add it to GenIdFactory. Later, you can use the public static long getId(String idFlag) method to get the next ID value of the corresponding idFlag.
[0043] In addition, the Long type in the above example can also be other integer types such as Integer. Related classes can be defined in a generic way.
[0044] Embodiment 2:
[0045] Except when creating an ID generator object, you need to synchronize with the DB, and you don't want to access the DB at other times. Every time you start the system, you can search for the maximum value of a certain ID identifier in the table using the ID identifier in the database; and use it as the initial value to initialize the generator. There is no need to synchronize with the DB in the future; it is equivalent to claiming all ID numbers at once when creating the ID generator object in the case of embodiment one. When creating the ID generator object, just use the maximum value of the ID identifier that has been searched out as initNum, and let leftAmount = Long.MAX-initNum; the size attribute, isNotRich() and ClaimedAmountThread can be removed; there is no need to synchronize with the DB in the future.
[0046] If skipping is not allowed, the first embodiment also has a similar problem. In the first embodiment, if the system is shut down suddenly, the unused IDs will be wasted. If skipping is not allowed, when the system is restarted, the initNum value of the ID generator cannot be obtained from the claim table, but the maximum value of the corresponding field must be obtained from the table using the ID identifier.
[0047] Advantages: After initialization, the program directly generates the next ID in memory, without updating the database each time. Disadvantages: The maximum value of the used ID must be found from the usage.
[0048] Embodiment three:
[0049] The maximum value of the receivable ID in the first embodiment is the maximum value of the type. If the Long type is used, it is the maximum value of the Long type. For the distributed situation, further improvements are made. Take the highest few digits as the node code, and the rest are used as the maximum value of the receivable ID. The value of the highest few digits must be added to each ID collected, such as: 10*1000000000000000000+10000; you can also use binary and use the highest few digits of the binary as the node code. A ds field should also be added to the claim table to save the data table node (data node) corresponding to the table, and the ID identifier should also add the ds part. The ID identifier can use the "ds::table name::field name" format; in the case of sharding, the table name is the name of the actual table, such as: dsl:users0:userid. Add the ds field to the claim table (if ds is parsed out through the ID identifier naming rules, the ds field can also be omitted); in addition, you can also add an ID identifier to indicate whether the ID generated is used for the sharding table.
[0050] In the above embodiment, the claim table only exists in one DB node, which has a single point problem. Changing the claim table to a sharded broadcast table can prevent the single point problem of the claim table. Because the sharded broadcast table is the same in each node. When updating the broadcast table, although all nodes need to be changed, it is completed by another thread, which runs separately from the thread for obtaining the ID, and will not affect the efficiency of the thread for obtaining the ID.
[0051] Although the embodiments of the present invention have been given and described above, they are not intended to limit the present invention. For those skilled in the art, various changes and variations may be made in the present application, including different computer languages for implementation. Without departing from the principles and design ideas of the present invention, various changes, modifications, substitutions and variations of these embodiments shall all be included within the scope of the claims of the present invention.
[0052] Appendix: Java source code
[0053] The Java source code used in the embodiment is shown below.
[0054]
[0055]
[0056]
[0057]
[0058]
Claims
1. A method for designing an ID generator, characterized in that: include: Generating IDs is divided into two stages. First, batches of IDs are obtained from the database into the program, and then the user obtains the ID from the program. The database records the maximum ID value currently allowed to be used; the ID generator first obtains a batch of ID numbers from the database, and then allows the user to obtain the ID number from the ID generator object; Design a database claim table to record the maximum value of the ID that has been claimed for a certain identifier. The fields should at least include the id identifier and the maximum ID that has been claimed. When the system starts or is used for the first time, find the maximum value of the ID that has been claimed for the specified identifier from the table. Claim a certain number of IDs and update the database. After the update is successful, create an ID generator object using the three parameters of the ID identifier, the maximum value of the ID that has been claimed before the update, and the number of IDs claimed. The ID generator has three properties: the ID identifier, the ID counter that records the maximum ID number that has been distributed to the current object, and the number of remaining ID numbers. To obtain the ID of the identifier in the future, only obtain it from the ID generator object, and the acquisition range is [the maximum value of the ID that has been claimed before the update + 1, the maximum value of the ID that has been claimed before the update + the number of claims]. When they are used up or almost used up, apply for a new batch.
2. The method according to claim 1, characterized in that: When the ID numbers are about to run out, start a thread to apply for a batch of new ID numbers; The performance of generating ID numbers is the same as when they are generated directly by the program without relying on the DB.
3. The method according to claim 1, characterized in that: When generating an ID generator object, claim all ID numbers at once and do not need to claim from the DB again later.
4. The method according to claim 1, characterized in that: By adding a ds field in the claim form or an ID identifier to add a part that distinguishes ds, the ID generator can have a distributed feature.
5. The method according to claim 1, characterized in that: Changing the claim table to a sharded broadcast table can prevent the single point problem of the claim table.
6. An ID generator device, characterized in that: include: DB operation module, ID generator module, generator factory module; DB operation module includes operation application form, obtains the maximum value of used ID, and applies for a certain number of IDs (updates application form); ID generator module includes ID generator and application class; generator factory module includes ID generator factory; before generating ID generator object, ID generator factory must first know or define ID identification, and then obtain the maximum value of used ID of corresponding identification through DB operation module, and then apply for a certain number of IDs; next, use ID identification, maximum value of used ID before update, and number of IDs applied for to generate ID generator object; in the future, the ID of the identification can be obtained only from the ID generator object; the object can also be remembered in ID generator factory, and then the next ID of the corresponding ID identification can be obtained from ID generator factory; Generating IDs is divided into two stages. First, the IDs are obtained in batches from the database to the program, and then the user obtains the IDs from the program.
7. The device according to claim 6, characterized in that The generator factory module can call the DB operation module and the ID generator module, and the ID generator module can call the DB operation module.