Identification information processing method and device, electronic equipment and storage medium
By generating and allocating identifiers in the cache table, the problem of low identifier allocation efficiency in the prior art is solved, achieving efficient identifier allocation and optimized node affinity, thereby improving read and write performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-03-27
AI Technical Summary
In existing technologies, identifier allocation schemes suffer from poor performance, especially in storage architectures based on redirected writes or log structures, where allocating globally unique OLOG identifiers during data writing is inefficient.
In response to a received identifier allocation request from a target node, the system parses the request content, queries the cache table, generates a preset number of identifiers to be allocated, stores them in the corresponding cache queue, and finally sends them to the target node in batches, thus achieving batch identifier allocation.
It improves the efficiency of identifier allocation, enhances the affinity between identifiers and nodes, improves subsequent read and write performance, reduces the dependence on the global identifier generator, and reduces computational overhead and network latency.
Smart Images
Figure CN121750607A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for processing identification information. Background Technology
[0002] With the popularization of cloud computing and big data technologies, distributed storage systems need to handle massive amounts of data and high-concurrency access. Storage architectures based on redirect-on-write or log-structured architectures are widely used. In such architectures, data writes allocate new object logs (OLOGs) for storage instead of overwriting old data. This requires the system to efficiently and reliably allocate globally unique OLOG identifiers (IDs).
[0003] In existing technologies, common identifier allocation schemes often suffer from poor performance. Summary of the Invention
[0004] Embodiments of this disclosure provide a method, apparatus, electronic device, and storage medium for processing identification information.
[0005] In a first aspect, embodiments of this disclosure provide an identification information processing method, comprising: in response to receiving an identification allocation request sent by a target node, parsing the identification allocation request to determine a target node identification and a target request quantity; querying a cache queue corresponding to the target node identification in a cache table to determine whether it includes idle identifications; in response to determining that it does not include idle identifications, generating a preset number of identifications to be allocated; determining a corresponding node identification for each identification to be allocated and storing each identification to be allocated in a cache queue corresponding to each node identification; and retrieving the target request quantity of identifications from the cache queue corresponding to the target node and sending them to the target node.
[0006] Secondly, embodiments of this disclosure provide an identifier information processing apparatus, comprising: a request parsing unit configured to parse an identifier allocation request in response to receiving an identifier allocation request sent by a target node, and determine a target node identifier and a target request quantity; a cache query unit configured to query a cache queue corresponding to the target node identifier in a cache table to determine whether it includes idle identifiers; an identifier generation unit configured to generate a preset number of identifiers to be allocated in response to determining that idle identifiers are not included; an identifier storage unit configured to determine the corresponding node identifier for each identifier to be allocated and to store each identifier to be allocated in a cache queue corresponding to each node identifier; and an identifier allocation unit configured to retrieve the target request quantity of identifiers from the cache queue corresponding to the target node and send them to the target node.
[0007] Thirdly, embodiments of this disclosure provide an electronic device including a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the identification information processing method as described in the first aspect.
[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the identification information processing method as described in the first aspect.
[0009] By applying the technical solution of this disclosure, after receiving an identifier allocation request sent by the target node, the identifier allocation request can be parsed to determine the target node identifier and the target request quantity. The cache queue corresponding to the target node identifier in the cache table is then queried to determine if it includes free identifiers. If no free identifiers are included, a preset number of identifiers to be allocated are generated, and a corresponding node identifier is determined for each identifier to be allocated, and each identifier to be allocated is stored in the cache queue corresponding to that node identifier. Finally, the target request quantity of identifiers is retrieved from the cache queue corresponding to the target node and sent to the target node. In this solution, identifiers are allocated to each storage node in batch processing, thereby improving identifier allocation efficiency. Furthermore, the allocated identifiers have affinity with the nodes, which can improve subsequent read and write performance.
[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0012] Figure 1 An exemplary system architecture diagram in which an embodiment of the identification information processing method of this disclosure can be applied;
[0013] Figure 2 This is a flowchart illustrating an embodiment of the identification information processing method disclosed herein;
[0014] Figure 3 This is a schematic flowchart of another embodiment of the identification information processing method of this disclosure;
[0015] Figure 4 This is a schematic diagram of the structure of one embodiment of the identification information processing device of this disclosure;
[0016] Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device disclosed herein. Detailed Implementation
[0017] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.
[0018] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0019] Where there is no conflict, the embodiments and features described herein can be combined with each other.
[0020] To make the technical solutions and advantages of this disclosure clearer, the following description, in conjunction with the accompanying drawings and specific embodiments, will provide a more detailed account of this disclosure.
[0021] Figure 1 An exemplary system architecture 100 is shown, to which embodiments of the identification information processing method or identification information processing apparatus of this disclosure may be applied.
[0022] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and hard disk pools 103, 104, and 105. Network 102 serves as the medium for providing a communication link between terminal device 101 and hard disk pools 103, 104, and 105. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.
[0023] Users can use terminal device 101 to interact with disk pools 103, 104, and 105 via network 102 to write data to nodes in the disk pools, etc. Various communication client applications, such as data processing applications, can be installed on terminal device 101.
[0024] Terminal device 101 can be hardware or software. When terminal device 101 is hardware, it can be various electronic devices, including but not limited to smartphones, tablets, in-vehicle computers, laptops, and desktop computers. When terminal device 101 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module. No specific limitations are made here.
[0025] Each of disk pools 103, 104, and 105 may include multiple storage nodes (Rank), and users can write data to the nodes through terminal device 101. Each of disk pools 103, 104, and 105 may include a pool master service device for managing the nodes. The pool master service device can be a device selected by the nodes in the disk pool through a distributed consensus protocol.
[0026] It should be noted that the pool master service device can be either hardware or software. When the pool master service device is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the pool master service device is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.
[0027] It should be noted that the identification information processing method provided in this embodiment is generally executed by the pool owner service device. Accordingly, the identification information processing device is generally located in the pool owner service device.
[0028] It should be understood that Figure 1 The terminal devices, networks, disk pools, and the number of nodes in the disk pools shown are merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, disk pools, and nodes.
[0029] Figure 2 A flow 200 of one embodiment of the identification information processing method of this disclosure is shown. For example... Figure 2 As shown, the identification information processing method in this embodiment may include the following steps:
[0030] Step 201: In response to receiving the identifier allocation request sent by the target node, parse the identifier allocation request to determine the target node identifier and the target request quantity.
[0031] In this embodiment, the entity executing the identification information processing method (e.g.) Figure 1 The pool master service device shown can receive identifier allocation requests from any node in its disk pool. An identifier allocation request can be sent by a node in the disk pool to the execution entity when it receives a data write request from a client and finds that there is no cached identifier locally. It is understood that the execution entity and the node sending the identifier allocation request reside in the same disk pool. The identifier allocation request can include a node identifier and a request quantity. Here, when processing a received identifier allocation request, the node that sent the identifier allocation request is called the target node, the parsed node identifier is called the target node identifier, and the parsed request quantity is called the target request quantity.
[0032] Step 202: Query the cache queue corresponding to the target node identifier in the cache table to determine whether it includes an idle identifier.
[0033] After determining the target node identifier, the execution entity can query the cache table locally. This cache table can include multiple cache queues, each storing identifiers. It's understood that the identifiers in the cache queues are unassigned identifiers, called free identifiers. Each cache queue exists as an array, with the sub-index of each array containing node identifiers, and the contents of each array being node identifiers. There is a one-to-one correspondence between each cache queue and a node identifier. Each cache queue has an upper limit on the number of identifiers it can store, which is the maximum value the array can hold. The execution entity can query the cache table for the cache queue corresponding to the target node identifier to determine if that cache queue contains free identifiers.
[0034] Step 203: In response to determining that there are no idle identifiers, a preset number of identifiers to be assigned are generated.
[0035] If the executing entity finds that the local cache queue does not contain free identifiers, it can generate a preset number of identifiers to be allocated. Specifically, it can send a request to a global identifier generator shared by all disk pools, which will then generate the preset number of identifiers based on historical identifiers. Alternatively, the executing entity can obtain the maximum value of historical identifiers from the global identifier generator and then increment this maximum value to obtain the preset number of identifiers. The generated identifiers are the identifiers to be allocated. It's important to understand that these identifiers are allocated to each node in the disk pool, not individually to the target node. Here, the preset number can be a value N times larger than the number of nodes, for example, 10,000.
[0036] Step 204: Determine the corresponding node identifier for each identifier to be assigned and store each identifier to be assigned in the cache queue corresponding to each node identifier.
[0037] After obtaining each identifier to be assigned, the executing entity can determine the corresponding node identifier for each identifier. Specifically, it can perform a hash calculation on each identifier to be assigned and determine the corresponding node identifier based on the resulting hash value. Alternatively, it can determine the corresponding node identifier for each identifier to be assigned based on a pre-established mapping relationship between node identifiers and identifiers.
[0038] After determining the corresponding node identifier for each identifier to be assigned, the identifiers can be stored in the cache queue corresponding to each node identifier. This achieves identifier allocation for each node, effectively reducing the number of times identifiers are requested from the global identifier generator. Furthermore, the allocated identifiers have affinity with the nodes, improving subsequent read and write performance.
[0039] Step 205: Retrieve the target request quantity identifier from the cache queue corresponding to the target node and send it to the target node.
[0040] After storing each identifier in its respective cache queue, the executing entity can retrieve the target number of identifiers from the cache queue corresponding to the target node and send them to the target node. Upon receiving the target number of identifiers, the target node can sequentially retrieve the identifiers from the received identifiers and write the data to them.
[0041] The identification information processing method provided in the above embodiments of this disclosure can, after receiving an identification allocation request sent by a target node, parse the identification allocation request to determine the target node identification and the target request quantity. It then queries the cache queue corresponding to the target node identification in the cache table to determine if it includes idle identifications. If idle identifications are not included, a preset number of identifications to be allocated are generated, and a corresponding node identification is determined for each identification to be allocated, and each identification to be allocated is stored in the cache queue corresponding to its respective node identification. Finally, the target request quantity of identifications is retrieved from the cache queue corresponding to the target node and sent to the target node. In this scheme, identifications are allocated to each storage node in batch processing, thereby improving identification allocation efficiency.
[0042] See also Figure 3 This illustrates a flow 300 of another embodiment of the identification information processing method according to the present disclosure. For example... Figure 3 As shown, the method in this embodiment may include the following steps:
[0043] Step 301: In response to receiving the identifier allocation request sent by the target node, parse the identifier allocation request to determine the target node identifier and the target request quantity.
[0044] Step 302: Query the cache queue corresponding to the target node identifier in the cache table to determine whether it includes an idle identifier.
[0045] Step 303: In response to determining that there are no idle identifiers, a preset number of identifiers to be assigned are generated.
[0046] Step 304: Perform a hash operation on each identifier to be assigned to obtain the hash value corresponding to each identifier; determine the node identifier corresponding to each identifier based on the obtained hash value.
[0047] In this embodiment, a hash operation can be performed on each identifier to be assigned to obtain a hash value corresponding to each identifier. Based on the obtained hash value, the node identifier corresponding to each identifier to be assigned can be determined. Specifically, the correspondence between the identifier to be assigned and the node identifier can be determined by the following formula:
[0048] LPGID=(Hash(OLOG_ID)%65536).
[0049] Here, OLOG_ID is the identifier to be assigned, and LPG ID is the array identifier. After obtaining the LPG ID, the target node identifier (Rank ID) associated with that OLOG ID can be obtained by querying the LPG cache table using the LPG ID as the index.
[0050] Step 305: Obtain the status of each node.
[0051] After determining the correspondence between the identifier to be assigned and the node identifier, the status of each node can be obtained. Here, the node status can include normal status and abnormal status.
[0052] In some optional implementations of this embodiment, the executing entity can obtain the health status indicators of each node in real time; and determine whether the status of each node is normal based on the health status indicators.
[0053] In this implementation, health status indicators may include, but are not limited to, IO latency, heartbeat timeout count, and error rate. By comparing these health status indicators with their corresponding thresholds, the status of each node can be determined. For example, when a node's health performance indicators consistently exceed the threshold, it can be marked as a "slow disk," indicating an abnormal state.
[0054] Step 306: For each node, in response to determining that the node's state is abnormal or the replica of the node is abnormal, discard the unallocated identifier corresponding to the node according to a preset probability, and store the unallocated identifiers that are not discarded in the cache queue corresponding to the node.
[0055] If a node malfunctions or its replicas become abnormal, the unassigned identifiers corresponding to that node can be discarded according to a preset probability, and the remaining unassigned identifiers can be stored in the corresponding cache queue for that node. In some practical applications, the preset probability can be determined based on the node's health performance metrics. For example, a pre-determined functional relationship can be established between the above probability and the health performance metrics. The above probability is then updated in real time based on the health performance metric values.
[0056] Step 307: In response to determining that the node is in a normal state, store each identifier to be allocated into the cache queue corresponding to each node identifier until the cache queue is full.
[0057] In this embodiment, if the node is in a normal state, each identifier to be assigned can be stored in the cache queue corresponding to the node identifier until the cache queue is full.
[0058] Step 308: Retrieve the target request quantity identifier from the cache queue corresponding to the target node and send it to the target node.
[0059] Step 309: In response to the detection of a change in the number of nodes in the disk pool, determine the updated node topology; and rebuild the cache table based on the updated node topology.
[0060] In this embodiment, the executing entity can also monitor whether the number of nodes in the disk pool has changed. If the number of nodes changes, the updated node topology can be determined. Specifically, the topology can be determined based on the connection relationships between nodes. Alternatively, the updated node topology can be determined based on the location of the updated nodes combined with the node topology before the update. Then, the cache table is rebuilt based on the updated node topology. Specifically, the number of cache queues in the cache table can be re-determined based on the number of nodes. It is understood that after updating the cache table, identifiers can be allocated based on the updated cache table.
[0061] The identification information processing method provided in the above embodiments of this disclosure eliminates network latency because most identification allocation requests are satisfied by the local cache of the nodes; the identification allocation process is implemented through cache table lookup, resulting in extremely low computational overhead and supporting millions of IOPS. By reusing the CRUSH algorithm, the storage location calculated during allocation is guaranteed to be completely consistent with the actual read / write location, allowing subsequent data reads to be directly accessed locally, significantly reducing read latency. Batch allocation and cache queue mechanisms greatly reduce interaction with the central node; flow control mechanisms prevent identification waste; and the cache table persistence strategy eliminates database write amplification. Allocation responsibility is delegated to each pool, and the system can achieve horizontal scaling by adding disk pools, eliminating global bottlenecks. The steady-state event-based update strategy protects the system from temporary faults; the cache table persistence and fast recovery mechanism ensure data consistency and service continuity after failover.
[0062] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an identification information processing device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0063] like Figure 4 As shown, the identification information processing device 400 in this embodiment includes: a request parsing unit 401, a cache query unit 402, an identification generation unit 403, an identification storage unit 404, and an identification allocation unit 405.
[0064] The request parsing unit 401 is configured to parse the identifier allocation request in response to receiving an identifier allocation request sent by the target node, and determine the target node identifier and the target request quantity.
[0065] The cache query unit 402 is configured to query the cache queue corresponding to the target node identifier in the cache table to determine whether it includes an idle identifier.
[0066] The identifier generation unit 403 is configured to generate a preset number of identifiers to be assigned in response to determining that there are no idle identifiers.
[0067] The identifier storage unit 404 is configured to determine the corresponding node identifier for each identifier to be assigned and to store each identifier to be assigned in the cache queue corresponding to each node identifier.
[0068] The identifier allocation unit 405 is configured to retrieve the target request number of identifiers from the cache queue corresponding to the target node and send them to the target node.
[0069] In addition, an electronic device is also proposed in the technical solution of this application.
[0070] Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure is shown.
[0071] like Figure 5 As shown, the electronic device may include a processor 501, a memory 502, a bus 503, and a computer program stored in the memory 502 and executable on the processor 501. The processor 501 and the memory 502 communicate with each other via the bus 503. When the processor 501 executes the computer program, it implements the steps of the above method, including, for example: in response to receiving an identifier allocation request sent by a target node, parsing the identifier allocation request to determine the target node identifier and the target request quantity; querying the cache queue corresponding to the target node identifier in the cache table to determine whether it includes idle identifiers; in response to determining that it does not include idle identifiers, generating a preset number of identifiers to be allocated; determining the corresponding node identifier for each identifier to be allocated and storing each identifier to be allocated in the cache queue corresponding to each node identifier; and retrieving the target request quantity of identifiers from the cache queue corresponding to the target node and sending them to the target node.
[0072] In addition, one embodiment of this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the above-described method, including, for example,: in response to receiving an identifier allocation request sent by a target node, parsing the identifier allocation request to determine the target node identifier and the target request quantity; querying the cache queue corresponding to the target node identifier in the cache table to determine whether it includes idle identifiers; in response to determining that it does not include idle identifiers, generating a preset number of identifiers to be allocated; determining the corresponding node identifier for each identifier to be allocated and storing each identifier to be allocated in the cache queue corresponding to each node identifier; and retrieving the target request quantity of identifiers from the cache queue corresponding to the target node and sending them to the target node.
[0073] In summary, the technical solution disclosed herein can parse the identifier allocation request sent by the target node after receiving it, determine the target node identifier and the target request quantity, and query the cache queue corresponding to the target node identifier in the cache table to determine whether it includes free identifiers. If it does not include free identifiers, a preset number of identifiers to be allocated are generated, and a corresponding node identifier is determined for each identifier to be allocated, and each identifier to be allocated is stored in the cache queue corresponding to its respective node identifier. Finally, the target request quantity of identifiers is retrieved from the cache queue corresponding to the target node and sent to the target node. This solution allocates identifiers to each storage node in batch processing, thereby improving identifier allocation efficiency. Furthermore, the allocated identifiers have affinity with the nodes, which can improve subsequent read and write performance.
[0074] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A method for processing identification information, comprising: In response to receiving an identifier allocation request sent by the target node, the identifier allocation request is parsed to determine the target node identifier and the target request quantity; Query the cache queue corresponding to the target node identifier in the cache table to determine whether it includes an idle identifier; In response to the determination that there are no idle identifiers, a preset number of identifiers to be assigned are generated; Determine the corresponding node identifier for each identifier to be assigned and store each identifier to be assigned in the cache queue corresponding to each node identifier; The number of identifiers for the target request quantity are retrieved from the cache queue corresponding to the target node and sent to the target node.
2. The method according to claim 1, wherein, The step of determining the corresponding node identifier for each identifier to be assigned includes: Perform a hash operation on each identifier to be assigned to obtain the hash value corresponding to each identifier; Based on the obtained hash value, determine the node identifier corresponding to each identifier to be assigned.
3. The method according to claim 1, wherein, The step of storing each identifier to be assigned into the cache queue corresponding to each node identifier includes: Get the status of each node; For each node, in response to determining that the node's state is abnormal or the state of the node's replica is abnormal, the unallocated identifier corresponding to the node is discarded according to a preset probability, and the unallocated identifiers are stored in the cache queue corresponding to the node.
4. The method according to claim 3, wherein, The step of storing each identifier to be assigned into the cache queue corresponding to each node identifier includes: In response to determining that the node is in a normal state, each unassigned identifier is stored in the cache queue corresponding to each node identifier until the cache queue is full.
5. The method according to claim 4, wherein, The method further includes: Real-time acquisition of health status indicators for each node; Based on the health status indicators, determine whether the status of each node is normal.
6. The method according to claim 1, wherein, The method further includes: In response to a change in the number of nodes in the disk pool, the updated node topology is determined. The cache table is rebuilt based on the updated node topology.
7. The method according to claim 1, wherein, The method further includes: In response to receiving a data write request, the data write request is load-balanced to the target node, so that the target node can look up an available identifier in its local cache, and send the identifier allocation request when it is determined that there is no available identifier in its local cache.
8. An identification information processing device, comprising: The request parsing unit is configured to parse the identifier allocation request sent by the target node in response to receiving the identifier allocation request, and determine the target node identifier and the target request quantity; The cache query unit is configured to query the cache queue corresponding to the target node identifier in the cache table to determine whether it includes an idle identifier. The identifier generation unit is configured to generate a preset number of identifiers to be assigned in response to determining that there are no idle identifiers; The identifier storage unit is configured to determine the corresponding node identifier for each identifier to be assigned and to store each identifier to be assigned in the cache queue corresponding to each node identifier; The identifier allocation unit is configured to retrieve the target request number of identifiers from the cache queue corresponding to the target node and send them to the target node.
9. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the identification information processing method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the identification information processing method as described in any one of claims 1 to 7.