Method for improving bidirectional lookup delay and data consistency in large database

Abstract

A computer-implemented method includes the step of receiving a respective primary entity ID from a user. Each main entity ID includes a first bit string and a second bit string. The method further comprises the step of generating a corresponding mapping ID from each of the main entity IDs, such that: i) each of the mapping IDs is unique; and ii) each of the mapping IDs each includes the first bit string in the primary entity ID and a third bit string different from the second bit string in the primary entity ID.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to Singapore application serial number 10202300253R, filed on February 1, 2023, and titled "METHODS FOR IMPROVING LATENCY AND DATA CONSISTENCY IN LARGE DATABASES FOR BIDIRECTIONAL LOOKUP", the entire content of which is incorporated herein by reference in its entirety. Technical Field

[0003] This application relates to databases, and more particularly, to methods for improving bidirectional lookup latency and data consistency in large databases. Background Art

[0004] Web applications or services may need to create, store, and use bidirectional mappings between associated data elements. For this purpose, for any given data element, associated elements can be derived, and the forward and reverse mappings can be stored in a database table. Scalability issues, latency issues, and data consistency issues arise when the number of records stored in such a database table becomes large or when querying the database from different geographical regions. Traditional methods and tools for solving these problems are insufficient, especially in cases where web services need to support bidirectional mappings of billions of related data elements and global deployment. Summary of the Invention

[0005] The object of the present invention is to provide novel and useful methods for generating mapping IDs from entity IDs, as well as methods and database servers for providing mapping IDs to client devices. For example, the present invention enables a database for bidirectional lookup between entity IDs and mapping IDs, which can be partitioned into a distributed database, thereby avoiding the use of distributed transactions during read and write requests. In addition, the database can still ensure data consistency while having a reduced storage size. The present invention achieves this object by: first, storing an entity ID, an associated mapping ID, and a shard key indicating which partition of the distributed database the row belongs to in each row of the database; and second, generating a mapping ID from the entity ID such that the mapping ID is unique for the entity ID, and either the entity ID or the mapping ID can be used to generate the shard key. Therefore, the present invention can improve the speed of storing new mappings and utilize local database transactions to ensure data atomicity without involving distributed transactions.

[0006] For example, the present invention can also enable a copy of a database to be stored on servers located in different geographical regions, and at the same time prevent data inconsistency when conflicting requests for storing mappings (i.e., entity IDs and associated mapping IDs) are issued simultaneously by client devices located in different regions. The present invention achieves this purpose by initially allowing the received mapping to be stored in the copy on the request-receiving server. Then, the mapping is transmitted to other servers, which attempt to update their copies using the received mapping. If the received mapping conflicts with the mapping already stored on a particular server, the affected server determines the correct mapping by querying the relevant mappings stored on other servers so that the affected server can update its database copy with the correct mapping. In addition, the affected server transmits the correct mapping to other servers so that other servers can correct their copies when needed. The present invention can promote improved data consistency among multiple copies deployed in databases in different geographical regions without increasing the write request latency.

[0007] For example, the present invention can also enable improved consistency in handling lookup requests for users associated with a first service, regardless of the traffic load imposed by users associated with a second service. The present invention can achieve this purpose by using two memories of a cache (common) database, and directing the lookup requests from users associated with the first service to one of the memories, and directing the lookup requests from users associated with the first service to one of the memories. Thus, the present invention can protect users associated with the first server from inconsistent services caused by the (e.g., bursty) traffic load of users associated with the second service.

[0008] For example, the present invention can also enable a reduction in the lookup requests to be executed on a database for duplicate lookup requests, i.e., requests initiated by the same user for the same mapping. The present invention can achieve this purpose by storing the mapping of the request, i.e., the entity ID and the associated mapping ID, on the user device in response to an initial lookup request. The mapping can be stored in the form of an encrypted cookie with a predetermined expiration period. Any subsequent lookup requests are sent using the encrypted cookie so that when the request is received and the cookie is still valid, without searching the database, the cookie can be decrypted and the decrypted mapping can be used to process the lookup request. Thus, the present invention can reduce the read load on the database and improve the availability of the lookup service in the event of a database failure or outage.

[0009] According to a first aspect of the present invention, there is provided a computer-implemented method. The method includes the step of receiving, from a user, a respective master entity ID. Each master entity ID includes a first bit string and a second bit string. The method further includes the step of generating, from each master entity ID among the master entity IDs, a respective mapped ID such that: i) each mapped ID of the mapped IDs is unique, and ii) each mapped ID includes the first bit string of the master entity ID and a third bit string different from the second bit string of the master entity ID.

[0010] The first bit string may be an initial bit sequence of the master entity ID, or any other suitable sequence or set of bits of the master entity ID.

[0011] Optionally, the bit length of the first bit string is N, and the master entity ID and the mapped ID include the first bit string as the N least significant bits. The bit length N of each first bit string may be 10, and each mapped ID may have a bit length of 64 bits.

[0012] Optionally, the method may further include maintaining a counter, wherein the third bit string of the mapped ID includes counter information obtained from the counter.

[0013] Optionally, the method may further include, for each master entity ID, the step of generating a shard key from the master entity ID or the mapped ID using a hash function, wherein the hash function generates the shard key based on the value of the first bit string.

[0014] Optionally, the method may further include, for each master entity ID, the step of storing the master entity ID, the mapped ID, and the shard key in the same row of a database. The database may be configured to include a partitioned database of multiple shards. Each shard may be associated with a different shard key value. Storing the master entity ID, the mapped ID, and the shard key in the same row of the database may include storing the master entity ID, the mapped ID, and the shard key in the same row of the shard associated with the value of the shard key.

[0015] According to a second aspect of the present invention, there is provided a computer-implemented method for resolving conflicts between a first database and a second database. The first database and the second database each have a respective plurality of rows. Each row stores an entity ID and a mapping ID associated with the entity ID. The method comprises the steps of: (a) storing a new entity ID and a new mapping ID associated with the entity ID in a new row of the first database; (b) determining a conflict between the new entity ID and the new mapping ID and an existing entity ID and an associated existing mapping ID stored in the second database; (c) determining a valid mapping ID associated with the new entity ID; and (d) updating the second database based on the new entity ID and the associated valid mapping ID, provided that the determined valid mapping ID is different from the existing mapping ID stored in the second database.

[0016] Optionally, step (b) in the method may include determining that the new entity ID is the same as the existing entity ID and that the new mapping ID is different from the existing mapping ID.

[0017] Optionally, a third database may have an additional plurality of rows, each row storing an entity ID and a mapping ID associated with the entity ID, and step (d) of the method may include retrieving another mapping ID associated with the new mapping ID from the third database and selecting one mapping ID from the new mapping ID, the existing mapping ID stored in the second database, and the another mapping ID retrieved from the third database as the valid mapping ID associated with the new entity ID. The step of selecting the valid mapping ID may include determining a counter value for each of the new mapping ID, the existing mapping ID stored in the second database, and the another mapping ID retrieved from the third database and selecting the mapping ID with the lowest counter value as the valid mapping ID. Alternatively, the first database, the second database, and the third database may further store a creation timestamp in each row, the creation timestamp specifying the time at which the mapped entity ID stored in the row was created, and the step of selecting the valid mapping ID may include selecting the one mapping ID with the earliest timestamp from the new mapping ID, the existing mapping ID stored in the second database, and the another mapping ID retrieved from the third database.

[0018] Optionally, the method may further comprise: step (e): updating the first database based on the new entity ID and the associated valid mapping ID, provided that the determined valid mapping ID is different from the new mapping ID; and step (f): updating the third database based on the new entity ID and the associated valid mapping ID, provided that the determined valid mapping ID is different from the another mapping ID retrieved from the third database.

[0019] Optionally, according to the first aspect of the present invention, the method may further include an initial step of generating a new mapping ID from the new entity ID.

[0020] According to a third aspect of the present invention, there is provided a computer-implemented method for a database server to process requests from a client device for providing a mapping ID associated with a primary entity ID. The database server stores a primary copy of a database having multiple rows. Each row includes one of a plurality of entity IDs and a mapping ID associated with that entity ID. The database server also stores a first cache copy and a second cache copy of the database. The method includes: step (a): receiving a request from a client device, the request including a primary entity ID and service information that specifies whether the request is associated with a first service or a second service. The method further includes, conditional on determining that the service information specifies that the request is associated with the first service: step (b1): determining whether the mapping ID associated with the primary entity ID is stored in the first cache copy; step (b2): conditional on determining that the mapping ID is not stored in the first cache copy, determining whether the mapping ID associated with the primary entity ID is stored in the primary copy of the database; and step (b3): conditional on determining that the mapping ID is stored in the first cache copy or the database primary copy, obtaining the mapping ID and providing the mapping ID to the client device. The method further includes, conditional on determining that the service information specifies that the request is associated with the second service: step (c1): determining whether the mapping ID associated with the primary entity ID is stored in the second cache copy; step (c2): conditional on determining that the mapping ID is not stored in the second cache copy, determining whether the mapping ID associated with the primary entity ID is stored in the primary copy of the database; and step (c3): conditional on determining that the mapping ID is stored in the second cache copy or the database primary copy, obtaining the mapping ID and transmitting the mapping ID to the client device. The method further includes, conditional on determining that the mapping ID is not stored in the database primary copy: step (d1): generating a mapping ID associated with the primary entity ID; step (d2): transmitting the mapping ID to the client device; and step (d3): storing the primary entity ID and the associated mapping ID in a new row of the database.

[0021] Optionally, the first cache copy and the second cache copy may fully cache the database, and the database server may also store a first partial cache copy of the first cache copy and a second partial cache copy of the second cache copy. Step (b1) of the method may include: determining whether a mapping ID associated with the main entity ID is stored in the first partial cache copy, and on the condition that it is determined that the mapping ID is not stored in the first partial cache copy, determining whether the mapping ID associated with the main entity ID is stored in the first cache copy. Step (c1) of the method may include: determining whether a mapping ID associated with the main entity ID is stored in the second partial cache copy, and on the condition that it is determined that the mapping ID is not stored in the second partial cache copy, determining whether the mapping ID associated with the main entity ID is stored in the second cache copy.

[0022] According to a fourth aspect of the present invention, there is provided a computer-implemented method. The method includes: step (a): receiving, from a client device, a first request for providing a mapping ID associated with a main entity ID for an application, the request including the main entity ID; step (b): retrieving, from a database, the mapping ID associated with the main entity ID; step (c): providing the mapping ID to the application; step (d): processing the main entity ID and the mapping ID using an encryption key to generate an encrypted main entity ID and an encrypted mapping ID; and step (e): transmitting the encrypted entity ID, the encrypted mapping ID, and expiration information specifying the expiration period of the encrypted entity ID and the encrypted mapping ID to the client device.

[0023] Optionally, the method may further include: step (f): receiving, from the client device, a second request for providing a mapping ID associated with the main entity ID for the application, the request including the main entity ID, the encrypted entity ID, the encrypted mapping ID, and the expiration information; step (g): determining, based on the expiration information, whether the encrypted main entity ID and the encrypted mapping ID are valid; and step (h): on the condition that it is determined that the encrypted main entity ID and the encrypted mapping ID are valid, decrypting the received encrypted main entity ID and encrypted mapping ID using the encryption key, and providing the decrypted mapping ID to the application.

[0024] According to a fifth aspect of the present invention, there is provided a system, which includes one or more computers and one or more storage devices, and instructions are stored in the storage devices. When the instructions are executed by the one or more computers, the one or more computers execute the first aspect, the second aspect, the third aspect, or the fourth aspect of the present invention.

[0025] According to a sixth aspect of the present invention, there is provided one or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the first, second, third, or fourth aspect of the present invention.

[0026] According to a seventh aspect of the present invention, there is provided a system including a first database server and a second database server. The first database server and the second database server store a first database and a second database, respectively. Each of the databases has a plurality of rows, and each row stores an entity ID and a mapping ID associated with the entity ID. The first database server is configured to store a new entity ID and a new mapping ID associated with the entity ID in a new row of the first database and transmit the new entity ID and the new mapping ID to the second database server. The second database server is configured to determine a conflict between the received new entity ID and new mapping ID and the existing entity ID and associated existing mapping ID stored in the second database. The second database server is further configured to determine a valid mapping ID associated with the new entity ID and, on the condition that the determined valid mapping ID is different from the existing mapping ID stored in the second database, update the second database based on the new entity ID and the associated valid mapping ID.

[0027] The features of the system may be as explained above with respect to the second aspect of the present invention.

[0028] According to an eighth aspect of the present invention, there is provided a database server for processing requests from a client device for providing a mapping ID associated with a primary entity ID. The database server is configured to store a primary copy of a database having a plurality of rows. Each row includes one of a plurality of entity IDs and a mapping ID associated with the entity ID. The database server is further configured to store a first cache copy and a second cache copy of the database. The database server is further configured to receive a request from the client device, the request including a primary entity ID and service information specifying whether the request is associated with a first service or a second service. The database server is configured to: determine whether the mapping ID associated with the primary entity ID is stored in the first cache copy on condition that it is determined that the service information specifies that the request is associated with the first service; and determine whether the mapping ID associated with the primary entity ID is stored in the primary copy of the database on condition that it is determined that the mapping ID is not stored in the first cache copy. The database server is further configured to: obtain the mapping ID and provide the mapping ID to the client device on condition that it is determined that the mapping ID is stored in one of the first cache copy and the primary copy of the database. The database server is further configured to: determine whether the mapping ID associated with the primary entity ID is stored in the second cache copy on condition that it is determined that the service information specifies that the request is associated with the second service; and determine whether the mapping ID associated with the primary entity ID is stored in the primary copy of the database on condition that it is determined that the mapping ID is not stored in the second cache copy. The database server is further configured to: obtain the mapping ID and transmit the mapping ID to the client device on condition that it is determined that the mapping ID is stored in one of the second cache copy and the primary copy of the database. The database server is further configured to: generate a mapping ID associated with the primary entity ID, transmit the mapping ID to the client device, and store the primary entity ID and the associated mapping ID in a new row of the database on condition that it is determined that the mapping ID is not stored in the primary copy of the database.

[0029] The features of the database server may be as explained above with respect to the third aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of an example process for generating a mapping ID from a primary entity ID;

[0031] Figure 2 illustrates Figure 1 an example structure of a mapping ID in;

[0032] Figure 3 shows an example database generated according to the process of Figure 1 ;

[0033] Figure 4is a block diagram of an example computer network;

[0034] Figure 5 is a diagram illustrating an example process of client caching;

[0035] Figure 6 is a flowchart illustrating the relative order guaranteed by data provided by a memory used to store mappings in some embodiments;

[0036] Figure 7 is a flowchart of an example process for providing a mapping ID to a client device;

[0037] Figure 8 is a block diagram illustrating an example process for resolving conflicts between multiple databases;

[0038] Figure 9 is Figure 8 a flowchart of the process of;

[0039] Figure 10 is a diagram illustrating an example process for prewarming a database; and

[0040] Figure 11 shows a computer system that can be used to perform Figure 1 as well as Figures 7 - 10 the method of; DETAILED DESCRIPTION

[0041] Mapping ID Generation

[0042] Generally, a bijective mapping correlates two elements with each other to form a one-to-one correspondence. For example, given multiple entities, each identified by a corresponding entity ID, a related mapping ID can be derived for each entity ID such that each mapping ID is unique for the associated entity ID. To facilitate the ability to perform a two-way lookup, i.e., to look up the mapping ID associated with any particular entity ID and vice versa, the mapping can be stored in a database. For example, the database can include a first column "Entity ID" and a second field "Mapping ID", and each row is populated with an entity ID and the associated mapping ID. For a given entity ID, the associated mapping ID can be looked up in the database by finding the row in the "Entity ID" column that contains the entity ID. Similarly, for a given mapping ID, the associated entity ID can be looked up in the database by finding the row in the "Mapping ID" column that contains the mapping ID. However, this way of storing mappings may not necessarily be suitable for large databases. For many applications, a database can typically contain at most 100 million records before the read / write latency becomes too high. However, some applications may need to access billions of mappings.

[0043] Known techniques for improving the performance and scalability of large databases involve partitioning the database (also known as sharding) into smaller partitions (or shards). This enables the distribution of the database across multiple servers, computing systems, clusters, and so on. One way to partition a database is called horizontal sharding, where the partitions have the same schema as the original database, and each row of the original database is assigned to a specific partition based on a sharding key. The sharding key for each row can be derived from elements in a particular column of that row. However, horizontal sharding of the example databases mentioned above will no longer allow for efficient two-way lookups. For example, when the sharding key is derived from elements in the "entity ID" column, for a given entity ID, it is still possible to efficiently look up the associated mapping ID because the database partition storing the relevant row can be derived from the entity ID. However, the reverse lookup is no longer efficient because for a given mapping ID, the database partition storing the relevant row cannot be identified.

[0044] To support two-way lookups, the way of storing the mappings mentioned above can be modified such that each mapping can be stored in two rows in the database. In one row, the entity ID is stored in the first column and the mapping ID is stored in the second column. In the other row, the mapping ID is stored in the first column and the entity ID is stored in the second column. The sharding key can be derived from the elements in the first column, and the rows can be partitioned accordingly. Efficient two-way lookups can now be achieved because for a given entity ID, the partition containing the row with the entity ID in the first column and the mapping ID in the second column can be identified. Similarly, for a given mapping ID, the partition containing the row with the mapping ID in the first column and the entity ID in the second column can be identified. Although this modified storage scheme enables two-way lookups, since the entity ID and the mapping ID are generally independent of each other, they will generate different sharding key values and will be stored in different partitions. Therefore, writing such a mapping to the database can result in a distributed transaction, i.e., accessing partitions on different servers, and may translate into additional latency.

[0045] Reference Figures 1 - 3 , describes an example process for generating a mapping ID from a given entity ID for two-way lookups and overcomes the drawbacks of the methods mentioned above. Figure 1 Illustrates the steps of the example process. Figure 2 Illustrates an example structure of mapping ID1, which illustrates Figure 1 the steps in the method of Figure 3 Illustrates based on Figure 1 the process in

[0046] Reference Figure 1, in step S101, a main entity ID is received from a user. The main entity ID is one of a plurality of entity IDs and includes a first bit string and a second bit string. In step S102, a mapping ID is generated from the main entity ID. The mapping ID is generated such that: i) the mapping ID is unique for the main entity ID; and ii) the mapping ID includes the first bit string in the main entity ID and another bit string different from the second bit string in the main entity ID.

[0047] In an embodiment, the first bit string has a bit length of N and corresponds to the N least significant bits of both the main entity ID and the mapping ID, where N is an integer greater than zero. In an embodiment, the mapping ID can be generated such that the values of the N least significant bits are the same as the values of the N least significant bits in the main entity ID.

[0048] In Figure 2 In the illustrated embodiment, the mapping ID 1 has a bit length of 64 bits. In this embodiment, the length of the first bit string of the main entity ID included in the mapping ID 1 is 10 bits. Further, in Figure 2 In the illustrated embodiment, the length of the another bit string can be 54 bits.

[0049] In an embodiment, the mapping ID is generated such that the mapping ID is unique for the main entity ID by including a counter category value in the another bit string included in the mapping ID. The counter category value can be provided by a counter module that stores and manages the counter category values. The counter module can be configured to change the counter category value each time a new mapping ID is generated. Each time a new mapping ID is generated, the counter category value can increase monotonically. In an embodiment, the counter module is configured such that any counter category value is provided only once for generating a mapping ID, i.e., the provided counter category value can be globally unique. Thus, a first mapping ID including a counter category value among counter category values lower than a second mapping ID can indicate that the first mapping ID was created at an earlier time point. In Figure 2 In the illustrated embodiment, the counter category value is included in the mapping ID as a bit string of length 43. This supports generating at least 2 43 unique mapping IDs.

[0050] In an embodiment, the mapping ID can also be generated to encode advanced information, such as service information, in the another bit string in the mapping ID. In Figure 2 In the illustrated embodiment, the service information (biz_id) is encoded in a bit string of length 4, and the counter source information is encoded in a single bit of the mapping ID. The counter source information can specify the counter module.

[0051] Return to reference Figure 1 , in step S103, a hash function is used to generate a shard key from the primary entity ID or the mapped ID, where the hash function generates the shard key based on the value of the first bit string. The hash function can be a computational method that receives an input and generates a shard key value as output, and is also simply referred to as the shard key. The shard key can represent an integer. In an embodiment, where the bit length of the first bit string is N and corresponds to the N least significant bits of the primary entity ID and the mapped ID, the hash function can be configured to generate a shard key based on the N least significant bits of the received input. In this embodiment, regardless of whether the received input is the primary entity ID or the mapped ID, the hash function generates the same shard key value. Additionally, in this embodiment, the shard key value represents an integer from 0 to (2 N -1).

[0052] In step S104, the primary entity ID, the mapped ID, and the shard key are stored in the same row of the database. The database can be a relational database. The database can be stored on a database server. The database server can be a distributed computing system or a server cluster.

[0053] In an embodiment, the database can be configured to include a partitioned database with multiple shards. Each shard can be associated with a different shard key value. In this case, the database server can be a server cluster, and each shard in the database can be stored on a different server. In this embodiment, in step S104, the primary entity ID, the mapped ID, and the shard key can be stored in the same row of the shard associated with the shard key value.

[0054] In an embodiment, for a large number of entity IDs, steps S101 to S104 in Figure 1 can be repeated such that the database includes multiple rows, where each row includes an entity ID, an associated mapped ID, and a shard key.

[0055] Reference Figure 1 The advantages of the method described will be described by way of example with reference to Figure 3 Description. Figure 3 FIG. shows database 30, which can be obtained by repeatedly executing Figure 1Generated by steps S101 to S104 therein. The database 30 includes a first field 32 "entity ID", a second field "mapping ID", and a third field "sharding key". In an embodiment, the database 30 may further include other fields, such as a fourth field 38 "creation time", the elements of which specify the time when the mapping stored in the corresponding row was created. In this embodiment, the database 30 has X records, where X is a positive integer. Horizontal sharding can be applied to the database 30 to divide the database 30 into multiple partitions (commonly referred to as "shards"). Figure 3 Shows four of them, partitions 40-0, 40-1, 40-2, and 40-1023. In Figure 3 The example shown, the partition 40-0 is associated with the sharding key value zero. The partition 40-0 includes two rows, which correspond to the two rows in the database 30 that have a sharding key value equal to zero.

[0056] Further referring to Figure 3 , the partitioned database with partitions 40-0 to 40-1023 enables efficient two-way lookup between the entity ID and the associated mapping ID, because the relevant partition of the database can be identified from the entity ID or the mapping ID using a hash function. For example, for any given entity ID, such as Figure 3 "entity ID 3" in, a hash function can be used to identify the partition in the database that stores the relevant row (i.e., the row including the given entity ID and the associated mapping ID). In this example, the sharding key value obtained by applying the hash function to "entity ID 3" is 1. The row including "entity ID 3" and the associated mapping ID "mapping ID 3" can be found in the corresponding partition 40-1. Similarly, if a reverse mapping is desired, the mapping ID "mapping ID 3" can be provided to the hash function. The hash function can again generate a sharding key value equal to 1, because the mapping ID was created in S102 such that it includes the first bit string of the associated entity ID, and the hash function generates the sharding key value based on the value of the first bit string. Therefore, the mapping ID generated in S102 will deterministically generate the same sharding key value as the associated entity ID. In addition, since each mapping is stored in a single row, the required database storage size is half compared to the implementation of the two rows mentioned above. Additionally, since only one row is stored, the write speed when storing the mapping is increased, and local database transactions can be utilized to ensure data atomicity without involving distributed transactions. Furthermore, these mappings can be effectively stored in the same shards for both SQL and NoSQL data stores.

[0057] Caching strategy

[0058] Figure 4A computer network is shown that includes a first database server 50 connected to a client device 52 via a network. This computer network enables the client device 52 to transmit data to and receive data from the first database server 50. The database server may store a database 54 configured to store multiple rows, where each row includes an entity ID, an associated mapping ID, and a shard key. The database 54 may be configured as a partitioned database including multiple shards. The first database server 50 may be configured to execute the method described with reference to Figure 1 which will be described in detail below. The computer network may also include a second database server 61 and a third database server 67, which will be described in detail below.

[0059] The first database server 50 may also be configured to provide a service to the client device 52 that enables the client device 52 to perform a two-way lookup of entity IDs and associated mapping IDs stored in the database 54. The database server 50 may include an interface module 56 configured to receive data from and send data to the client device 52 via the network. Broadly speaking, the client device may transmit a request including an entity ID to the interface module 56 of the first database server 50 to receive the mapping ID associated with the transmitted entity ID from the first database server 50. Similarly, the client device may transmit a request including a mapping ID to the interface module 56 of the first database server 50 to receive the entity ID associated with the transmitted mapping ID from the first database server 50.

[0060] The client device 52 may be a computer system as described below with reference to Figure 11 which will be described. The first database server 50 may be implemented in a variety of ways, such as on a single server, on a distributed computing system, on a server cluster or similar system, on a cloud-based system, etc. When the first database server 50 is implemented on a distributed computing system or server cluster, each shard of the partitioned database may be stored on a different server. The network may be the Internet or other network capable of connecting the client device 52 and the first database server 50 together.

[0061] A user with access rights to an entity ID can run a local application on their client device. The local application can be configured to use a web application. The web application can receive a request including a mapping ID as input and can reply to the request with a response including the mapping ID. To be able to use the web application, the local application can obtain the mapping ID associated with the entity ID by transmitting a corresponding request to an interface module 56 in a first database server 50. In a typical usage scenario, the local application may transmit multiple lookup requests for the same entity ID within a short period of time, resulting in a high traffic load in the first database server 50.

[0062] Reference Figure 5 , the process of reducing the number of redundant lookup requests sent to the interface module 56 by caching previously looked-up mappings on the client side is described below. In this embodiment, the local application 70 can transmit a request for the web application 72 to a service application 74 that can run on a gateway server. The service application 74 can extract the entity ID from the received request and transmit a request for the associated mapping ID to the interface module 56 of the first database server 50. After receiving the mapping ID, the service application 74 can replace the entity ID included in the request received from the local application with the mapping ID. The service application 74 can also temporarily store the entity ID and the mapping ID as a key / value pair in local memory. When the local application 70 sends a subsequent request including the same entity ID, the service application 74 can use the mapping ID stored in local memory without transmitting a request to the first database server 50. This reduces the traffic load on the database server. Similarly, the service application 74 can replace the mapping ID in any response from the web application 72 with the associated entity ID by retrieving the entity ID from the first database server 50 or, if available, using the entity ID stored in local memory.

[0063] In an embodiment, reference Figure 5 The process described above can also include an automatic cookie cache as described below to further reduce the traffic load caused by redundant lookup requests and to provide a fault-tolerant setting for disaster recovery. Generally, it is desirable to provide a process such that: even if the database of its database server 50 is temporarily unavailable (e.g., due to downtime or extreme failure), the two-way lookup service provided by the database server 50 remains available.

[0064] As described above, the service application 74 can extract the entity ID from the request transmitted by the local application 70 and transmit a request for providing an associated mapping ID to the interface module 56 of the first database server 50. After receiving the mapping ID, the service application 74 can replace the entity ID included in the request received from the local application with the mapping ID. The service application 74 can also temporarily store the entity ID and the mapping ID as a key-value pair in the local memory. In this embodiment, the service application 74 can also encrypt the entity ID and the mapping ID using an encryption key stored on the user device. When the service application 74 receives a response from the Web application 72, the service application 74 can use the entity ID stored in the key-value pair to replace the mapping ID with the associated entity ID and transmit the response together with the encrypted entity ID and mapping ID to the local application 70. The service application 74 can also transmit expiration information specifying the validity period of the encrypted entity ID and the encrypted mapping ID. The service application 74 can transmit the encrypted entity ID and the mapping ID in the form of an HTTP Web Cookie. The Cookie can include a time-to-live (TTL) value as the expiration information, which is used to describe the period during which the Cookie is valid. In response to the received Cookie, the local application 70 can store the Cookie on the client device.

[0065] Any subsequent request transmitted by the local application 70 can include the encrypted Cookie. In response to the received request, the service application 74 can determine whether the Cookie is valid by comparing the TTL value of the Cookie with the current time. If it is determined that the Cookie is invalid, the service application 74 can process the request in the manner described above, i.e., as if the request does not include a Cookie. If it is determined that the Cookie is valid, then the service application 74 can decrypt the Cookie using the encryption key to obtain the decrypted entity ID and mapping ID. Then, the service application 74 can use the decrypted entity ID and mapping ID to process the received request. Thus, this operation enables the processing of a lookup request without transmitting a lookup request to the database server. Therefore, this operation further reduces the business load of the database server and also enables the availability of the lookup service to be maintained when the database server is temporarily unavailable.

[0066] Returning to the reference Figure 4 the first database server 50 may also include a first memory 58 configured to cache the database 54. In an embodiment, the first memory 58 is a key-value storage device. For example, the first memory 58 can cache the entire database 54. As described with respect to Figure 5More specifically, in this case, the lookup requests received by the interface module 56 can be directed to the first memory 58 instead of the database 54. This can reduce the lookup request latency experienced by the client device 52 because the database search cached in the first memory 58 can be faster than the database 54. The first database server 50 can also include a second memory 60 configured to partially cache the database 54. In this case, the second memory 60 can store only a subset of the records stored in the first memory. Additionally, in this embodiment, the lookup requests received by the interface module 56 can be directed to the second memory 60 instead of the first memory 58 or the database 54. This can further reduce the lookup request latency experienced by the client device 52 because the partial database search cached in the second memory 60 can be faster than the first memory 58 or the database 54.

[0067] In other words, Figure 4 the first database server 50 in provides a three - tier access layer. The first tier includes the second memory 60 and can provide fast access. The storage size associated with the first tier can be less than the size of the entire database 54. The second tier includes the first memory 60 and can provide slower access than the first tier and faster access than the third tier. The size of the storage device associated with the second tier can be greater than the size of the storage device of the first tier. The third tier includes the database 54 and can provide slower access than either of the first and second tiers. The third tier can store data permanently and / or persistently.

[0068] By adopting the caching strategies described hereinafter with respect to Figure 6 and Figure 7 the two - way lookup service can reduce latency and increase the number of user requests it can handle at any time. Figure 6 illustrates the relative order of data guarantees provided by the memories used to store mappings in some embodiments. Figure 7 Shows a flowchart of an example method of providing a mapping ID associated with an entity ID to an application running on a client device. In step S501, the client device can transmit a request including the mapping ID. The request can be processed by the interface module 56 of the first database server 50. In step S502, the interface module 56 can determine whether the mapping ID is stored in the second memory 60. If so, the interface module 56 can retrieve the mapping ID from the second memory 60 and can provide it to the application (S502). If the mapping ID is not stored in the second memory 60, the interface module 56 can determine in step S503 whether the mapping ID is stored in the first memory 58. If so, the interface module 56 can retrieve the mapping ID from the first memory 58 and can provide it to the application (S502).

[0069] If the mapping ID is not stored in the first memory 58 or the second memory 60, the interface module 56 may determine whether the mapping ID is stored in the database 54 (S505). If so, the interface module 56 may retrieve the mapping ID from the database 54. In step S507, the first memory 58 and the second memory 60 may be updated to store the entity ID and the retrieved mapping ID. The interface module 56 may provide the mapping ID to the application (S502).

[0070] If the mapping ID is not stored in the database 54, the database server may generate a mapping ID from the transmitted entity ID in step S506 according to the method described in the reference Figure 1 and store the entity ID and the generated mapping ID in the database 54. The first memory 58 and the second memory 60 may be updated to store the entity ID and the generated mapping ID (S507). The interface module 56 may provide the mapping ID to the application (S502).

[0071] Generally speaking, each user in the bidirectional mapping service may be associated with one of multiple services. Users associated with the same service may belong to an organization, a business unit, or a similar institution. The business load caused by users associated with the same service may be similar, and may be different from the business load caused by users associated with another service. For example, a user may be associated with a first service or a second service. Users associated with the first service may cause a consistent business load, while users associated with the second service may cause a highly fluctuating business load. During periods of high business load caused by users of the second service, users of the first service may experience higher latency.

[0072] As described in the reference below Figure 4 users of the first service may be protected from the negative impact of such bursty traffic. Figure 4 The second database server 61 shown in is a variant of the first database server 50. Similar to the first database server 50, the second database server 61 may transmit data to and receive data from the client device 62 via a network. In addition, the database server may store a database 63 similar to the database 54, that is, configured to store multiple rows, where each row includes an entity ID, an associated mapping ID, and a shard key. The database 63 may be configured to be a partitioned database including multiple shards. The second database server 61 may be configured to execute the method described in the reference Figure 1

[0073] ​The second database server 61 may include two interface modules 64-1, 64-2, two first memories 65-1, 65-2, and two second memories 66-1, 66-2. The first memories 65-1, 65-2 may each be configured to independently cache the database 63. The first memory 65-1 may be configured to partially cache the second memory 66-1. The first memory 65-2 may be configured to partially cache the second memory 66-2.

[0074] The client device may be used to transmit a two-way lookup request to the database server 63. The request may include an entity ID and service information. The service information may specify which service the user is associated with. If the service information specifies that the user is associated with the first service, the interface module 64-1 may process the request. In this case, similar to Figure 7 Steps S502 to S507 in, the interface module 64-1 may determine whether the mapping ID is stored in the second memory 66-1. If so, the interface module 64-1 may retrieve the mapping ID from the second memory 66-1 and provide the mapping ID to the user device. If the mapping ID is not stored in the second memory 66-1, the interface module 64-1 may determine whether the mapping ID is stored in the first memory 66-1. If so, the interface module 64-1 may retrieve the mapping ID from the first memory 66-1 and provide the mapping ID to the user device. If the mapping ID is not stored in the first memory or the second memories 65-1, 66-1, the interface module 64-1 may determine whether the mapping ID is stored in the database 63. If so, the interface module 64-1 may retrieve the mapping ID from the database 63. The first memories and the second memories 65-1, 65-2, 66-1, 66-2 may be updated to store the entity ID and the retrieved mapping ID. The interface module 64-1 may provide the mapping ID to the user device.

[0075] If the mapping ID is not stored in the database 63, the database server may generate a mapping ID from the transmitted entity ID according to the method described in the reference Figure 1 and store the entity ID and the generated mapping ID in the database 63. The first memories and the second memories 65-1, 65-2, 66-1, 66-2 may be updated to store the entity ID and the generated mapping ID. The interface module 64-1 may provide the mapping ID to the user device.

[0076] If the service information designates that the user is associated with the second service, the interface module 64-2 can process the request. In this case, the interface module 64-2 can perform the steps mentioned above, that is, the interface module 64-2 can look up the mapping ID in the second memory 66-2 and, if necessary, look up in the first memory 65-2 and the database 63. Therefore, in this embodiment, the lookup request from a user associated with the first service is not performed on the first memory and the second memory 65-2, 66-2. Similarly, the request from a user associated with the second service is not performed on the first memory and the second memory 65-1, 66-1. Since the first memory and the second memory cache the database, most of the lookup requests can be satisfied without accessing the database 63. Therefore, in this embodiment, the traffic congestion caused by the users of the second service does not affect the latency of the user requests of the first service, because most of these requests are satisfied by accessing the first memory and the second memory 65-1, 66-1, and these memories do not receive any traffic from the users of the second service. Since all write operations are performed on the (common) database 63, data consistency is guaranteed, that is, no mapping conflicting with any other mapping is stored in the first memory and the second memories 65-1, 65-2, 66-1, 66-2.

[0077] Multi-region conflict resolution

[0078] Generally speaking, compared with users located closer to the database server, users of the two-way mapping service located in a geographical location different from the location of the database server may experience additional latency. This additional latency may be dozens or hundreds of milliseconds. By providing a database server in each geographical region where the users are located and directing the mapping requests to the nearest database server, it is desirable to reduce the latency of the lookup requests. In this case, each of the database servers in the database servers can store a local copy of the database containing the two-way mapping. As described above with reference to Figure 5 As described, the lookup request can result in the generation and storage of a new mapping ID, for example, when the requested mapping is not stored in the database. In order to also keep the latency of such write requests low, a "master-to-master" implementation may be more appropriate. In this master-to-master implementation, each database server can be allowed to generate and store a new mapping in its own database copy. After storing the new mapping, other database copies need to be updated. However, if two users from different geographical regions simultaneously transmit requests to two different database servers based on the same entity ID, a data conflict can occur. In this case, the two database servers will generate two different mapping IDs from the same entity ID. In order to avoid problems such as data consistency and data conflicts, a method capable of identifying and correcting such conflicts needs to be provided.

[0079] Reference Figure 4 、 Figure 8 and Figure 9 describe a method for resolving data conflicts between different databases. Figure 8 is a block diagram illustrating the architecture and data flow of the method. Figure 9 is a flowchart of the steps in the method.

[0080] In an embodiment, the computer network may include a first database server 50, a second database server 61, and a third database server 67, as Figure 4 shown and as described above. In this example, the third database server 67 is configured similarly to the first database server 50, i.e., having a first memory and a second memory, while the second database server, as described above, is configured to include two first memories and a second memory. In some embodiments, any one of the first database server, the second database server, and the third database server may be configured in any manner, i.e., including one or two first memories and a second memory.

[0081] In an embodiment, each of the first database server, the second database server, and the third database server may be located in a different geographical region, may be configured to host the bidirectional lookup service as described above, and may be configured to receive lookup requests from the corresponding client devices 52, 62, 68, as Figure 4 shown. In this embodiment, the databases 54, 63, 69 may initially include the same data, i.e., the same mapping.

[0082] Figure 9 is a flowchart of an example method for resolving conflicts between the databases 54, 63, 69. In step S901, the entity ID and the mapping ID associated with the entity ID are stored in a new row of the database 52 of the first database server 50. For example, as referenced above Figure 7 described, the entity ID and the mapping ID may be generated and stored in response to a lookup request from the client device 52. This means that the entity ID and the mapping ID may be stored in response to determining that the mapping ID associated with the entity ID is stored in the database 52.

[0083] In step S902, the change to database 52, i.e., the new row including the entity ID and the generated mapping ID, can be transmitted to the second database server 61 and the third database server 67. In step S903, in response to receiving the entity ID and the mapping ID, each of the second database server 61 and the third database server 67 attempts to update their respective databases 63, 69. In other words, the second database server 61 and the third database server 67 attempt to add the received entity ID and mapping ID to their respective databases 63, 69. The second database server 61 may have a data conflict between the received entity ID and mapping ID and another entity ID and the associated existing mapping ID stored in the second database 63. For example, the second database server 61 may determine that the received entity ID is the same as the existing entity ID, and the received mapping ID is different from the existing mapping ID. In this case, adding the received data to the new row in database 63 may result in a primary key / index guarantee failure.

[0084] In step S904, the second database server 61 determines which mapping ID should be associated with the received entity ID. In other words, the second database server 61 determines the valid mapping ID for the received entity ID. The second database server 61 may write the conflict data specifying the data conflict in the data conflict log file 80 stored in the second database server 61. The data conflict log file 80 may be implemented as a Kafka topic. In an embodiment, the data conflict log file 80 may be mirrored to the first database server 50 and the third database server 67.

[0085] The checker module 82 is an application running on each of the first database server 50, the second database server 61, and the third database server 67, and can be configured to read the data conflict log file 80. When the data conflict log file 80 is implemented as a Kafka topic, the checker module 82 can be implemented as a consumer subscribing to the topic. The checker module 82 running on the second database server 61 can determine the valid mapping ID. For this purpose, the checker module 82 may transmit a lookup request including the entity ID to the third database server 67, and receive, from the response from the third database server 67, the mapping ID associated with the transmitted entity ID stored in the database 69 as another mapping ID. Then, the checker module 82 running on the second database server 61 can select one mapping ID from the mapping ID initially received from the first database server, the existing mapping ID stored in the second database 63, and the other mapping ID received from the third database server 67 as the valid mapping ID.

[0086] In an embodiment, the checker module 82 may select a valid mapping ID by determining a counter value for each of the mapping IDs initially received from the first database server, the existing mapping IDs stored in the second database 63, and another mapping ID received from the third database server 67, and select the mapping ID with the lowest counter value as the valid mapping ID. Alternatively, if, as described above with reference to Figure 2 and Figure 3 the first database 52, the second database 63, and the third database 69 also store a creation timestamp in each row that specifies the creation time of the mapping entity ID stored in the row, in step S902, when the first database server 50 transmits the entity ID and the mapping ID to the second database server 61, the corresponding creation timestamp may also be transmitted. Further, in this embodiment, when the third database server 67 transmits another mapping ID to the second database server 61, the corresponding creation timestamp may also be transmitted. In this case, the checker module 82 selects the mapping ID with the earliest timestamp as the valid mapping ID from the mapping ID initially received from the first database server 50, the existing mapping IDs stored in the second database 63, and another mapping ID received from the third database server 67.

[0087] In step S905, the second database server 61 may write the valid mapping ID to a data repair log file 84 stored in the second database server 61. The data repair log file 84 may be implemented as a Kafka topic.

[0088] The repairer module 86 is an application running on the first database server 50, the second database server 61, and the third database server 67 and may be configured to read the data repair log file 84. When the data repair log file 84 is implemented as a Kafka topic, the repairer module 86 may be implemented as a consumer subscribing to the topic. In this case, when the checker module 82 determines in step S904 that the valid mapping ID is not stored in one of the databases 63, the repairer module 86 running on the second database server 61 may update the database 63 in response to reading the data repair log 84 such that the entity ID is stored together with the valid mapping ID and the previously stored corresponding mapping ID is removed.

[0089] In step S906, the entity ID and the valid mapping ID can be transmitted to the first database server 50 and the third database server 67. For this purpose, the data repair log file 84 can be mirrored to the first database server 50 and the third database server 67. In step S907, the first database server 50 and the third database server 67 can update the corresponding first database 52 and third database 69 based on the entity ID and the associated valid mapping ID. Depending on whether the first database 52 and the third database 69 store a valid mapping ID, the corresponding repairer modules 86 running on the first database server 50 and the third database server 67 can, in response to reading the mirrored data repair log, update the databases 52 and 69 such that the entity ID is stored together with the valid mapping ID and, where applicable, remove the previously stored corresponding mapping ID. This enables any data conflicts that may be caused by multiple (near) simultaneous lookup requests sent to database servers in different geographical regions to be resolved.

[0090] Warm-up architecture

[0091] Generally speaking, the two-way lookup service described above can be used for a large number of entity IDs, such as billions. Although each mapping ID may only be generated when the first corresponding user request is sent to the database server, it is often desirable to pre-populate (commonly referred to as warm up) the database. This is because the write speed is generally lower than the read speed, and if the database server receives a large number of write requests (e.g., during peak hours), users may experience significant latency. Therefore, it is desirable to provide a process for continuously pre-populating the database with the most relevant mappings. This is described below with reference to Figure 10 Describe this process.

[0092] In an embodiment, a large number of entity IDs can be stored in the data warehouse 90. The data warehouse 90 can be stored on a cluster or server connected to the network. In an embodiment, the data warehouse can be stored in the database server 50. In another embodiment, the data warehouse 90 is stored on a separate cluster or server. The number of stored entity IDs can exceed one billion. In the data warehouse 90, each of the stored entity IDs can be associated with a timestamp indicating the last time the entity ID was used, e.g., in a web application.

[0093] A selector module 92 running on a cluster in a data warehouse 90 can be configured to query the data warehouse for entity IDs whose timestamps fall within a selection period. The selection period can be the past three months. The selector module 92 can write the received entity IDs to a selection log file 94. The selection log file 94 can be implemented as a Kafka topic. In an embodiment, the selection log file 94 can be mirrored to a database server 50.

[0094] A warm-up module 96 is an application running on the database server 50, and the warm-up module 96 can be configured to read the selection log file 94. When the data conflict log file 80 is implemented as a Kafka topic, the warm-up module 96 can be implemented as a consumer subscribing to this topic. In response to reading the selection log file 94, the warm-up module 96 can transmit a lookup request for each of the entity IDs in the selection log file 94 to the interface module 56 of the database server 50, thereby causing the database server 50 to generate and store the associated mapping IDs in the database 54, as described above with reference to Figure 1 and Figure 7 described. In an embodiment, the warm-up module 96 can also be configured to receive recently generated entity IDs, such as the previous day, for example, by a web application. In response to receiving these recently generated entity IDs, the warm-up module 96 can transmit a lookup request for each of these entity IDs to the interface module 56 of the database server 50, thereby causing the database server 50 to generate and store the associated mapping IDs in the database 54. This enables improving the stability of the two-way lookup service and reducing latency caused by a large number of write requests, because the database is pre-populated with entity IDs that have been recently used or generated and are thus most likely to be requested by users in the near future. Although this process is described with reference to the database 50, those skilled in the art will understand that various modifications can be made to the embodiments described above (such as applying this process to all database servers in a network) without departing from the scope of the present invention.

[0095] Figure 11 is a block diagram showing the technical architecture 500 of a server that can execute according to Figure 1 、 Figure 2 、 Figure 5 、 Figure 6 、 Figure 9 or Figure 10Some or all of the methods. The technical architecture includes a processor 522 (also known as a central processing unit or CPU), which communicates with memory devices including auxiliary storage devices 524 (such as disk drives), read-only memory (ROM) 526, and random access memory (RAM) 528. The processor 522 can be implemented as one or more CPU chips. The technical architecture may also include input / output (I / O) devices 530 and network connection devices 532.

[0096] The auxiliary storage device 524 generally includes one or more disk drives or tape drives and is used for non-volatile storage of data and, if the capacity of the RAM 528 is insufficient to hold all working data, as an overflow data storage device. The auxiliary storage device 524 can be used to store programs that, when selected for execution, are loaded into the RAM 528.

[0097] In this embodiment, the auxiliary storage device 524 has an order processing component 524a, which includes non-transitory instructions executed by the processor 522 to perform various operations of the methods of the present disclosure. The ROM 526 is used to store instructions and data read during program execution. In some contexts, the auxiliary storage device 524, the RAM 528, and / or the ROM 526 may be referred to as computer-readable storage media and / or non-transitory computer-readable media.

[0098] The I / O device 530 can include printers, video monitors, liquid crystal displays (LCDs), plasma displays, touchscreen displays, keyboards, keypads, switches, knobs, mice, trackballs, voice recognizers, card readers, paper tape readers, or other known input devices.

[0099] The processor 522 executes instructions, code, computer programs, scripts accessed from a hard disk, floppy disk, optical disk (these disk-based systems are considered auxiliary storage devices 524), flash drive, ROM 526, RAM 528, or network connection device 532. Although only one processor 522 is shown, multiple processors may be presented. Thus, although instructions may be described as being executed by a processor, the instructions may be executed by one or more processors simultaneously, serially, or in other execution manners.

[0100] Although the technical architecture is described with reference to a computer, it should be understood that the technical architecture may consist of two or more computers that communicate with each other and cooperate to perform tasks. For example, but not by way of limitation, an application may be partitioned in this way to allow for concurrent and / or parallel processing of the application's instructions. Alternatively, the data processed by the application may be partitioned in this way to allow different parts of the data set to be processed concurrently and / or in parallel by two or more computers. In an embodiment, the technical architecture 500 may employ virtualization software to provide the functionality of multiple servers in the technical architecture 500 that do not directly limit the number of computers. In an embodiment, the functions disclosed above may be performed by executing an application and / or the application may be provided in a cloud computing environment. Cloud computing may include providing computing services via a network connection using dynamically scalable computing resources. The cloud computing environment may be established by an enterprise and / or may be leased from a third-party provider as needed.

[0101] By programming and / or loading executable instructions onto the technical architecture, at least one of the CPU 522, RAM 528, and ROM 526 is changed, causing the technical architecture to be partially transformed into a special-purpose machine or device with the novel functions taught by the present disclosure. Implementing functions by loading executable software onto a computer can be transformed into a hardware implementation through well-known design rules, which are fundamental in the fields of electronic engineering and software engineering.

[0102] While the foregoing description has described exemplary embodiments, those skilled in the art will understand that many variations can be made to the embodiments within the scope and spirit of the present invention.

Claims

1. A computer-implemented method comprising: a) receiving corresponding master entity IDs from the user, each master entity ID comprising a first bit string and a second bit string; as well as b) generating a corresponding mapping ID from each of the primary entity IDs, such that i) each of the mapping IDs is unique, and ii) The mapping IDs each include the first bit string of an associated master entity ID and a third bit string different from the second bit string in the associated master entity ID. 2 . The method of claim 1 , wherein a bit length of the first bit string is N, and the master entity ID and the mapping ID include the first bit string as N least significant bits. 3 . The method according to claim 1 , wherein the bit length N of each first bit string is 10, and the bit length of each mapping ID is 64.

4. The method of any preceding claim, further comprising maintaining a counter, wherein the third bit string of the mapping ID comprises counter information obtained from the counter.

5. The method according to any of the preceding claims further includes a step of generating a shard key from the master entity ID or the associated mapping ID using a hash function for each master entity ID, wherein the hash function generates the shard key based on the value of the first bit string.

6. The method according to claim 5 further includes the step of storing, for each primary entity ID, the primary entity ID, the associated mapping ID and the shard key in the same row of a database.

7. The method of claim 6, wherein the database is configured as a partitioned database comprising a plurality of shards, each shard being associated with a different shard key value, and wherein storing the primary entity ID, the mapping ID, and the shard key in the same row of the database comprises: The primary entity ID, the mapping ID, and the shard key are stored in a same row of a shard associated with the value of the shard key.

8. A computer-implemented method for resolving conflicts between a first database and a second database, the first database and the second database each having a plurality of rows, each row storing an entity ID and a mapping ID associated with the entity ID, the method comprising: (a) storing a new entity ID and a new mapping ID associated with the entity ID in a new row of the first database; (b) determining a conflict between the new entity ID and the new mapping ID and an existing entity ID and an associated existing mapping ID stored in the second database; (c) determining a valid mapping ID associated with the new entity ID, and (d) updating the second database based on the new entity ID and the associated valid mapping ID, conditional on determining that the valid mapping ID is different from the existing mapping ID stored in the second database.

9. The method of claim 8, wherein step (b) comprises determining that the new entity ID is the same as the existing entity ID, and determining that the new mapping ID is different from the existing mapping ID.

10. The method according to claim 8 or 9, wherein the third database has another plurality of rows, each row storing an entity ID and a mapping ID associated with the entity ID, and step (d) comprises: retrieving from the third database another mapping ID associated with the new mapping ID, and One of the new mapping ID, the existing mapping ID stored in the second database, and the other mapping ID retrieved from the third database is selected as the effective mapping ID associated with the new entity ID.

11. The method of claim 10, wherein the step of selecting the valid mapping ID comprises: A counter value is determined for each of the new mapping ID, the existing mapping ID stored in the second database, and the another mapping ID retrieved from the third database, and the mapping ID having the lowest counter value is selected as the valid mapping ID.

12. The method of claim 10, wherein the first database, the second database, and the third database further store a creation timestamp in each row, the creation timestamp specifying the time when the mapping entity ID stored in the row was created, and the step of selecting the valid mapping ID comprises: A mapping ID having an earliest timestamp is selected from the new mapping ID, the existing mapping ID stored in the second database, and the another mapping ID retrieved from the third database.

13. The method according to any one of claims 8 to 12, further comprising: (e) updating the first database based on the new entity ID and the associated valid mapping ID, conditional on determining that the valid mapping ID and the new mapping ID are different; as well as (f) updating the third database based on the new entity ID and the associated valid mapping ID, conditional on determining that the valid mapping ID and the other mapping ID retrieved from the third database are different.

14. The method according to any one of claims 8 to 13, further comprising an initial step of generating the new mapping ID from the new entity ID according to the method according to any one of claims 1 to 7.

15. A computer-implemented method of processing, by a database server, a request from a client device for providing a mapping ID associated with a primary entity ID, the database server storing a master copy of a database, the master copy of the database having a plurality of rows, each row including an entity ID from a plurality of entity IDs and a mapping ID associated with the entity ID, and the database server further storing a first cached copy and a second cached copy of the database, the method comprising: (a) receiving the request from the client device, the request comprising the host entity ID and service information specifying whether the request is associated with a first service or a second service; (b) conditional on determining that the service information specifies that the request is associated with the first service: (b1) determining whether the mapping ID associated with the primary entity ID is stored in the first cache copy; (b2) determining whether the mapping ID associated with the primary entity ID is stored in the primary copy of the database, conditional on determining that the mapping ID is not stored in the first cache copy; (b3) obtaining the mapping ID and providing the mapping ID to the client device on the condition that the mapping ID is stored in one of the first cache copy and the master copy in the database; (c) subject to determining that the service information specifies that the request is associated with the second service: (c1) determining whether the mapping ID associated with the primary entity ID is stored in the second cache copy; (c2) determining whether the mapping ID associated with the primary entity ID is stored in the primary copy of the database, conditional on determining that the mapping ID is not stored in the second cache copy; (c3) obtaining the mapping ID and transmitting the mapping ID to the client device on the condition that the mapping ID is stored in one of the second cache copy and the master copy in the database; as well as (d) Conditioned on determining that the mapping ID is not stored in the master copy of the database: (d1) generating the mapping ID associated with the main entity ID; (d2) transmitting the mapping ID to the client device, and (d3) Storing the primary entity ID and the associated mapping ID in a new row in the database.

16. The method of claim 15, wherein the first cache copy and the second cache copy completely cache the database, and the database server further stores a first partial cache copy in the first cache copy and a second partial cache copy in the second cache copy; The step (b1) of the method comprises: determining whether the mapping ID associated with the primary entity ID is stored in the first partial cache copy; Conditioned on determining that the mapping ID is not stored in the first partial cache copy, determining whether the mapping ID associated with the primary entity ID is stored in the first cache copy; and The step (c1) of the method comprises: determining whether the mapping ID associated with the primary entity ID is stored in the second partial cache copy, and Conditional on determining that the mapping ID is not stored in the second partial cache copy, determining whether the mapping ID associated with the primary entity ID is stored in the second cache copy.

17. A computer-implemented method, the method comprising: (a) receiving a first request from a client device for providing an application with a mapping ID associated with a master entity ID, the request including the master entity ID; (b) retrieving the mapping ID associated with the primary entity ID from a database; (c) providing the mapping ID to the application; (d) processing the master entity ID and the mapping ID using an encryption key to generate an encrypted master entity ID and an encrypted mapping ID; as well as (e) transmitting the encrypted entity ID, the encrypted mapping ID, and expiration information specifying a validity period of the encrypted entity ID and the encrypted mapping ID to the client device.

18. The method according to claim 17, further comprising: (f) receiving, from the client device, a second request for providing the application with the mapping ID associated with the master entity ID, the request comprising the master entity ID, the encrypted entity ID, the encrypted mapping ID, and expiration information; (g) determining whether the encrypted master entity ID and the encrypted mapping ID are valid based on the expiration information, and (h) Under the condition that the encrypted master entity ID and the encrypted mapping ID are valid, the received encrypted master entity ID and the encrypted mapping ID are decrypted using the encryption key, and the decrypted mapping ID is provided to the application.

19. A system comprising one or more computers and one or more storage devices, the storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the method according to any one of claims 1 to 18.

20. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of any one of claims 1 to 18.

21. A system comprising a first database server and a second database server, wherein the first database server and the second database server store a first database and a second database respectively, wherein each of the databases has a plurality of rows, each row storing an entity ID and a mapping ID associated with the entity ID; The first database is configured as: storing a new entity ID and a new mapping ID associated with the entity ID in a new row in the first database, and transmitting the new entity ID and the new mapping ID to the second database server; and The second database server is configured as follows: determining a conflict between the received new entity ID and the received new mapping ID and an existing entity ID and an associated existing mapping ID stored in the second database; determining a valid mapping ID associated with the new entity ID, and Conditional on determining that the effective mapping ID is different from the existing mapping ID stored in the second database, updating the second database based on the new entity ID and the associated effective mapping ID.

22. A system according to claim 21, wherein the second database server is configured to determine a conflict between the received new entity ID and the new mapping ID and the existing entity ID and the existing mapping ID by determining that the received new entity ID and the existing entity ID stored in the second database are the same, and the received new mapping ID and the associated existing mapping ID stored in the second database are different.

23. The system according to claim 21 or 22, further comprising a third database server storing a third database, the third database having a plurality of rows, each row storing an entity ID and a mapping ID associated with the entity ID, and The first database server is further configured to transmit the new entity ID and the new mapping ID to the third database server, and The second database server is configured to determine a valid mapping ID associated with the new entity ID by: retrieving from the third database server another mapping ID associated with the new mapping ID and stored in the third database, and One of the new mapping ID received from the first database server, the existing mapping ID stored in the second database, and the other mapping ID retrieved from the third database server is selected as the effective mapping ID associated with the new entity ID.

24. The system of claim 23, wherein the second database server is configured to select the valid mapping ID by: A counter value is determined for each of the new mapping ID received from the first database server, the existing mapping ID stored in the second database, and the another mapping ID retrieved from the third database server, and the mapping ID having the lowest counter value is selected as the valid mapping ID.

25. The system of claim 23, wherein the first database, the second database, and the third database are further configured to store a creation timestamp in each row, the creation timestamp specifying a time when the mapping entity ID stored in the row was created; The first database server, the second database server and the third database server are further configured to transmit corresponding creation timestamps to the second database server when transmitting the new entity ID, the new mapping ID and the other mapping ID respectively; and wherein the second database server is configured to select the valid mapping ID by: The mapping ID is selected from among the new mapping ID received from the first database server, the existing mapping ID stored in the second database, and the another mapping ID retrieved from the third database server, and the mapping ID has the earliest timestamp.

26. The system according to any one of claims 21 to 25, wherein the second database server is further configured to transmit the new entity ID and the valid mapping ID to the first database server and the third database server; The first database server is further configured to update the first database based on the new entity ID and the associated valid mapping ID, conditional on determining that the valid mapping ID and the new mapping ID are different, and The third database server is further configured to update the third database based on the new entity ID and the associated valid mapping ID, conditional on determining that the valid mapping ID and the another mapping ID are different.

27. The system according to any one of claims 21 to 26, wherein the first database server is further configured to generate the new mapping ID from the new entity ID according to the method according to any one of claims 1 to 8.

28. A database server for processing a request from a client device for providing a mapping ID associated with a master entity ID, the database server being configured to store: a master copy of a database having a plurality of rows, each row including an entity ID of the plurality of entity IDs and a mapping ID associated with the entity ID, and a first cached copy and a second cached copy of the database; The database server is also configured to: receiving the request from the client device, the request including the host entity ID and service information, the service information specifying whether the request is associated with a first service or a second service; Conditioned on determining that the service information specifies that the request is associated with the first service: determining whether the mapping ID associated with the primary entity ID is stored in the first cache copy; determining whether the mapping ID associated with the primary entity ID is stored in the primary copy of the database, conditional on determining that the mapping ID is not stored in the first cache copy; Conditioned on determining that the mapping ID is stored in one of the first cache copy and the master copy of the database, obtaining the mapping ID and providing the mapping ID to the client device; Conditioned on determining that the server information specifies that the request is associated with the second server: determining whether the mapping ID associated with the primary entity ID is stored in the second cache copy; determining whether the mapping ID associated with the primary entity ID is stored in the primary copy of the database, conditional on determining that the mapping ID is not stored in the second cache copy; obtaining the mapping ID and transmitting the mapping ID to the client device on the condition that the mapping ID is stored in one of the second cache copy and the master copy of the database; as well as Conditional on determining that the mapping ID is not stored in the master copy of the database: generating the mapping ID associated with the main entity ID; transmitting the mapping ID to the client device, and The primary entity ID and the associated mapping ID are stored in a new row in the database.

29. The database server according to claim 28, wherein the first cache copy and the second cache copy completely cache the database, and the database server further stores a first partial cache copy in the first cache copy and a second partial cache copy in the second cache copy; The database server is configured to determine whether the mapping ID associated with the primary entity ID is stored in the first cache copy by: determining whether the mapping ID associated with the primary entity ID is stored in the first partial cache copy; determining whether the mapping ID associated with the primary entity ID is stored in the first cache copy, conditional on determining that the mapping ID is not stored in the first partial cache copy; and The database server is configured to determine whether the mapping ID associated with the primary entity ID is stored in the second cache copy by: determining whether the mapping ID associated with the primary entity ID is stored in the second partial cache copy, and Conditional on determining that the mapping ID is not stored in the second partial cache copy, determining whether the mapping ID associated with the primary entity ID is stored in the second cache copy.

Images