Method, database server, system, medium and computer program for improving latency and data consistency in large databases for bidirectional lookups

Abstract

The computer-implemented method includes receiving respective primary entity IDs from a user, each primary entity ID including a first and a second bit string, and further includes generating a respective mapped ID from each of the primary entity IDs such that: i) each of the mapped IDs is unique; and ii) each of the mapped IDs includes the first bit string of the primary entity ID and a third bit string that is different from the second bit string of the primary entity ID.

Description

[Technical Field]

[0001] This application claims priority from Singapore Application No. 10202300253R, filed on February 1, 2023, entitled "METHODS FOR IMPROVING LATENCY AND DATA CONSISTENCY IN LARGE DATABASES FOR BIDIRECTIONAL LOOKUP," the entire contents of which are incorporated herein by reference in their entirety.

[0002] This application relates to databases, and more particularly to methods for improving latency and data consistency in large databases for bidirectional lookups. [Background technology]

[0003] A web application or service may need to create, store, and use bidirectional mappings between associated data elements. To this end, for any given data element, related elements may be derived and forward and reverse mappings may be stored in a database table. When the number of records stored in such database tables becomes large, or when the database is queried from different geographic regions, scalability, latency, and data consistency issues arise. Conventional methods and tools for addressing these issues are inadequate, especially when web services require support for bidirectional mappings of billions of associated data elements and global deployment. Summary of the Invention [Means for solving the problem]

[0004] The present invention aims to provide a novel and useful method for generating a mapped ID from an entity ID, and a method and database server for providing the mapped ID to a client device. The present invention makes it possible to provide a database for bidirectional lookup between an entity ID and a mapped ID that can be partitioned into a distributed database, for example, so that the use of distributed transactions is avoided for both read and write requests. Furthermore, the database can have a small storage size while maintaining data consistency. The present invention achieves this by first storing, in each row of the database, the entity ID, the associated mapped ID, and a shard key indicating which partition of the distributed database the row belongs to, and then generating the mapped ID from the entity ID such that the mapped ID is unique to the entity ID, but either the entity ID or the mapped ID can be used to generate the shard key. Thus, the present invention improves the speed of storing new mappings and can ensure data atomicity using local database transactions without involving distributed transactions.

[0005] The present invention further enables storing replicas of a database on servers located in different regions while preventing data inconsistencies, for example, when conflicting requests to store mappings (i.e., entity IDs and associated mapped IDs) are made simultaneously from client devices located in different regions. The present invention accomplishes this by first enabling a received mapping to be stored in the replica of the request-receiving server. The mapping is then transmitted to other servers that attempt to update their replicas with the received mapping. If a conflict occurs between the received mapping and a mapping already stored on a particular server, the affected server determines the correct mapping by querying relevant mappings stored on other servers so that the affected server can update its database replica with the correct mapping. The affected server then transmits the correct mapping to the other servers so that the other servers can modify their replicas as needed. The present invention facilitates improving data consistency across multiple replicas of a database deployed in different geographic regions without increasing the latency of write requests.

[0006] The present invention may further improve the consistency of handling lookup requests by users associated with a first service, for example, regardless of the traffic load of users associated with a second service. The present invention may achieve this by using two memories that cache a (common) database and directing lookup requests from users associated with the first service to one of the memories. The present invention may protect users associated with the first service from experiencing inconsistent service caused by traffic loads (e.g., spikes) of users associated with the second service.

[0007] The present invention further reduces lookup requests performed on a database, for example, for repeated lookup requests, i.e., requests made by the same user for the same mapping. The present invention can achieve this by storing the requested mapping, i.e., the entity ID and associated mapped ID, on the user device in response to the initial lookup request. The mapping may be stored in the form of an encrypted cookie with a predetermined expiration time. Any subsequent lookup requests are sent with the encrypted cookie, so that when a request is received and the cookie is still valid, the cookie can be decrypted and the decrypted mapping can be used to service the lookup request rather than searching the database. Thus, the present invention can reduce the read load on the database and improve the availability of the lookup service in the event of database failure or downtime.

[0008] According to a first aspect of the present invention, there is provided a computer-implemented method, the method including receiving respective primary entity IDs from a user, each primary entity ID including a first bit string and a second bit string, the method further including generating respective mapped IDs from each of the primary entity IDs such that: i) each mapped ID is unique; and ii) each mapped ID includes the first bit string of the primary entity ID and a third bit string that is different from the second bit string of the primary entity ID.

[0009] The first bit string may be the first bit sequence of the primary entity ID or any other suitable bit sequence or set of bits of the primary entity ID.

[0010] Optionally, the first bit string has a bit length N, and the primary entity ID and the mapped IDs include the first bit string as their least significant N bits. The bit length N of each first bit string may be 10, and each mapped ID may have a bit length of 64.

[0011] Optionally, the method may further comprise the step of maintaining a counter, wherein a third bit string of the mapped ID includes counter information obtained from the counter.

[0012] Optionally, the method may further include, for each primary entity ID, generating a shard key from the primary entity ID or the mapped ID using a hash function, where the hash function generates the shard key based on the value of the first bit string.

[0013] Optionally, the method may further include, for each primary entity ID, storing the primary entity ID, the mapped ID, and the shard key in the same row in the database. The database may be configured as a partitioned database including multiple shards. Each shard may be associated with a different shard key value. Storing the primary entity ID, the mapped ID, and the shard key in the same row in the database includes storing the primary entity ID, the mapped ID, and the shard key in the same row in the shard associated with the shard key value.

[0014] According to a second aspect of the present invention, there is provided a computer-implemented method for resolving conflicts between a first database and a second database. The first database and the second database each have a plurality of rows, each row storing an entity ID and a mapped ID associated with the entity ID. The method includes the steps of: (a) storing a new entity ID and the new mapped ID associated with the entity ID in a new row in the first database; (b) determining a conflict between the new entity ID and the new mapped ID and an existing entity ID and the associated existing mapped ID stored in the second database; (c) determining a valid mapped ID associated with the new entity ID; and (d) updating the second database based on the new entity ID and the associated valid mapped ID, provided that it is determined that the valid mapped ID differs from the existing mapped ID stored in the second database.

[0015] Optionally, step (b) of the method may include determining that the new entity ID and the existing entity ID are the same and that the new mapped ID and the existing mapped ID are different.

[0016] Optionally, the third database may have a further plurality of rows, each row storing an entity ID and a mapped ID associated with the entity ID, and step (d) of the method may include retrieving from the third database another mapped ID associated with the new mapped ID, and selecting one of the new mapped ID, the existing mapped ID stored in the second database, and the other mapped ID retrieved from the third database as a valid mapped ID associated with the new entity ID. The step of selecting a valid mapped ID may include determining a counter value for each of the new mapped ID, the existing mapped ID stored in the second database, and the other mapped ID retrieved from the third database. Selecting the mapped ID with the smallest counter value as the valid mapped ID may also include selecting the mapped ID with the smallest counter value as the valid mapped ID. Alternatively, the first database, the second database and the third database may further store in each row a creation timestamp that identifies the time when the mapped entity ID stored in the row was created, and the step of selecting a valid mapped ID may include selecting the one having the earliest timestamp from the new mapped ID, an existing mapped ID stored in the second database and another mapped ID retrieved from the third database.

[0017] Optionally, the method may further include the steps of (e) updating the first database based on the new entity ID and the associated valid mapped ID, if it is determined that the valid mapped ID and the new mapped ID are different, and (f) updating the third database based on the new entity ID and the associated valid mapped ID, if it is determined that the valid mapped ID and another mapped ID retrieved from the third database are different.

[0018] Optionally, the method may further comprise an initial step of generating a new mapped ID from the new entity ID according to the first aspect of the invention. According to a third aspect of the present invention, there is provided a computer-implemented method for processing a request from a client device by a database server to provide a mapped ID associated with a primary entity ID. The database server stores a primary copy of a database having a plurality of rows, each row including an entity ID of a plurality of entity IDs and a mapped ID associated with the entity ID. The database server further stores a first cached copy and a second cached copy of the database. The method includes the step (a) of receiving a request from a client device, the request including the primary entity ID and service information specifying whether the request is associated with a first service or a second service. The method further includes the steps of: (b1) determining whether a mapped ID associated with the primary entity ID is stored in a first cached copy, on condition that it has determined that the service information specifies that the request is associated with a first service; (b2) determining whether a mapped ID associated with the primary entity ID is stored in a primary copy of the database, on condition that it has determined that the mapped ID is not stored in the first cached copy; and (b3) obtaining the mapped ID and providing the mapped ID to the client device, on condition that it has determined that the mapped ID is stored in one of the first cached copy and the primary copy of the database.The method further includes the steps of: (c1) determining whether a mapped ID associated with the primary entity ID is stored in a second cached copy, on condition that it has determined that the service information specifies that the request is associated with a second service; (c2) determining whether a mapped ID associated with the primary entity ID is stored in a primary copy of the database, on condition that it has determined that the mapped ID is not stored in the second cached copy; and (c3) obtaining the mapped ID and sending the mapped ID to the client device, on condition that it has determined that the mapped ID is stored in one of the second cached copy and the primary copy of the database. The method further includes the steps of (d1) generating a mapped ID associated with the primary entity ID, (d2) sending the mapped ID to the client device, and (d3) storing the mapped ID associated with the primary entity ID in a new row in the database, on condition that it has determined that the mapped ID is not stored in the primary copy of the database.

[0019] Optionally, the first cached copy and the second cached copy may completely cache the database, and the database server may further store a first partial cached copy of the first cached copy and a second partial cached copy of the second cached copy. Step (b1) of the method may include determining whether a mapped ID associated with the primary entity ID is stored in the first partial cached copy, and, conditional on determining that the mapped ID is not stored in the first partial cached copy, determining whether the mapped ID associated with the primary entity ID is stored in the first cached copy. Step (c1) of the method may include determining whether a mapped ID associated with the primary entity ID is stored in the second partial cached copy, and, conditional on determining that the mapped ID is not stored in the second partial cached copy, determining whether the mapped ID associated with the primary entity ID is stored in the second cached copy.

[0020] According to a fourth aspect of the present invention, there is provided a computer-implemented method, the method comprising the steps of: (a) receiving a first request from a client device for providing a mapped ID associated with a primary entity ID to an application, the request including the primary entity ID; (b) retrieving the mapped ID associated with the primary entity ID from a database; (c) providing the mapped ID to the application; (d) processing the primary entity ID and the mapped ID with an encryption key to generate an encrypted primary entity ID and an encrypted mapped ID; and (e) sending to the client device the encrypted entity ID, the encrypted mapped ID, and expiration information specifying expiration dates for the encrypted entity ID and the encrypted mapped ID.

[0021] Optionally, the method may further include the steps of: (f) receiving, from the client device, a second request to provide the application with a mapped ID associated with the primary entity ID, the second request including the primary entity ID, the encrypted entity ID, the encrypted mapped ID, and expiration information; (g) determining whether the encrypted primary entity ID and the encrypted mapped ID are valid based on the expiration information; and (h) on the condition that it is determined that the encrypted primary entity ID and the encrypted mapped ID are valid, decrypting the received encrypted primary entity ID and encrypted mapped ID using an encryption key and providing the decrypted mapped ID to the application.

[0022] According to a fifth aspect of the present invention, there is provided a system comprising one or more computers and one or more storage devices having stored thereon instructions which, when executed by the one or more computers, cause the one or more computers to carry out the first, second, third or fourth aspect of the present invention.

[0023] According to a sixth aspect of the present invention there is provided one or more non-transitory computer storage media having stored thereon instructions which, when executed by one or more computers, cause the one or more computers to carry out the first, second, third or fourth aspects of the present invention.

[0024] According to a seventh aspect of the present invention, there is provided a system comprising a first database server and a second database server. The first database server and the second database server store a first database and a second database, respectively. The databases each have a plurality of rows, each row storing an entity ID and a mapped ID associated with the entity ID. The first database server is configured to store a new entity ID and a new mapped ID associated with the entity ID in a new row of the first database and transmit the new entity ID and the new mapped ID to the second database server. The second database server is configured to resolve conflicts between the received new entity ID and new mapped ID and existing entity IDs and associated existing mapped IDs stored in the second database. The second database server is further configured to determine a valid mapped ID associated with the new entity ID and, if it determines that the valid mapped ID differs from the existing mapped ID stored in the second database, update the second database based on the new entity ID and the associated valid mapped ID.

[0025] Features of the system may be as described above with reference to the second aspect of the invention.

[0026] According to an eighth aspect of the present invention, there is provided a database server for processing a request from a client device to provide a mapped ID associated with a primary entity ID. The database server is configured to store a primary copy of a database having a plurality of rows, each row including an entity ID of the plurality of entity IDs and a mapped ID associated with the entity ID. The database server is further configured to store a first cached copy and a second cached copy of the database. The database server is further configured to receive a request from the client device, the request including the primary entity ID and service information specifying whether the request is associated with a first service or a second service. The database server is configured to, on condition that it determines that the service information specifies that the request is associated with the first service, determine whether the mapped ID associated with the primary entity ID is stored in the first cached copy, and, on condition that it determines that the mapped ID is not stored in the first cached copy, determine whether the mapped ID associated with the primary entity ID is stored in the primary copy of the database. The database server is further configured to, on the condition that it determines that the mapped ID is stored in one of the first cached copy or the primary copy of the database, obtain the mapped ID and provide the mapped ID to the client device. The database server is further configured to, on the condition that it determines that the service information specifies that the request is associated with a second service, determine whether the mapped ID associated with the primary entity ID is stored in the second cached copy, and, on the condition that it determines that the mapped ID is not stored in the second cached copy, determine whether the mapped ID associated with the primary entity ID is stored in the primary copy of the database.The database server is further configured to, on the condition that it determines that the mapped ID is stored in one of the second cached copy or the primary copy of the database, obtain the mapped ID and send the mapped ID to the client device. On the condition that it determines that the mapped ID is not stored in the primary copy of the database, the database server is further configured to, generate a mapped ID associated with the primary entity ID, send the mapped ID to the client device, and store the primary entity ID and the associated mapped ID in a new row in the database.

[0027] The database server may be characterized as described above with reference to the third aspect of the invention. [Brief explanation of the drawings]

[0028] [Figure 1] 10 is a flowchart of an example process for generating a mapped ID from a primary entity ID. [Figure 2] FIG. 2 illustrates an exemplary configuration of the mapped ID of FIG. 1. [Figure 3] FIG. 2 illustrates an exemplary database generated according to the process of FIG. 1. [Figure 4] FIG. 1 is a block diagram of an exemplary computer network. [Figure 5] FIG. 1 illustrates an exemplary process for client-side caching. [Figure 6] 10 is a flowchart illustrating the relative order of data guarantees provided by a memory used in some embodiments to store mappings. [Figure 7] 10 is a flowchart of an example process for providing a mapped ID to a client device. [Figure 8] FIG. 1 is a block diagram illustrating an exemplary process for resolving conflicts between multiple databases. [Figure 9] 9 is a flowchart of the process of FIG. 8. [Figure 10] FIG. 1 illustrates an exemplary process for warming up a database. [Figure 11] FIG. 11 illustrates a computer system that can be used to implement the methods of FIGS. 1 and 7-10. DETAILED DESCRIPTION OF THE INVENTION

[0029] Mapped ID Generation In general, a bidirectional mapping associates two elements with each other to form a one-to-one correspondence. For example, given multiple entities, each identified by a corresponding entity ID, an associated mapped ID may be derived for each entity ID such that each mapped ID is unique with respect to the associated entity ID. To enable bidirectional lookups, i.e., to look up the associated mapped ID for any particular entity ID and vice versa, the mappings may be stored in a database. For example, the database may include a first column "Entity ID" and a second field "Mapped ID," where each row may be populated with an entity ID and an associated mapped ID. For a given entity ID, the associated mapped ID can be looked up in the database by finding a row that contains the entity ID in the "Entity ID" column. Similarly, for a given mapped ID, the associated entity ID can be looked up in the database by finding a row that contains the mapped ID in the "Mapped ID" column. However, this method of storing mappings is not necessarily suitable for large databases: for many applications, databases typically contain only up to 100M records before read / write latency becomes too high. However, some applications may require access to billions of mappings.

[0030] A known technique for improving the performance and scalability of large databases is to divide the database into smaller partitions (or shards), also known as sharding. This allows the database to be distributed across multiple servers, computing systems, clusters, etc. One way to divide a database is called horizontal sharding, in which the partitions have the same schema as the original database and every row in the original database is assigned to a specific partition based on a shard key. The shard key for each row may be derived from elements in a column of that row. However, horizontal sharding of the above example database would make efficient bidirectional lookups impossible. For example, if the shard key were derived from elements in an “entity ID” column, the associated mapped ID could still be efficiently looked up for a given entity ID because the database partition in which the related row is stored can be derived from the entity ID. However, reverse lookups would no longer be efficient because it would be impossible to identify, for a given mapped ID, the database partition in which the related row is stored.

[0031] To support bidirectional lookups, the above mapping storage method may be modified so that each mapping may be stored in two rows in the database. In one row, the entity ID is stored in a first column and the mapping-ID is stored in a second column. In another row, the mapped ID is stored in a first column and the entity ID is stored in a second column. A shard key may be derived from the elements in the first column, and the row may be partitioned accordingly. Then, for a given entity ID, a partition may be identified whose row contains the entity ID in the first column and the mapped ID in the second column, enabling efficient bidirectional lookups. Similarly, for a given mapped ID, a partition may be identified whose row contains the mapped ID in the first column and the entity ID in the second column. This modified storage scheme enables bidirectional lookups, but because the entity ID and the mapped ID generally do not reference each other, they will generate different shard key values ​​and be stored in different partitions. Therefore, writing such a mapping to the database may cause distributed transactions, i.e., accessing partitions on different servers, which may directly lead to additional delays.

[0032] 1 to 3, an exemplary process for generating a mapped ID from a given entity ID for bidirectional lookup and overcoming the drawbacks of the above-described method will be described. FIG. 1 illustrates steps of the exemplary process. FIG. 2 illustrates an exemplary configuration of a mapped ID 1 representing S102 of the method of FIG. 1. FIG. 3 illustrates a database generated based on the process of FIG. 1.

[0033] Referring to Figure 1, in step S101, a primary entity ID is received from a user. The primary entity ID is one of a plurality of entity IDs and includes a first bit string and a second bit string. In step S102, a mapped ID is generated from the primary entity ID. The mapped ID is generated such that: i) the mapped ID is unique with respect to the primary entity ID; and ii) the mapped ID includes the first bit string of the primary entity ID and another bit string that is different from the second bit string of the primary entity ID.

[0034] In one embodiment, the first bit string has a bit length of N and corresponds to the least significant N bits of both the primary entity ID and the mapped ID, where N is an integer greater than 0. In one embodiment, the mapped ID may be generated such that the value of the least significant N bits is the same as the value of the least significant N bits of the primary entity ID.

[0035] In the embodiment shown in Figure 2, Mapped ID 1 has a bit length of 64 bits. In this embodiment, the first bit string of the primary entity ID included in Mapped ID 1 is 10 bits long. Furthermore, in the embodiment shown in Figure 2, another bit string may be 54 bits long.

[0036] In one embodiment, mapped IDs are generated such that they are unique to the primary entity ID by including a counter category value in a separate bit string included in the mapped ID. The counter category value may be provided by a counter module that stores and manages counter category values. The counter module may be configured to change the counter category value each time a new mapped ID is generated. The counter category value may monotonically increase each time a new mapped ID is generated. In one embodiment, the counter module is configured to provide any counter category value only once for the generation of a mapped ID, i.e., the provided counter category values ​​are globally unique. Therefore, a first mapped ID containing a counter category value lower than a counter category value of a second mapped ID indicates that the first mapped ID was created at an earlier point in time. In the embodiment shown in FIG. 2, the counter category value is included in the mapped ID as a bit string of length 43. This is because the counter category value is at least 2 43 Supports generating unique mapped IDs.

[0037] In one embodiment, the mapped ID may be generated to further encode high-level information, such as business information, in another bit string of the mapped ID. In the embodiment shown in Figure 2, the business information (biz_id) is encoded in a bit string of length 4, and the counter source information is encoded in one bit of the mapped ID. The counter source information may specify a counter module.

[0038] Referring again to FIG. 1 , in step S103, a hash function is used to generate a shard key from the primary entity ID or the mapped ID, where the hash function generates the shard key based on the value of the first bit string. A hash function may be a computational method that receives an input and generates a shard key value (also referred to simply as a shard key) as an output. The shard key may represent an integer. In an embodiment where the first bit string has a bit length of N and corresponds to the least significant N bits of both the primary entity ID and the mapped ID, the hash function may be configured to generate the shard key based on the least significant N bits of the received input. In this embodiment, the hash function generates the same shard key value regardless of whether the primary entity ID or the mapped ID is received as input. Also, in this embodiment, the shard key value ranges from 0 to (2 N Represents an integer (-1).

[0039] In step S104, the primary entity ID, the mapped ID, and the shard key are stored in the same row in a database. The database may be a relational database. The database may be stored on a database server. The database server may be a distributed computing system or a cluster of servers.

[0040] In an embodiment, the database may be configured as a partitioned database including multiple shards. Each shard may be associated with a different shard key value. In this case, the database server may be a cluster of servers, and each shard of the database may be stored on a different server. In this embodiment, in step S104, the primary entity ID, mapped ID, and shard key may be stored in the same row of the shard associated with the shard key value.

[0041] In one embodiment, steps S101 to S104 of FIG. 1 may be repeated for multiple entity IDs such that the database contains multiple rows, each row containing an entity ID, an associated mapped ID, and a shard key.

[0042] The advantages of the method described with reference to FIG. 1 will be explained by way of example with reference to FIG. 3. FIG. 3 illustrates a database 30 that may be generated by repeatedly executing steps S101 to S104 of FIG. 1. The database 30 includes a first field 32, “Entity ID,” a second field, “Mapped ID,” and a third field, “Shard Key.” In one embodiment, the database 30 may further include another field, such as a fourth field 38, “Creation Time,” that identifies the time when the mapping stored in the corresponding row was created. In this embodiment, the database 30 has X records, where X is a positive integer. Horizontal sharding may be applied to the database 30 to divide the database 30 into multiple partitions (commonly referred to as “shards”). FIG. 3 illustrates four of these partitions: partitions 40-0, 40-1, 40-2, and 40-1023. In the example illustrated in FIG. 3, partition 40-0 is associated with shard key value 0. Partition 40-0 contains two rows that correspond to the two rows in database 30 that have a shard key value equal to 0.

[0043] Further, with reference to FIG. 3 , a partitioned database having partitions 40-0 through 40-1023 allows efficient bidirectional lookups between entity IDs and associated mapped IDs, since a hash function can be used to identify the relevant partition of the database from either the entity ID or the mapped ID. For example, for any given entity ID, e.g., “Entity ID 3” in FIG. 3 , a hash function may be used to identify the database partition that stores the relevant row, i.e., the row containing the given entity ID and the associated mapped ID. In this example, applying the hash function “Entity ID 3” results in a shard key value equal to 1. A row containing “Entity ID 3” and the associated mapped ID “Mapped ID 3” is likely to be found in the corresponding partition 40-1. Similarly, if a reverse mapping is desired, the mapped ID “Mapped ID 3” may be provided to the hash function. Also, because the mapped ID was created in S102 to include a first bit string of the associated entity ID, and the hash function generates a shard key value based on the value of the first bit string, the hash function may generate a shard key value equal to 1. Therefore, the mapped ID generated in S102 deterministically generates the same shard key value as the associated entity ID. Also, because each mapping is stored in a single row, the required storage size of the database is halved compared to the two-row implementation described above. Furthermore, because only a single row is stored, write speeds are increased when storing the mappings, and local database transactions can be used to ensure data atomicity without involving distributed transactions. Furthermore, these mappings can be efficiently stored in the same shard for SQL and NoSQL data stores.

[0044] Caching Strategies FIG. 4 illustrates a computer network including a first database server 50 connected to a client device 52 via a network. The computer network enables the client device 52 to send data to and receive data from the first database server 50. The database server may store a database 54 configured to store multiple rows, where each row includes an entity ID, an associated mapped ID, and a shard key. The database 54 may be configured as a partitioned database including multiple shards. The first database server 50 may be configured to perform the method described with reference to FIG. 1. The computer network may further include a second database server 61 and a third database server 67, which are described in detail below.

[0045] The first database server 50 may be further configured to provide services to the client device 52, enabling the client device 52 to perform a bidirectional lookup of entity IDs and associated mapped IDs stored in the database 54. The database server 50 may include an interface module 56 configured to transmit data to and receive data from the client device 52 over a network. In broad terms, a client device may send a request including an entity ID to the interface module 56 of the first database server 50 and receive a mapped ID associated with the transmitted entity ID from the first database server 50. Similarly, a client device may send a request including a mapped ID to the interface module 56 of the first database server 50 and receive an entity ID associated with the transmitted mapped ID from the first database server 50.

[0046] The client device 52 may be a computer system described below with reference to Figure 11. The first database server 50 may be implemented in a number of ways, for example, on a single server, on a distributed computing system, on a cluster of servers, on a cloud-based system, etc. If the first database server 50 is implemented on a distributed computing system or a cluster of servers, each shard of the partitioned database may be stored on a different server. The network may be the Internet or another network capable of connecting the client device 52 and the first database server 50 together.

[0047] A user with access to the entity ID may run a local application on their client device. The local application may be configured to use the web application. The web application may receive as input a request including the mapped ID and respond to the request with a response including the mapped ID. To enable use of the web application, the local application may obtain the mapped ID associated with the entity ID by sending a corresponding request to the interface module 56 of the first database server 50. In a typical use case, the local application may send multiple lookup requests for the same entity ID within a short period of time, which may result in a heavy traffic load on the first database server 50.

[0048] Referring to FIG. 5 , the following describes a process that enables reducing the number of redundant lookup requests sent to the interface module 56 by caching previously looked up mappings on the client side. In this embodiment, the local application 70 may send a request for a web application 72 to a service application 74, which may be running on a gateway server. The service application 74 may extract the entity ID from the received request and send a request to the interface module 56 of the first database server 50 to provide the associated mapped ID. After receiving the mapped ID, the service application 74 may replace the entity ID included in the request received from the local application with the mapped ID. The service application 74 may also temporarily store the entity ID and the mapped ID as a key / value pair in local memory. When the local application 70 sends a subsequent request containing the same entity ID, the service application 74 may use the mapped ID stored in local memory rather than sending the request to the first database server 50. This reduces the traffic load on the database server. Similarly, the service application 74 may replace the mapped ID in any response from the web application 72 with the associated entity ID by retrieving the entity ID from the first database server 50 or, if possible, using an entity ID stored in local memory.

[0049] 5 may also include automated cookie caching, as described below, to further reduce the traffic load caused by redundant lookup requests and to provide a fault-tolerant setup for disaster recovery. In general, it is desirable to provide a process so that the bidirectional lookup service provided by database server 50 remains available even when the database is temporarily unavailable, for example due to downtime or extreme failure.

[0050] As described above, the service application 74 may extract the entity ID from the request sent by the local application 70 and may send a request to the interface module 56 of the first database server 50 to provide the associated mapped ID. After receiving the mapped ID, the service application 74 may replace the entity ID included in the request received from the local application with the mapped ID. The service application 74 may also temporarily store the entity ID and the mapped ID as a key / value pair in local memory. In this embodiment, the service application 74 may further encrypt the entity ID and the mapped ID using an encryption key stored on the user device. When the service application 74 receives a response from the web application 72, the service application 74 may replace the mapped ID with the associated entity ID using the entity ID stored in the key / value pair and may send the response to the local application 70 along with the encrypted entity ID and mapped ID. The service application 74 may also send expiration information specifying an expiration date for the encrypted entity ID and the encrypted mapped ID. The service application 74 may send the encrypted entity ID and the mapped ID in the form of an HTTP web cookie. The cookie may include expiration information, such as a time-to-live (TTL) value that describes the period for which the cookie is valid. In response to receiving the cookie, the local application 70 may store the cookie on the client device.

[0051] Any subsequent requests sent by the local application 70 may include the encrypted cookie. In response to receiving a request, the service application 74 may determine whether the cookie is valid by comparing the cookie's TTL value with the current time. If the service application 74 determines that the cookie is invalid, it may process the request as described above, i.e., as if the request did not include a cookie. If the service application 74 determines that the cookie is valid, it may decrypt the cookie using the encryption key to obtain a decrypted entity ID and a mapped ID. The service application 74 may then process the received request using the decrypted entity ID and the mapped ID. This allows lookup requests to be processed without sending the lookup request to the database server. This further reduces the traffic load on the database server and ensures that lookup services remain available even if the database server is temporarily unavailable.

[0052] Returning to FIG. 4 , the first database server 50 may further include a first memory 58 configured to cache the database 54. In this embodiment, the first memory 58 is a key-value storage. For example, the first memory 58 may cache the entire database 54. As will be described in more detail with reference to FIG. 5 , in this case, lookup requests received by the interface module 56 may be directed to the first memory 58 rather than the database 54. This reduces the latency of lookup requests experienced by the client device 52 because the database cached in the first memory 58 can be searched more quickly than the database 54. The first database server 50 may further include a second memory 60 configured to partially cache the database 54. In this case, the second memory 60 may store only a subset of the records stored in the first memory. Also, in this embodiment, lookup requests received by the interface module 56 may be directed to the second memory 60 rather than the first memory 58 or the database 54. This allows the partial database cached in the second memory 60 to be searched faster than the first memory 58 or the database 54, further reducing the delay in lookup requests experienced by the client device 52.

[0053] In other words, the first database server 50 of FIG. 4 provides three access tiers. The first tier, including the second memory 60, can provide fast access. The storage size associated with the first tier can be smaller than the size of the entire database 54. The second tier, including the first memory 60, can provide slower access than the first tier but faster access than the third tier. The storage size associated with the second tier can be larger than the storage size of the first tier. The third tier, including the database 54, can provide slower access than either the first or second tier. The third tier may store data permanently and / or persistently.

[0054] The bidirectional lookup service can reduce latency and increase the number of user requests that can be processed at any one time by employing a caching strategy, as described below with reference to FIGS. 6 and 7. FIG. 6 illustrates the relative order of data guarantees provided by a memory used in some embodiments to store mappings. FIG. 7 illustrates a flowchart of an exemplary method for providing a mapped ID associated with an entity ID to an application running on a client device. In step S501, the client device may send a request including the mapped ID. The request may be processed by the interface module 56 of the first database server 50. In step S502, the interface module 56 may determine whether the mapped ID is stored in the second memory 60. If so, the interface module 56 may retrieve the mapped ID from the second memory 60 and provide the mapped ID to the application (S502). If the mapped ID is not stored in the second memory 60, the interface module 56 may determine in step S503 whether the mapped ID is stored in the first memory 58. If so, the interface module 56 may retrieve the mapped ID from the first memory 58 and provide the mapped ID to the application (S502).

[0055] If the mapped ID is not stored in the first memory 58 or the second memory 60, the interface module 56 may determine whether the mapped ID is stored in the database 54 (S505). If so, the interface module 56 may retrieve the mapped ID from the database 54. In step S507, the first memory 58 and the second memory 60 may be updated to store the entity ID and the retrieved mapped ID. The interface module 56 may provide the mapped ID to the application (S502).

[0056] If the mapped ID is not stored in the database 54, the database server may generate the mapped ID from the transmitted entity ID in step S506 according to the method described with reference to Figure 1 and store the entity ID and the generated mapped ID in the database 54. The first memory 58 and the second memory 60 may be updated to store the entity ID and the generated mapped ID (S507). The interface module 56 may provide the mapped ID to the application (S502).

[0057] In general, each user of a two-way mapping service may be associated with one of multiple services. Users associated with the same service may belong to an organization, a business unit, etc. The traffic load on the two-way mapping service caused by users associated with the same service may be similar or different from the traffic load caused by users associated with another service. For example, a user may be associated with a first or a second service. Users associated with the first service may cause a continuous traffic load, while users associated with the second service may cause a highly fluctuating traffic load. During high traffic loads caused by users of the second service, users of the first service may experience higher latency.

[0058] As described below with reference to FIG. 4, users of the first service can be protected from the adverse effects of such traffic spikes. The second database server 61 shown in FIG. 4 is a variation of the first database server 50. Like the first database server 50, the second database server 61 may receive data from and send data to client devices 62 over a network. Additionally, the database server may store a database 63 similar to database 54, i.e., configured to store multiple rows, where each row includes an entity ID, an associated mapped ID, and a shard key. The database 63 may be configured as a partitioned database including multiple shards. The second database server 61 may be configured to perform the method described with reference to FIG. 1.

[0059] The second database server 61 may include two interface modules 64-1 and 64-2, two first memories 65-1 and 65-2, and two second memories 66-1 and 66-2. The first memories 65-1 and 65-2 may both be configured to independently cache the database 63. The first memory 65-1 may be configured to partially cache the second memory 66-1. The first memory 65-2 may be configured to partially cache the second memory 66-2.

[0060] The client device may send a request for a bidirectional lookup to the database server 63. The request may include an entity ID and service information. The service information may specify which service the user is associated with. If the service information identifies that the user is associated with the first service, the interface module 64-1 may process the request. In this case, similar to steps S502 to S507 of FIG. 7, the interface module 64-1 may determine whether the mapped ID is stored in the second memory 66-1. If so, the interface module 64-1 may retrieve the mapped ID from the second memory 66-1 and provide the mapped ID to the user device. If the mapped ID is not stored in the second memory 66-1, the interface module 64-1 may determine whether the mapped ID is stored in the first memory 66-1. If so, the interface module 64-1 may retrieve the mapped ID from the first memory 66-1 and provide the mapped ID to the user device. If the mapped ID is not stored in the first memory 65-1 or the second memory 66-1, the interface module 64-1 may determine whether the mapped ID is stored in the database 63. If so, the interface module 64-1 may retrieve the mapped ID from the database 63. The first memories 65-1, 65-2 and the second memories 66-1, 66-2 may be updated to store the entity ID and the retrieved mapped ID. The interface module 64-1 may provide the mapped ID to the user device.

[0061] If the mapped ID is not stored in the database 63, the database server may generate a mapped ID from the transmitted entity ID according to the method described with reference to Figure 1 and store the entity ID and the generated mapped ID in the database 63. The first memories 65-1, 65-2 and the second memories 66-1, 66-2 may be updated to store the entity ID and the generated mapped ID. The interface module 64-1 may provide the mapped ID to the user device.

[0062] If the service information identifies that the user is associated with the second service, the interface module 64-2 may process the request. In this case, the interface module 64-2 may perform the above steps. That is, the interface module 64-2 may look up the mapped ID in the second memory 66-2, and, if necessary, in the first memory 65-2 and the database 63. Therefore, in this embodiment, lookup requests from users associated with the first service do not need to be executed on the first memory 65-2 and the second memory 66-2. Similarly, requests from users associated with the second service do not need to be executed on the first memory 65-1 and the second memory 66-1. Because the first memory and the second memory cache the database, most lookup requests can be satisfied without accessing the database 63. Therefore, in this embodiment, heavy traffic caused by users of the second service does not affect the delay of requests by users from the first service. This is because most of these requests have already been satisfied by accessing the first memory 65-1 and the second memory 66-1, which do not receive traffic from users of the second service. Since all write operations are performed on the (common) database 63, data consistency is guaranteed, i.e., neither the first memory 65-1, 65-2 nor the second memory 66-1, 66-2 stores mappings that conflict with any other mapping.

[0063] Multi-regional conflict resolution In broad terms, users of a bidirectional mapping service located in a different geographic location than the database server may experience additional latency compared to users closer to the database server. This additional latency may be tens or hundreds of milliseconds. It may be desirable to provide a database server in each geographic region where users are located and direct mapping requests to the nearest database server to reduce the latency of lookup requests. In this case, each database server may store a local replica of the database containing the bidirectional mapping. As discussed above with reference to FIG. 5, a lookup request may trigger the generation and storage of a new mapped ID, for example, if the requested mapping is not stored in the database. To keep the latency of such write requests low, a "master-master" implementation may be preferable. In such a master-master implementation, each database server can generate and store a new mapping in its own replica of the database. After storing the new mapping, the other database replicas must be updated. However, a data conflict may occur if two users from different geographic regions simultaneously send requests from the same entity ID to two different database servers. In this case, the two database servers generate two different map IDs from the same entity ID. To avoid such data consistency and data conflict problems, methods are provided by which such conflicts can be identified and corrected.

[0064] A method for resolving data conflicts between different databases will now be described with reference to Figures 4, 8 and 9. Figure 8 is a block diagram illustrating the architecture and data flow of the method. Figure 9 is a flowchart of the steps of the method.

[0065] In an embodiment, the computer network may include a first database server 50, a second database server 61, and a third database server 67 as shown in FIG. 4 and described above. In this example, the third database server 67 is configured similarly to the first database server 50, i.e., having one first memory and one second memory, while the second database server is configured with two first memories and two second memories, as described above. In some embodiments, any of the first database server, the second database server, and the third database server may be configured in either manner, i.e., to include one or two first memories and two second memories.

[0066] In one embodiment, the first database server, the second database server, and the third database server may each be located in a different geographic region and configured to host a bidirectional lookup service as described above and configured to receive lookup requests from respective client devices 52, 62, 68, as shown in Figure 4. In this embodiment, the databases 54, 63, 69 may initially contain the same data, i.e., the same mappings.

[0067] 9 is a flowchart illustrating an exemplary method for resolving conflicts between databases 54, 63, and 69. In step S901, an entity ID and a mapped ID associated with the entity ID are stored in a new row in database 52 of first database server 50. For example, as described above with reference to FIG. 7, the entity ID and the mapped ID may be generated and stored in response to a lookup request from client device 52. This means that the entity ID and the mapped ID may be stored in response to determining that a mapped ID associated with the entity ID is stored in database 52.

[0068] In step S902, the change in database 52, i.e., a new row including the entity ID and the generated mapped ID, may be sent to the second database server 61 and the third database server 67. In step S903, in response to receiving the entity ID and the mapped ID, each of the second database server 61 and the third database server 67 may attempt to update their respective databases 63, 69. In other words, the second database server 61 and the third database server 67 may attempt to add the received entity ID and mapped ID to their respective databases 63, 69. The second database server 61 may determine a data conflict between the received entity ID and mapped ID and another entity ID and associated existing mapped ID stored in the second database 63. For example, the second database server 61 may determine that the received entity ID and an existing entity ID are the same, and that the received mapped ID and the existing mapped ID are different. In this case, adding the received data to the new row in database 63 may cause a failure of primary key / index guarantees.

[0069] In step S904, the second database server 61 determines which mapped ID should be associated with the received entity ID. In other words, the second database server 61 determines which mapped ID is valid for the received entity ID. The second database server 61 may write conflict data identifying the data conflict to a data conflict log file 80 stored on the second database server 61. The data conflict log file 80 may be implemented as a Kafka topic. In one embodiment, the data conflict log file 80 may be mirrored to the first database server 50 and the third database server 67.

[0070] The checker module 82 may be an application running on each of the first database server 50, the second database server 61, and the third database server 67, and may be configured to read the data conflict log file 80. If the data conflict log file 80 is implemented as a Kafka topic, the checker module 82 may be implemented as a consumer subscribing to this topic. The checker module 82 running on the second database server 61 may determine a valid mapped ID. To this end, the checker module 82 may send a lookup request including an entity ID to the third database server 67 and receive, as another mapped ID, a mapped ID associated with the sent entity ID stored in the database 69 in a response from the third database server 67. A checker module 82 running on the second database server 61 may then select one of the mapped ID originally received from the first database server, an existing mapped ID stored in the second database 63, and another mapped ID received from the third database server 67 as a valid mapped ID.

[0071] In one embodiment, the checker module 82 may select a valid mapped ID by determining a counter value for each of the mapped ID initially received from the first database server, the existing mapped ID stored in the second database 63, and another mapped ID received from the third database server 67, and selecting the mapped ID with the lowest counter value as the valid mapped ID. Alternatively, as described above with reference to Figures 2 and 3, if the first database 52, the second database 63, and the third database 69 further store in each row a creation timestamp that specifies the time when the mapped entity ID stored in the row was created, the first database server 50 may also send the corresponding creation timestamp to the second database server 61 when sending the entity ID and the mapped ID in step S902. Furthermore, in this embodiment, the third database server 67 may also send the corresponding creation timestamp to the second database server 61 when sending the another mapped ID. In this case, the checker module 82 may select the valid mapped ID from the mapped ID originally received from the first database server 50, the existing mapped IDs stored in the second database 63, and another mapped ID received from the third database server 67 that has the earliest timestamp.

[0072] In step S905, the second database server 61 may write the valid mapped ID to a data modification log file 84 stored on the second database server 61. The data modification log file 84 may be implemented as a Kafka topic.

[0073] A fixer module 86 may be an application running on each of the first database server 50, the second database server 61, and the third database server 67, and may be configured to read the data modification log file 84. If the data modification log file 84 is implemented as a Kafka topic, the fixer module 86 may be implemented as a consumer subscribing to this topic. If the checker module 82 determines in step S904 that the valid mapped ID is not one stored in the database 63, the fixer module 86 running on the second database server 61 may, in response to reading the data modification log 84, update the database 63 such that the entity ID is stored with the valid mapped ID and the corresponding previously stored mapped ID is deleted.

[0074] In step S906, the entity ID and the valid mapped ID may be sent to the first database server 50 and the third database server 61. Therefore, the data modification log file 84 may be mirrored to the first database server 50 and the third database server 67. In step S907, the first database server 50 and the third database server 67 may update their respective first database 52 and third database 69 based on the entity ID and the associated valid mapped ID. Depending on whether the first database 52 and the third database 69 store a valid mapped ID, the respective fixer modules 86 running on the first database server 50 and the third database server 67 may, in response to reading the mirrored data modification log, update their respective databases 52, 69 so that the entity ID is stored with the valid mapped ID and, if applicable, delete the corresponding previously stored mapped ID. This makes it possible to resolve any data conflicts that may be caused by multiple (near-)simultaneous lookup requests sent to database servers located in different geographic regions.

[0075] Warm-up Architecture In general, the bidirectional lookup service described above may be used for a large number of entity IDs, e.g., billions. Although each mapped ID is generated only when the first corresponding user request is sent to the database server, it is usually desirable to pre-populate the database (commonly referred to as warming up). This is because write speeds are typically lower than read speeds, and if the database server receives a large number of write requests, for example, during peak periods, users may experience significant delays. It is therefore desirable to provide a process for continuously pre-populating the database with the most relevant mappings. Such a process is described below with reference to FIG. 10.

[0076] In one embodiment, a large number of entity IDs may be stored in the data warehouse 90. The data warehouse 90 may be stored in a cluster or server connected to a network. In one embodiment, the data warehouse may be stored in the database server 50. In another embodiment, the data warehouse 90 is stored in a separate cluster or server. The number of stored entity IDs may be greater than one billion. Each stored entity ID in the data warehouse 90 may be associated with a timestamp indicating the last time this entity ID was used, for example, in a web application.

[0077] A selector module 92 running on a cluster of data warehouses 90 may be configured to query the data warehouse for entity IDs whose timestamps fall within a selected period. The selected period may be the past three months. The selector module 92 may write the received entity IDs to a selection log file 94. The selection log file 94 may be implemented as a Kafka topic. In one embodiment, the selection log file 94 may be mirrored to database server 50.

[0078] The warm-up module 96 may be an application running on the database server 50 and configured to read the selected log file 94. If the data conflict log file 80 is implemented as a Kafka topic, the warm-up module 96 may be implemented as a consumer subscribing to this topic. In response to reading the selected log file 94, the warm-up module 96 may prompt the database server 50 to generate and store associated mapped IDs in the database 54 by sending a lookup request to the interface module 56 of the database server 50 for each of the entity IDs in the selected log file 94, as described above with reference to FIGS. 1 and 7 . In one embodiment, the warm-up module 96 may also be configured to receive entity IDs recently generated (e.g., the previous day) by, for example, a web application. In response to receiving the recently generated entity IDs, the warm-up module 96 may prompt the database server 50 to generate and store associated mapped IDs in the database 54 by sending a lookup request to the interface module 56 of the database server 50 for each of the entity IDs. This improves the stability of the bidirectional lookup service and reduces delays caused by large numbers of write requests because the database is pre-populated with entity IDs that have been recently used or created and are therefore most likely to be requested by users in the near future. Although this process has been described with reference to database 50, those skilled in the art will appreciate that various modifications can be made to the above embodiment without departing from the scope of the present invention, for example, the process could be applied to all database servers in a network.

[0079] Figure 11 is a block diagram illustrating a technical architecture 500 of a server capable of executing some or all of the methods of Figures 1, 2, 5, 6, 9, or 10. The technical architecture includes a processor 522 (which may also be referred to as a central processing unit or CPU) in communication with memory devices including secondary storage 524 (e.g., a disk drive), read-only memory (ROM) 526, and random access memory (RAM) 528. The processor 522 may be implemented as one or more CPU chips. The technical architecture may further include input / output (I / O) devices 530 and network connection devices 532.

[0080] Secondary storage 524 typically includes one or more disk or tape drives and is used for non-volatile storage of data and as overflow data storage in the event that RAM 528 is not large enough to hold all working data. Secondary storage 524 may also be used to store programs that are loaded into RAM 528 when selected for execution.

[0081] In this embodiment, secondary storage 524 includes an order processing component 524a that includes non-transitory instructions operable by processor 522 to perform various operations of the disclosed methods. ROM 526 is used to store instructions, and possibly data, that are read during program execution. Secondary storage 524, RAM 528, and / or ROM 526 may also be referred to as computer-readable storage media and / or non-transitory computer-readable media.

[0082] The I / O devices 530 may include a printer, a video monitor, a liquid crystal display (LCD), a plasma display, a touch screen display, a keyboard, a keypad, a switch, a dial, a mouse, a trackball, a voice recognition device, a card reader, a paper tape reader, or other known input devices.

[0083] The processor 522 executes instructions, code, computer programs, and scripts that it accesses from a hard disk, floppy disk, optical disk (any of these various disk-based systems may be considered secondary storage 524), flash drive, ROM 526, RAM 528, or network-connected device 532. Although only one processor 522 is shown, there may be multiple processors. Thus, while instructions may be discussed as being executed by a processor, the instructions may be executed simultaneously, serially, or otherwise by one or more processors.

[0084] While the technical architecture has been described with reference to computers, it should be understood that the technical architecture may be formed by two or more computers that cooperate to perform tasks and communicate with each other. By way of example and not limitation, an application may be partitioned to allow simultaneous and / or parallel processing of the application's instructions. Alternatively, data processed by the application may be partitioned to allow simultaneous and / or parallel processing of different portions of the data set by two or more computers. In one embodiment, virtualization software may be employed by technical architecture 500 to provide the functionality of multiple servers that is not directly bound by the number of computers in technical architecture 500. In one embodiment, the functionality disclosed above may be provided by running one or more applications in a cloud computing environment. Cloud computing may involve providing computing services over a network connection using dynamically scalable computing resources. A cloud computing environment may be established by an enterprise and / or rented as needed from a third-party provider.

[0085] Programming and / or loading executable instructions into the technical architecture modifies at least one of CPU 522, RAM 528, and ROM 526, transforming a portion of the technical architecture into a specific purpose machine or apparatus having the novel functionality embraced by this disclosure. It is fundamental to the fields of electrical engineering and software engineering that the functionality that can be achieved by loading executable software into a computer can be transformed into a hardware implementation using known design rules.

[0086] While exemplary embodiments have been described above, those skilled in the art will recognize that many variations of the embodiments are possible within the scope and spirit of the invention.

Claims

1. 1. A computer-implemented method comprising: a) receiving respective primary entity IDs from users, each primary entity ID comprising a first bit string and a second bit string; From each of the primary entity IDs: i) each mapped ID is unique; ii) such that each of the mapped IDs includes the first bit string of an associated primary entity ID and a third bit string that is different from the second bit string of the associated primary entity ID; b) generating each of said mapped IDs; 10. A computer-implemented method comprising:

2. the first bit string has a bit length N, and the primary entity ID and the mapped ID include the first bit string as the least significant N bits; The method of claim 1.

3. The bit length N of each first bit string is 10, and the bit length of each mapped ID is 64.

3. The method according to claim 1 or 2.

4. maintaining a counter, wherein the third bit string of the mapped ID includes counter information obtained from the counter.

10. A method according to any one of the preceding claims.

5. for each primary entity ID, generating a shard key from the primary entity ID or an associated mapped ID using a hash function; The hash function generates the shard key based on the value of the first bit string.

10. A method according to any one of the preceding claims.

6. for each primary entity ID, storing the primary entity ID, the associated mapped ID, and the shard key in the same row in a database; The method of claim 5 further comprising:

7. the database is configured as a partitioned database including a plurality of shards, each shard associated with a different shard key value, and storing the primary entity ID, the mapped ID, and the shard key in the same row in the database comprises storing the primary entity ID, the mapped ID, and the shard key in the same row in the shard associated with the shard key value; The method of claim 6.

8. 1. A computer-implemented method for resolving conflicts between a first database and a second database, comprising: The first database and the second database each have a plurality of rows, each row storing an entity ID and a mapped ID associated with the entity ID, and the method includes: (a) storing a new entity ID and a new mapped ID associated with said entity ID in a new row of said first database; (b) determining conflicts between the new entity ID and the new mapped ID and existing entity IDs and associated existing mapped IDs stored in the second database; (c) determining a valid mapped ID associated with the new entity ID; (d) updating the second database based on the new entity ID and the associated valid mapped ID, subject to determining that the valid mapped ID and the existing mapped ID stored in the second database are different; A method comprising:

9. step (b) comprising determining that the new entity ID and the existing entity ID are the same and that the new mapped ID and the existing mapped ID are different; The method of claim 8.

10. a third database having a further plurality of rows, each row storing an entity ID and a mapped ID associated with said entity ID; and step (d) comprising: retrieving from the third database another mapped ID associated with the new mapped ID; selecting one of the new mapped ID, the existing mapped ID stored in the second database, and the other mapped ID retrieved from the third database as the valid mapped ID associated with the new entity ID.

10. The method according to claim 8 or 9.

11. The step of selecting a valid mapped ID comprises: determining a counter value for each of the new mapped ID, the existing mapped ID stored in the second database, and the other mapped ID retrieved from the third database; and selecting the mapped ID with the smallest counter value as the valid mapped ID. The method of claim 10.

12. the first database, the second database and the third database further store in each row a creation timestamp that identifies a time when the mapped entity ID stored in the row was created, and the step of selecting a valid mapped ID comprises: selecting the new mapped ID, the existing mapped ID stored in the second database, and the other mapped ID retrieved from the third database that has the earliest timestamp; The method of claim 10.

13. (e) updating the first database based on the new entity ID and the associated valid mapped ID, provided that the valid mapped ID and the new mapped ID are determined to be different; (f) upon determining that the valid mapped ID and the other mapped ID retrieved from the third database are different, updating the third database based on the new entity ID and the associated valid mapped ID; The method of any one of claims 8 to 12, further comprising:

14. an initial step of generating said new mapped ID from said new entity ID according to the method of any one of claims 1 to 7; The method of any one of claims 8 to 13, further comprising:

15. 1. A computer-implemented method for processing, by a database server, a request from a client device to provide a mapped ID associated with a primary entity ID, the method comprising: The database server stores a primary copy of a database having a plurality of rows, each row including an entity ID of a plurality of entity IDs and a mapped ID associated with the entity ID, the database server further stores a first cached copy and a second cached copy of the database, and the method includes: (a) receiving the request from the client device, the request including the primary entity ID and service information specifying whether the request is associated with a first service or a second service; As a condition for determining that the service information specifies that the request is associated with the first service, determining whether the mapped ID associated with the primary entity ID is stored in the first cached copy; (b2) determining whether the mapped ID associated with the primary entity ID is stored in the primary copy of the database, conditional on determining that the mapped ID is not stored in the first cached copy; (b3) conditioned on determining that the mapped ID is stored in one of the first cached copy and the primary copy of the database, obtaining the mapped ID and providing the mapped ID to the client device; and step (b) comprising: As a condition for determining that the service information specifies that the request is associated with the second service, determining whether the mapped ID associated with the primary entity ID is stored in the second cached copy; (c2) determining whether the mapped ID associated with the primary entity ID is stored in the primary copy of the database, conditional on determining that the mapped ID is not stored in the second cached copy; (c3) conditioned on determining that the mapped ID is stored in one of the second cached copy and the primary copy of the database, obtaining the mapped ID and sending the mapped ID to the client device; and step (c) comprising: conditional on determining that the mapped ID is not stored in the primary copy of the database; generating the mapped ID associated with the primary entity ID; (d2) sending the mapped ID to the client device; storing (d3) said primary entity ID and said associated mapped ID in a new row in said database; and step (d) comprising: A method comprising:

16. the first cached copy and the second cached copy completely cache the database, and the database server further stores a first partial cached copy of the first cached copy and a second partial cached copy of the second cached copy; Step (b1) of the method comprises: determining whether the mapped ID associated with the primary entity ID is stored in the first partial cached copy; determining whether the mapped ID associated with the primary entity ID is stored in the first cached copy, conditional on determining that the mapped ID is not stored in the first partial cached copy; Step (c1) of the method comprises: determining whether the mapped ID associated with the primary entity ID is stored in the second partial cached copy; determining whether the mapped ID associated with the primary entity ID is stored in the second cached copy, conditional on determining that the mapped ID is not stored in the second partial cached copy; 16. The method of claim 15.

17. 1. A computer-implemented method comprising: (a) receiving a first request from a client device, the first request including a primary entity ID, for providing a mapped ID associated with the primary entity ID to an application; (b) retrieving from a database the mapped ID associated with the primary entity ID; (c) providing the mapped ID to the application; (d) processing the primary entity ID and the mapped ID with an encryption key to generate an encrypted primary entity ID and an encrypted mapped ID; (e) sending to the client device the encrypted entity ID, the encrypted mapped ID, and expiration information specifying an expiration date for the encrypted entity ID and the encrypted mapped ID; 11. A computer-implemented method comprising:

18. (f) receiving a second request from the client device to provide the mapped ID associated with the primary entity ID to the application, the second request including the primary entity ID, the encrypted entity ID, the encrypted mapped ID, and expiration information; (g) determining whether the encrypted primary entity ID and the encrypted mapped ID are valid based on the expiration information; (h) upon determining that the encrypted primary entity ID and the encrypted mapped ID are valid, decrypting the received encrypted primary entity ID and encrypted mapped ID using the encryption key and providing the decrypted mapped ID to the application; 20. The method of claim 17, further comprising:

19. one or more computers; one or more storage devices storing instructions that, when executed by said one or more computers, cause said one or more computers to carry out the method of any one of claims 1 to 18; A system comprising:

20. storing instructions which, when executed by one or more computers, cause said one or more computers to perform the operations of any one of claims 1 to 18; One or more non-transitory computer storage media.

21. A system including a first database server and a second database server, the first database server and the second database server store a first database and a second database, respectively, each of the databases having a plurality of rows, each row storing an entity ID and a mapped ID associated with the entity ID; The first database server storing the new entity ID and the new mapped ID associated with said entity ID in a new row in said first database; configured to send the new entity ID and the new mapped ID to the second database server; The second database server determining conflicts between the received new entity ID and the new mapped ID and existing entity IDs and associated existing mapped IDs stored in the second database; determining a valid mapped ID associated with the new entity ID; configured to update the second database based on the new entity ID and the associated valid mapped ID, upon determining that the valid mapped ID and the existing mapped ID stored in the second database are different. system.

22. the second database server is configured to resolve conflicts between the received new entity ID and new mapped ID and existing entity IDs and associated existing mapped IDs stored in the second database by determining that the received new entity ID and existing entity ID are the same and that the received new mapped ID and existing mapped ID are different.

22. The system of claim 21.

23. a third database server having a third database stored thereon, the third database having another plurality of rows, each row storing an entity ID and a mapped ID associated with the entity ID; the first database server is further configured to send the new entity ID and the new mapped ID to the third database server; The second database server retrieving from the third database server another mapped ID associated with the new mapped ID and stored in the third database; selecting one of the new mapped ID received from the first database server, the existing mapped ID stored in the second database server, and the other mapped ID retrieved from the third database server as the valid mapped ID associated with the new entity ID, thereby configured to determine a valid mapped ID associated with the new entity ID; 23. A system according to claim 21 or 22.

24. The second database server determining a counter value for each of the new mapped ID received from the first database server, the existing mapped ID stored in the second database server, and the other mapped ID retrieved from the third database server, and selecting the mapped ID with the lowest counter value as the valid mapped ID; configured to select the valid mapped ID; 24. The system of claim 23.

25. the first database, the second database and the third database are further configured to store in each row a creation timestamp identifying a time when the mapped entity ID stored in the row was created; The first database server, the second database server, and the third database server are further configured to, when transmitting the new entity ID, the new mapped ID, and the another mapped ID, respectively, transmit a corresponding creation timestamp to the second database server, and the second database server: selecting a mapped ID with the earliest timestamp from the new mapped ID received from the first database server, the existing mapped ID stored in the second database server, and the other mapped ID retrieved from the third database server; configured to select the valid mapped ID; 24. The system of claim 23.

26. the second database server is further configured to send the new entity ID and the valid mapped ID to the first database server and a third database server; the first database server is further configured to, upon determining that the valid mapped ID and the new mapped ID are different, update the first database based on the new entity ID and the associated valid mapped ID; the third database server is further configured to, upon determining that the valid mapped ID and the other mapped ID are different, update the third database based on the new entity ID and the associated valid mapped ID. A system according to any one of claims 21 to 25.

27. The first database server is further configured to generate the new mapped ID from the new entity ID according to the method of any one of claims 1 to 8. A system according to any one of claims 21 to 26.

28. a database server for processing a request from a client device to provide a mapped ID associated with a primary entity ID, a primary copy of a database having a plurality of rows, each row including an entity ID of a plurality of entity IDs and a mapped ID associated with the entity ID; a first cached copy and a second cached copy of the database; is set to remember moreover, receiving the request from the client device, the request including the primary entity ID and service information specifying whether the request is associated with a first service or a second service; As a condition for determining that the service information specifies that the request is associated with the first service, determining whether the mapped ID associated with the primary entity ID is stored in the first cached copy; determining whether the mapped ID associated with the primary entity ID is stored in the primary copy of the database, conditional on determining that the mapped ID is not stored in the first cached copy; conditioned on determining that the mapped ID is stored in one of the first cached copy and the primary copy of the database, obtain the mapped ID and provide the mapped ID to the client device; As a condition for determining that the service information specifies that the request is associated with the second service, determining whether the mapped ID associated with the primary entity ID is stored in the second cached copy; determining whether the mapped ID associated with the primary entity ID is stored in the primary copy of the database, conditional on determining that the mapped ID is not stored in the second cached copy; conditioned on determining that the mapped ID is stored in one of the second cached copy and the primary copy of the database, obtaining the mapped ID and transmitting the mapped ID to the client device; conditional on determining that the mapped ID is not stored in the primary copy of the database; generating the mapped ID associated with the primary entity ID; sending the mapped ID to the client device; configured to store the primary entity ID and the associated mapped ID in a new row in the database. Database server.

29. the first cached copy and the second cached copy completely cache the database, and the database server further stores a first partial cached copy of the first cached copy and a second partial cached copy of the second cached copy; The database server determining whether the mapped ID associated with the primary entity ID is stored in the first partial cached copy; determining whether the mapped ID associated with the primary entity ID is stored in the first cached copy, conditional on determining that the mapped ID is not stored in the first partial cached copy; and is configured to determine whether the mapped ID associated with the primary entity ID is stored in the first cached copy by The database server determining whether the mapped ID associated with the primary entity ID is stored in the second partial cached copy; determining whether the mapped ID associated with the primary entity ID is stored in the second cached copy, conditional on determining that the mapped ID is not stored in the second partial cached copy; and is configured to determine whether the mapped ID associated with the primary entity ID is stored in the second cached copy by 29. The database server of claim 28.

Images