A method for achieving consistency between cached data and database data

Abstract

The present invention discloses a method for achieving consistency between cached data and database data. Initialize a memory queue, create a listener for the memory queue and start it; when an update request is received, increment the modification count of the data by 1 and add it to the end of the memory queue; when a query data request is received, if it is not in the process of being updated, directly query the cache. If the data is not in the cache, query the database and save the data to the cache; if it is in the process of being updated, add it to the end of the memory queue; when the listener detects that there is data to be processed in the memory queue, execute the update data process or the query data process according to the request type. The present invention utilizes queue technology. When there are a large number of concurrent user requests and there are both updated data and query data at the same time, it converts the concurrent requests into sequential queued processing, avoiding the situation of inconsistent cached data and database data, and improving the reliability and stability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and specifically, to a method for achieving consistency between cached data and database data. Background Art

[0002] To improve system performance and support concurrent access of a large number of users, the system generally uses caching technology, and the industry commonly uses redis middleware to cache data. Using redis caching can reduce the read operations of disk I / O, relieve the pressure on disk I / O, improve the availability and scalability of the database, relieve the pressure on the database, and improve the stability of the system.

[0003] The commonly used methods for querying and updating the database in the prior art are based on the edge caching mode, i.e., the Cache-Aside Pattern. The specific process is as follows: When a user requests to query data, first query the cache. If there is data in the cache, directly return it. If the data is not found in the cache, query the database, put the queried data into the cache, and finally return it. When updating the data in the database, first update the database data, and then delete the data in the cache. Since the cache and the database are two different storage systems, in the following scenarios, when querying and updating data, the data in the cache and the data in the database may be inconsistent, resulting in system data errors, reducing the reliability of the system, and affecting the normal use of users:

[0004] 1. Scenario of failed cache deletion: When updating data, if the database is successfully updated but the cache deletion fails, at this time, the data in the cache and the data in the database are inconsistent, and when the user queries the data, incorrect data is always obtained from the cache.

[0005] 2. Database read-write separation scenario: When the database has a read-write separation architecture, i.e., data modification is performed on the master database and data query is performed on the slave database, and the master database synchronizes data to the slave database regularly. When user A updates a certain piece of data in the master database and deletes the cache, if the data in the master database has not been synchronized to the slave database yet, at this time, when user B queries this piece of data, first query the cache, find that it has been deleted, then query the slave database, obtain the old data, and then write the old data into the cache, thus causing the situation where the cache data and the database data are inconsistent. Similarly, when the user queries the data, incorrect data is always obtained from the cache.

[0006] 3. Cache invalidation scenario: When user A initiates Request 1 - to update a certain piece of data and has not completed the update yet, user B initiates Request 2 - to query that piece of data. If the data in the cache has expired, then Request 2 will directly query the old data in the database. At this time, if Request 1 finishes updating the database and has executed the cache deletion operation, and then Request 2 continues to execute, the old data will be written back into the cache. This results in the inconsistency between the cache data and the database data. Similarly, when users query data, they always get incorrect data from the cache. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for achieving consistency between cache data and database data, which is used to solve the problem of inconsistency between cache data and database data in the existing methods for updating data and querying data.

[0008] The present invention solves the above problems through the following technical solutions:

[0009] A method for achieving consistency between cache data and database data includes:

[0010] Step S1: Initialize the memory queue when the system starts, create a listener for the memory queue, and start the listener;

[0011] Step S2: When a data update request is received, use a write lock to determine whether the modification count of the data is empty. If it is empty, initialize the data modification count to 1; otherwise, increment the data modification count by 1, and add the data and the request type to the end of the memory queue. When a data query request is received, use a read lock to determine whether the modification count of the data is 0 or empty. If so, query the cache data. If the data is not in the cache, query the database, return the data to the user, and save the data to the cache at the same time. Otherwise, add the data unique identifier and the request type to the end of the memory queue;

[0012] Step S3: When the listener detects that there is data to be processed in the memory queue, take out the data in the memory queue, and execute the data update process or the data query process according to the request type, where:

[0013] The data update process is: delete the cache data, then update the database, and then use a write lock to decrement the modification count of the data by 1, and delete the data and the request type from the memory queue;

[0014] The data query process is: query the cache data. If the data is not in the cache, query the database, save the data to the cache, and delete the data unique identifier and the request type from the memory queue.

[0015] In the present invention, when the system first receives a query data request, the modification count of the data is empty, indicating that the data is not in the update process. Then, the cache is directly read to improve the concurrent processing ability.

[0016] When the system receives a request to update data, the data and the request type (update data or query data) are placed at the end of the memory queue. At the same time, by setting the modification count of the data to 1, the status of the data is marked as being updated. When the queue listener discovers that there is data in the queue, the data is processed in the order of first in, first out. First, the cached data is deleted, and then the database data is updated. After the database update is completed, the modification count of the data is set to 0, and the data is deleted from the memory queue. When the database is updated and the system receives a query data request for this data, the modification count of the data is first judged at this time. Since the modification count is 0 at this time, the cache is queried. Since the cached data was deleted during the update process, there is no such data in the cache at this time. The database is queried, and the updated data is retrieved and saved to the cache. At this time, the data in the cache and the database is consistent. If the query data request precedes the update data request, the modification count of the data is empty at this time, and the cache is also queried. Since the cached data was deleted during the update process, there is no such data in the cache at this time. The database is queried, and the updated data is retrieved and saved to the cache. At this time, the data in the cache and the database is consistent.

[0017] Suppose that during the data update process, when the system receives a query data request, since the modification count of the data is still 1 at this time, it indicates that the data is still in the update state. At this time, the unique data identifier and the request type are added to the end of the memory queue and also enter the memory queue for queued processing. When the data update request is completed, the listener retrieves the data from the memory queue and queries the cached data according to the query data request. Since the cached data is first deleted during the update process, there is no such data in the cache at this time. Then the database is queried, and the updated data in the database is retrieved and the updated data is saved to the cache. The database data and the cached data are consistent.

[0018] When the system starts, multiple memory queues are initialized, and a listener is created for each memory queue. Before adding the data and the request type, or the unique data identifier and the request type to the end of the memory queue, it is also necessary to calculate the memory queue to which the data needs to be placed using the hash algorithm based on the unique data identifier of the data. The query data requests and update data requests for the same data are in the same memory queue and are executed in the order of first in, first out. Using the queue technology, the concurrent requests are converted into queued processing one by one, ensuring the consistency of the cached data and the database data in the concurrent scenario and improving the reliability and stability of the system. For requests for different data, they are stored in different memory queues, which can realize the processing of user concurrent requests and improve the processing speed and efficiency.

[0019] In the described updated data process, when the cache deletion fails or the database update fails after the cache deletion is successful, the write lock decrements the modification count of the data by 1, deletes the data and the request type from the memory queue, the system directly returns to step S2, and feeds back that the data update fails.

[0020] In the updated data process, when the cache deletion fails, the subsequent steps will no longer be executed. The data still exists in the cache, and the data in the database is still the data before the update. The data in the cache is consistent with the data in the database. The next time a data query request is executed, since the modification count of the data has been modified to 0, the cache is queried, and the data obtained is still the data before the update. When the cache deletion is successful and the database update fails in the updated data process, the subsequent steps will no longer be executed. The data in the cache is empty, and the modification count of the data has been modified to 0. If a data query request is encountered thereafter, the cache data is queried. At this time, the cache data is empty, so the database is queried, and the data queried from the database is saved to the cache, and the two are still consistent.

[0021] In the updated data process, when the cache deletion fails or the database update fails after the cache deletion is successful, if a data update request is encountered thereafter, it will be executed normally.

[0022] When there are a large number of concurrent user requests and there are both updated data and queried data, the present invention avoids the situation of inconsistent cache data and database data by queuing and processing sequentially in order, and solves the problem of inconsistent cache data and database data caused by cache deletion failure, master-slave database synchronization delay and cache invalidation in the prior art.

[0023] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0024] (1) The invention uses queue technology. When there are a large number of concurrent user requests and there are both updated data and queried data, the concurrent requests are converted into sequential queuing processing, avoiding the situation of inconsistent cache data and database data, and improving the reliability and stability of the system.

[0025] (2) The present invention uses a hash algorithm to place each piece of data in the same memory queue, and different data are placed in different memory queues, improving the concurrent processing ability and processing efficiency.

[0026] (3) The present invention solves the problem of inconsistent cache data and database data caused by cache deletion failure, master-slave database synchronization delay and cache invalidation in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a principle block diagram of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0028] The present invention will be further described in detail below in conjunction with embodiments, but the implementation manners of the present invention are not limited thereto.

[0029] Embodiment 1:

[0030] Combined with the attached Figure 1 As shown, a method for realizing the consistency between cached data and database data includes:

[0031] Step S1: Initialize the memory queue when the system starts, create a listener for the memory queue, and start the listener;

[0032] Step S2: When a request for updating data is received, use a write lock to determine whether the modification count of the data is empty. If it is empty, initialize the modification count of the data to 1; otherwise, increment the modification count of the data by 1, and add the data and the request type to the end of the memory queue. When a request for querying data is received, use a read lock to determine whether the modification count of the data is 0 or empty. If so, query the cached data. If the data is not in the cache, query the database, return the data to the user, and save the data to the cache at the same time. Otherwise, add the unique identifier of the data and the request type to the end of the memory queue;

[0033] Step S3: When the listener detects that there is data to be processed in the memory queue, take out the data in the memory queue, and execute the update data process or the query data process according to the request type, where:

[0034] The update data process is: delete the cached data, then update the database, and then use a write lock to decrement the modification count of the data by 1, and delete the data and the request type from the memory queue;

[0035] The query data process is: query the cached data. If the data is not in the cache, query the database, and save the data to the cache, and delete the unique identifier of the data and the request type from the memory queue.

[0036] In the present invention, when the system first receives a request for querying data and the modification count of the data is empty, it indicates that the data is not in the update process, and the cache is directly read to improve the concurrent processing ability.

[0037] When the system receives a request for updated data, it puts the data and the request type (whether it is for updating data or querying data) at the end of the memory queue. At the same time, it sets the modification count of the data to 1 to indicate that the status of this piece of data is being updated. When the queue listener detects that there is data in the queue, it processes the data in the order of first in, first out. First, it deletes the cached data, and then it updates the database data. After the database update is completed, it simultaneously sets the modification count of the data to 0 and deletes the data from the memory queue. When the database is updated and the system receives a query data request for this data, it first checks the modification count of the data. At this time, the modification count is 0, so it queries the cache. Since the cached data was deleted during the update process, there is no such data in the cache at this time. It then queries the database, retrieves the updated data, and saves the data to the cache. At this time, the data in the cache and the database are consistent. If the query data request arrives before the update data request, the modification count of the data is empty at this time. Similarly, it queries the cache. Since the cached data was deleted during the update process, there is no such data in the cache at this time. It then queries the database, retrieves the updated data, and saves the data to the cache. At this time, the data in the cache and the database are consistent.

[0038] Suppose during the data update process, the system receives a query data request. Since the modification count of the data is still 1 at this time, it means that the data is still in the update state. At this time, it adds the data unique identifier and the request type to the end of the memory queue and also enters the memory queue for queued processing. After the data update request is executed, the listener retrieves the data from the memory queue and queries the cached data according to the query data request. Since the cached data was first deleted during the update process and there is no such data in the cache at this time, it then queries the database. At this time, it retrieves the updated data from the database and saves the updated data to the cache. The data in the database and the cache are consistent.

[0039] When the system starts up, it initializes multiple memory queues and creates a listener for each memory queue. Before adding the data and the request type, or the data unique identifier and the request type, to the end of the memory queue, it also needs to calculate the memory queue into which the data needs to be placed using the hash algorithm based on the data unique identifier of the data. The query data requests and update data requests for the same data are in the same memory queue and are executed in the order of first in, first out. Using the queue technology, it converts concurrent requests into sequential queued processing, ensuring the consistency of the cached data and the database data in a concurrent scenario and improving the reliability and stability of the system. For requests for different data, they are stored in different memory queues, which can achieve the processing of user concurrent requests and improve the processing speed and efficiency.

[0040] In the above update data process, when the cache deletion fails or the database update fails after successful cache deletion, the write lock decrements the modification count of the data by 1, deletes the data and the request type from the memory queue, the system directly returns to step S2, and feedbacks that the data update fails.

[0041] In the update data process, when the cache deletion fails, the subsequent steps will not be executed anymore. The data still exists in the cache, and the data in the database remains the data before the update. The data in the cache is consistent with the data in the database. The next time a data query request is executed, since the modification count of this data has been modified to 0, the cache is queried, and the data obtained is still the data before the update. When the cache is successfully deleted but the database update fails in the update data process, the subsequent steps will not be executed anymore. The data in the cache is empty, and the modification count of this data has been modified to 0. If a data query request is encountered thereafter, the cache data is queried. At this time, the cache data is empty, so the database is queried, and the data queried from the database is saved to the cache, and the two are still consistent.

[0042] In the update data process, when the cache deletion fails or the database update fails after successful cache deletion, if a data update request is encountered thereafter, it will be executed normally.

[0043] In the case of a large number of concurrent user requests, with both data update and data query existing simultaneously, the present invention avoids the situation of inconsistent cache data and database data by queuing and processing sequentially in order, and solves the problem of inconsistent cache data and database data caused by cache deletion failure, master-slave database synchronization delay, and cache invalidation in the prior art.

[0044] Although the present invention has been described herein with reference to its explanatory embodiments, the above embodiments are only the preferred embodiments of the present invention. The embodiments of the present invention are not limited by the above embodiments. It should be understood that those skilled in the art can design many other modifications and embodiments, and these modifications and embodiments will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A method for achieving consistency between cached data and database data, characterized in that, Including: Step S1: Initialize the memory queue when the system starts, create a listener for the memory queue, and start the listener. Step S2: When a data update request is received, use a write lock to determine whether the modification count of the data is empty. If it is empty, initialize the modification count of the data to 1; otherwise, increment the modification count of the data by 1, and add the data and the request type to the end of the memory queue. When a data query request is received, use a read lock to determine whether the modification count of the data is 0 or empty. If so, query the cached data. If the data is not in the cache, query the database, return the data to the user, and save the data to the cache at the same time. Otherwise, add the data unique identifier and the request type to the end of the memory queue. Step S3: When the listener detects that there is data to be processed in the memory queue, retrieve the data from the memory queue, and execute the data update process or the data query process according to the request type, where: The data update process is: delete the cached data, then update the database, and then use a write lock to decrement the modification count of the data by 1, and delete the data and the request type from the memory queue. The data query process is: query the cached data. If the data is not in the cache, query the database, and save the data to the cache, and delete the data unique identifier and the request type from the memory queue.

2. The method for implementing the consistency between cached data and database data according to claim 1, characterized in that, When the system starts, initialize multiple memory queues, create a listener for each memory queue respectively. Before adding the data and the request type, or the data unique identifier and the request type to the end of the memory queue, it is also necessary to calculate the memory queue to which the data needs to be placed using a hash algorithm based on the data unique identifier of the data.

3. A method for achieving consistency between cached data and database data according to claim 1 or 2, characterized in that, In the data update process described above, when the deletion of the cached data fails or the update of the database fails after the deletion of the cached data is successful, the write lock decrements the modification count of the data by 1, deletes the data and the request type from the memory queue, the system directly returns to Step S2, and feedbacks that the data update fails.

Images