A cache database hotspot data identification method and system

Abstract

The application provides a cache database hotspot data identification method and system. When a first server builds a Session, a Hash linked list is created for a client. The Hash linked list stores a Hash value of a web page address, a number of times of accessing the web page, and a time of staying on the web page. The Hash linked list is destroyed along with the Session. The first server periodically traverses all Sessions on the first server, and sends a Hash linked list corresponding to a Session in a valid period to a second server. The Hash linked list is emptied after the Hash linked list is sent. The second server pre-processes the Hash linked list, and obtains N web page addresses with the most number of times of accessing according to the pre-processed Hash linked list. The second server obtains hotspot data according to the N web page addresses, and informs a cache database to load the hotspot data to the cache database. The application can discover the hotspot data in time and load the hotspot data to the cache database under the condition of occupying very few server resources.

Description

Technical Field

[0001] This invention relates to the field of caching database technology, and in particular to hot data detection in caching databases. Background Technology

[0002] A session is a method used by web servers to maintain session state. When a client and server establish a connection, the server creates a new session for the client in memory. Each session has a unique ID and stores information such as login details, thus ensuring session persistence between the client and server. When the client and server disconnect or after a period of inactivity, the session is destroyed.

[0003] Redis and memcached are commonly used open-source caching databases that store data in memory in the form of key-value pairs. They are used to cache data queried from MySQL databases, so that when other users perform queries, they can directly retrieve the results from Redis or memcached without repeatedly reading from MySQL. The principle is similar to CPU caching; CPU reads from the cache much faster than disk I / O, significantly saving read time. Redis and memcached reside in memory; when MySQL needs to be read, Redis and memcached cache the data in memory. Furthermore, since accessing memory is much faster than directly accessing MySQL, it effectively improves access speed.

[0004] Redis and memcached servers have limited memory, and they cannot cache all MySQL data. Therefore, it's necessary to periodically remove data from Redis and memcached that is not frequently accessed. However, accurately identifying the data to be evicted directly impacts the performance of Redis and memcached. If Redis and memcached store low-access data, it will lead to direct requests to MySQL, especially during peak sales or promotional events, potentially causing cache breakdown and cache penetration. This raises the issue of identifying "hot data." Hot data is essentially data with high access volume or frequency over a given period. Adding hot data to Redis and memcached promptly can reduce MySQL access, but if the hot data set is inaccurate, Redis and memcached will not significantly alleviate MySQL pressure. Identifying hot data is a critical problem that needs to be solved when using Redis and memcached. Summary of the Invention

[0005] This invention provides a method and system for identifying hot data in a cached database. By utilizing the characteristic that a new Session is created when a connection is established between the client and the server, a hash list is created and destroyed synchronously with the Session. Without consuming too many server resources, the method can obtain the hot web pages currently accessed by the client on the server, and thus obtain the hot data, effectively solving the problem of hot data discovery in a cached database.

[0006] On the one hand, the present invention provides a method for identifying hot data in a cache database, comprising the following steps:

[0007] When the first server creates a session for the client, it triggers the creation of a hash list for the client. The hash list stores the hash value of the web page address, the number of times the web page is accessed, and the cumulative time spent on the web page. The hash list is destroyed when the session is destroyed.

[0008] The first server periodically traverses all sessions on the first server and sends the hash list corresponding to the valid sessions to the second server. After sending the hash list, the hash list is cleared.

[0009] The second server preprocesses the hash chain and analyzes the preprocessed hash chain to obtain the N most accessed web page addresses. The second server obtains hot data based on the N web page addresses and notifies the cache database to load the hot data into the cache database.

[0010] Preferably, the second server preprocesses the hash list by deleting web page addresses whose cumulative dwell time on the web page is not within a preset time range, as well as web page addresses set by the administrator, from the hash list.

[0011] Preferably, the step of analyzing the preprocessed hash list to obtain the N most accessed web page addresses specifically involves: the second server storing the preprocessed hash list from the previous M times, and statistically analyzing the hash list sent this time and the preprocessed hash list from the previous M times to obtain the N most accessed web page addresses.

[0012] Preferably, the second server obtains hotspot data based on the N webpage addresses by: analyzing the webpages corresponding to the N webpage addresses, obtaining the database statements called by the corresponding webpages in the first server, and determining the hotspot data based on the database statements called in the first server.

[0013] Preferably, the cache database is memcached or Redis, and the hash list is a HashMap.

[0014] On the other hand, the present invention also provides a cache database hotspot data identification system, comprising the following modules:

[0015] The Hash list management module is used to trigger the first server to create a Hash list for the client when the first server creates a Session for the client. The Hash list stores the Hash value of the web page address, the number of times the web page is accessed, and the cumulative time spent on the web page. The Hash list is destroyed when the Session is destroyed.

[0016] The sending module is used by the first server to periodically traverse all Sessions on the first server and send the Hash list corresponding to the valid Sessions to the second server. After sending the Hash list, the Hash list is cleared.

[0017] The hotspot data discovery module is used by the second server to preprocess the hash chain, analyze the preprocessed hash chain to obtain the N most accessed web page addresses, obtain hotspot data based on the N web page addresses, and notify the cache database to load the hotspot data into the cache database.

[0018] Preferably, the second server preprocesses the hash list by deleting web page addresses whose cumulative dwell time on the web page is not within a preset time range, as well as web page addresses set by the administrator, from the hash list.

[0019] Preferably, the step of analyzing the preprocessed hash list to obtain the N most accessed web page addresses specifically involves: the second server storing the preprocessed hash list from the previous M times, and statistically analyzing the hash list sent this time and the preprocessed hash list from the previous M times to obtain the N most accessed web page addresses.

[0020] Preferably, the second server obtains hotspot data based on the N webpage addresses by: analyzing the webpages corresponding to the N webpage addresses, obtaining the database statements called by the corresponding webpages in the first server, and determining the hotspot data based on the database statements called in the first server.

[0021] Furthermore, the present invention also provides a computer-readable storage medium for storing computer program instructions, characterized in that the computer program instructions implement the method described above when executed by a processor.

[0022] When a client accesses a server, the server creates a session for the client. This session is destroyed when the client disconnects or after a period of disconnection. This invention utilizes this characteristic of sessions to analyze hot data. The cache database hot data identification method and system provided by this invention have the following characteristics: they do not excessively consume server resources, are real-time, facilitate rapid discovery of hot data, avoid pressure on the database server, and improve system stability and user experience. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the existing technology structure;

[0025] Figure 2 This is a schematic diagram of the structure of the present invention;

[0026] Figure 3 This is a flowchart of Embodiment 1 of the present invention. Detailed Implementation

[0027] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0029] Example 1

[0030] like Figure 3 As shown, the present invention provides a method for identifying hot data in a cache database, comprising the following steps:

[0031] When the first server creates a session for the client, it triggers the creation of a hash list for the client. The hash list stores the hash value of the web page address, the number of times the web page is accessed, and the cumulative time spent on the web page. The hash list is destroyed when the session is destroyed.

[0032] The first server periodically traverses all sessions on the first server and sends the hash list corresponding to the valid sessions to the second server. After sending the hash list, the hash list is cleared.

[0033] The second server preprocesses the hash chain and analyzes the preprocessed hash chain to obtain the N most accessed web page addresses. The second server obtains hot data based on the N web page addresses and notifies the cache database to load the hot data into the cache database.

[0034] In one embodiment, the hash list is used as an attribute of the Session, and the hash list is manipulated through methods such as getHashLinked() and setHashLinked(). This eliminates the need to maintain the hash list separately, and automatically completes the creation and destruction of the hash list and the Session simultaneously.

[0035] When a client accesses a server via a URL, the server creates a session for that client. For example, if a client accesses a website through Internet Explorer, the web server creates a session for Internet Explorer. If the same website is then opened through Chrome, the web server also creates a new session for Chrome, regardless of whether Internet Explorer and Chrome are on the same device. Different sessions have different IDs, distinguished by their SessionId. When the browser is closed, the corresponding session is destroyed. If 1000 users access the site through a browser or app, the server creates 1000 sessions. Since the Session object only has a few properties, its memory usage is negligible.

[0036] The number of sessions maintained on the server side reflects the popularity of a website; more sessions are maintained, resulting in more sessions being created. This invention utilizes this characteristic of sessions to track the URLs visited by users. A new hash list is created, storing the hash value of the webpage address, the number of times the webpage is visited, and the cumulative time spent on the webpage. For example, if a product A is currently available for purchase, and its corresponding URL is http: / / www.qianggou.com / a, then the URL is hashed, resulting in a hash value of c00bdd57f. When user 1 visits this URL, a HashMap is created along with the session, storing the hash value, the number of visits, and the time spent on the URL. Similarly, if user 2 visits the same URL, a HashMap is also created with the creation of a new session, and if the user visits the URL multiple times during the session's lifespan, the number of visits is recorded. For example, before the purchase event starts, the user refreshes the page multiple times.

[0037] In another embodiment, the number of times the webpage is accessed can be weighted. This weighting involves determining a weight based on the cumulative time spent on the webpage and the number of times it is accessed. When sending the hash list to the second server, the number of accesses is multiplied by the weight. For example, if the cumulative time spent is 5 minutes and the number of accesses is 10, a weight can be determined based on these two parameters. When sending the hash list to the second server, the number of accesses to the corresponding URL is multiplied by the weight. There are various ways to determine the weight, such as using a tiered approach: an average time spent of 10 seconds has a weight of 1, an average time spent of 20 seconds has a weight of 2, etc. This can effectively identify users opening and refreshing the corresponding webpage multiple times before the start of a flash sale, improving the accuracy of hotspot data identification.

[0038] When 10,000 users participate in the flash sale, 10,000 HashMaps will be created. The web server will then periodically send all the HashMaps corresponding to the Session to a second server. The second server will then collect the results, obtain the hot web pages, and thus obtain the hot data. The cache database will then cache the hot data in the cache database.

[0039] Since a flash sale page may contain multiple links or database queries related to the product, such as redirecting to another product page that receives few or no clicks, simply treating all content on the flash sale page as hot data would inevitably include some non-hot data. Therefore, in one specific embodiment, the second server preprocesses the hash list by deleting webpage addresses whose cumulative dwell time on the webpage is outside a preset time range, as well as webpage addresses set by the administrator, from the hash list.

[0040] User access to trending data is a short-term, continuous process. However, the hash list sent by the first server only reflects the access situation within that period. To avoid data loss due to jitter or other factors causing a user to suddenly access the same data within a period, the actual trending data is not considered. In a specific embodiment, the step of analyzing the preprocessed hash list to obtain the N most accessed web page addresses is as follows: The second server saves the previous M preprocessed hash lists. Based on the hash list sent this time and the previous M preprocessed hash lists, the N most accessed web page addresses are statistically analyzed to obtain the most accessed web page addresses. The second server saves the hash lists sent by the first server for the previous M times before the currently received hash list. Assuming the first server sends the hash list at a rate of 1 minute and M is 5, if the current time is 11:59, the second server will save the hash lists sent by the first server at 11:54, 11:55, 11:56, 11:57, and 11:58. Based on the statistical analysis of the hash lists sent by the first server at 11:54, 11:55, 11:56, 11:57, 11:58, and 11:59, hot data can be obtained.

[0041] The 1-minute period mentioned above is only for ease of understanding of the invention. In practical applications, it can be set to 10 seconds, 30 seconds, etc., or the periodicity can be related to the speed of session creation, that is, the duration of the periodicity is dynamically changing. In one embodiment, the duration of the periodicity is inversely proportional to the speed of session creation, so that when a large number of users access a certain page, hot data will be cached in the cache database in a timely manner.

[0042] As time passes after the flash sale begins, users will close the webpage after participating, and their corresponding sessions and hash lists will be destroyed. The trending data related to this event will gradually cool down and eventually be removed from the cache database. Trending data for other events or popular products will be cached in the cache database.

[0043] In one specific embodiment, the second server obtains hotspot data based on the N webpage addresses. Specifically, it analyzes the webpages corresponding to the N webpage addresses to obtain the database statements called by the corresponding webpages in the first server, and determines the hotspot data based on the database statements called in the first server. The above is merely an example; other methods can also be used to obtain hotspot data, and this invention does not further limit these methods.

[0044] In one specific embodiment, the cache database is memcached or Redis, and the hash list is a HashMap.

[0045] Example 2

[0046] This invention provides a system for identifying hot data in a cached database, comprising the following modules:

[0047] The Hash list management module is used to trigger the first server to create a Hash list for the client when the first server creates a Session for the client. The Hash list stores the Hash value of the web page address, the number of times the web page is accessed, and the cumulative time spent on the web page. The Hash list is destroyed when the Session is destroyed.

[0048] The sending module is used by the first server to periodically traverse all Sessions on the first server and send the Hash list corresponding to the valid Sessions to the second server. After sending the Hash list, the Hash list is cleared.

[0049] The hotspot data discovery module is used by the second server to preprocess the hash chain, analyze the preprocessed hash chain to obtain the N most accessed web page addresses, obtain hotspot data based on the N web page addresses, and notify the cache database to load the hotspot data into the cache database.

[0050] In one embodiment, the Hash list management module and the sending module are located on the first server, and the hot data discovery module is located on the second server. Together, they constitute the cache database hot data identification system. (See attached specification) Figure 2 This is a schematic diagram of the structure of the present invention, including a client, a first server, a second server, a database server, and a cache database. It should be noted that the specification includes... Figure 2 This is just one example; in other examples, the first server could include multiple servers, such as a cluster of servers.

[0051] In one specific embodiment, the second server preprocesses the hash list by deleting web page addresses whose cumulative dwell time on the web page is not within a preset time range, as well as web page addresses set by the administrator, from the hash list.

[0052] In one specific embodiment, the step of analyzing the preprocessed hash list to obtain the N most accessed web page addresses specifically involves: the second server storing the preprocessed hash lists from the previous M times, and statistically analyzing the hash lists sent this time and the previous M preprocessed hash lists to obtain the N most accessed web page addresses.

[0053] In one specific embodiment, the second server obtains hotspot data based on the N webpage addresses by: analyzing the webpages corresponding to the N webpage addresses to obtain the database statements called by the corresponding webpages in the first server, and determining the hotspot data based on the database statements called by the first server.

[0054] Example 3

[0055] Furthermore, the present invention also provides a computer-readable storage medium for storing computer program instructions, characterized in that the computer program instructions implement the method described above when executed by a processor.

[0056] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for identifying hot data in a cached database, characterized in that, The steps include the following: When the first server creates a Session for the client, it triggers the creation of a hash list for the client. The hash list stores the hash value of the webpage address, the number of times the webpage is accessed, and the cumulative time spent on the webpage. The hash list is destroyed when the Session is destroyed. Specifically, the hash list is used as an attribute of the Session, and the getHashLinked() and setHashLinked() methods are used to operate on the hash list. The first server periodically traverses all sessions on the first server and sends the hash list corresponding to the valid sessions to the second server. After sending the hash list, the hash list is cleared. The second server preprocesses the hash chain and analyzes the preprocessed hash chain to obtain the N web page addresses with the most visits. The second server obtains hot data based on the N web page addresses and notifies the cache database to load the hot data into the cache database. The second server preprocesses the hash list by removing web page addresses whose cumulative dwell time on the web page is outside the preset time range and web page addresses set by the administrator from the hash list. The step of analyzing the preprocessed hash list to obtain the N most accessed web page addresses is as follows: The second server saves the preprocessed hash list of the previous M times, and statistically analyzes the hash list sent this time and the preprocessed hash list of the previous M times to obtain the N most accessed web page addresses. The number of times the webpage is accessed is weighted. The weighting process means that the weight is determined based on the cumulative time spent on the webpage and the number of times the webpage is accessed. When sending the hash list to the second server, the number of accesses is multiplied by the weight.

2. The method according to claim 1, characterized in that, The second server obtains hotspot data based on the N webpage addresses. Specifically, it analyzes the webpages corresponding to the N webpage addresses to obtain the database statements called by the corresponding webpages in the first server, and determines the hotspot data based on the database statements called by the first server.

3. The method according to claim 1, characterized in that, The cache database is memcached or Redis, and the hash list is a HashMap.

4. A system for identifying hot data in a cached database, characterized in that, Includes the following modules: The Hash Linked List Management Module is used to trigger the creation of a Hash Linked List for the client when the first server creates a Session for the client. The Hash Linked List stores the Hash value of the webpage address, the number of times the webpage is accessed, and the cumulative time spent on the webpage. The Hash Linked List is destroyed when the Session is destroyed. Specifically, the Hash Linked List is treated as an attribute of the Session, and operations are performed on the Hash Linked List using the getHashLinked() and setHashLinked() methods. The sending module is used by the first server to periodically traverse all Sessions on the first server and send the Hash list corresponding to the valid Sessions to the second server. After sending the Hash list, the Hash list is cleared. The hotspot data discovery module is used by the second server to preprocess the hash chain, analyze the preprocessed hash chain to obtain the N web page addresses with the most visits, obtain hotspot data based on the N web page addresses, and notify the cache database to load the hotspot data into the cache database. The second server preprocesses the hash list by removing web page addresses whose cumulative dwell time on the web page is outside the preset time range and web page addresses set by the administrator from the hash list. The step of analyzing the preprocessed hash list to obtain the N most accessed web page addresses is as follows: The second server saves the preprocessed hash list of the previous M times, and statistically analyzes the hash list sent this time and the preprocessed hash list of the previous M times to obtain the N most accessed web page addresses. The number of times the webpage is accessed is weighted. The weighting process means that the weight is determined based on the cumulative time spent on the webpage and the number of times the webpage is accessed. When sending the hash list to the second server, the number of accesses is multiplied by the weight.

5. The system according to claim 4, characterized in that, The second server obtains hotspot data based on the N webpage addresses. Specifically, it analyzes the webpages corresponding to the N webpage addresses to obtain the database statements called by the corresponding webpages in the first server, and determines the hotspot data based on the database statements called by the first server.

6. A computer-readable storage medium for storing computer program instructions, characterized in that, The computer program instructions, when executed by a processor, implement the method as described in any one of claims 1-3.

Images