Distributed message caching method and device based on chat system

By calculating the hash value of the message identifier, it is determined that it exists in the cache or database, and two-way storage of message records in the Internet chat system is realized, solving the problem of slow user query speed under database storage, improving system performance and reducing modification costs.

CN117493515BActive Publication Date: 2024-11-05BEIJING YUNSHANG TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311496498.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-10
Publication Date
2024-11-05
Estimated Expiration
2043-11-10

AI Technical Summary

Technical Problem

In the Internet chat system, as the number of users increases, when only using the database to store message records, the speed of users querying historical message records is difficult to meet the needs, and the migration cost from the database to the cached double-write scheme is high and complex, increasing the probability of defects .

Method used

By calculating the MD5 and SHA256 values ​​of the message identifier, it is divided into H1, H2, H3, and H4, and based on these values, it is determined whether it exists in the cache. If it exists, it is directly returned to the cache data. If it does not exist, query the database and write to the cache , realize two-way storage and update of data, and reduce database pressure.

Benefits of technology

Improves system throughput, reduces transformation workload, is easy to transform from pure database solutions, reduces the cost of cache servers, and ensures system stability in the event of failure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117493515B_ABST
    Figure CN117493515B_ABST
Patent Text Reader

Abstract

The present application discloses a distributed message caching method and device based on a chat system. The method specifically includes: when the chat system queries the homepage message, calculating the MD5 value and SHA256 value corresponding to the identifier of the homepage message; selecting the first 16 bits and the last 16 bits of the MD5 value, and the first 16 bits and the last 16 bits of the SHA256 value; and judging whether they are all greater than 0; when they are all greater than 0, continuing to query whether the identifier in the cache exists; if it exists, directly returning the cached data. The present application greatly improves the throughput of the chat system, and the transformation workload is small.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of distributed systems, and in particular to a distributed message caching method and device based on a chat system. Background Art

[0002] In the Internet chat system, for the sake of legal compliance and user history, the server needs to save the user's message records for subsequent review or user query of historical message records. Messages are usually saved in a database, which can include two ways of saving message records: directly using the database to save message records or using cache and database double writing to save message records.

[0003] However, for the above two storage methods, when the client does not save local message records and needs to obtain chat records from the server every time, it can meet the needs when the number of users is small, but when the number of users increases, only using the database for storage will slow down the speed of users querying historical message records, which is often difficult to meet user needs.

[0004] Using cache and database dual writing to save message records is powerful enough to handle a large number of users, but it is necessary to consider the cache design at the beginning of the project design. Often, the number of users in the early stage of the project is small, and the use of cache often leads to over-design and increases the cost of the project. In addition, when migrating from a solution that only uses the database to a solution that dual writes to the database and cache, a lot of the original logic will be modified, which not only requires a lot of work to be done, but also increases the probability of defects. Summary of the invention

[0005] The present application provides a distributed message caching method and device based on a chat system, which can reduce database pressure, improve user experience, and is easy to transform and implement and easy to recover from failures.

[0006] In a first aspect, a distributed message caching method based on a chat system, the method comprising:

[0007] When the chat system queries the home page message, the MD5 value and SHA256 value corresponding to the identifier of the home page message are calculated;

[0008] The first 16 bits of the MD5 value are selected as H1, and the last 16 bits of the MD5 value are selected as H2; the first 16 bits of the SHA256 value are selected as H3, and the last 16 bits of the SHA256 value are selected as H4;

[0009] Determine whether H1, H2, H3, and H4 are all greater than 0;

[0010] If both are greater than 0, continue to query whether the identifier in the cache exists;

[0011] If it exists, the cached data is returned directly.

[0012] Optionally, after determining whether H1, H2, H3, and H4 are all greater than 0, the method further includes:

[0013] When H1, H2, H3, and H4 are less than or equal to 0, query whether the identifier exists in the database;

[0014] The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned.

[0015] Optionally, after continuing to query whether the identifier in the cache exists, the method further includes:

[0016] When the identifier in the cache does not exist, the database is queried to see if the identifier exists;

[0017] The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned.

[0018] Optionally, the method further comprises:

[0019] Update the new message into the database;

[0020] Calculate the MD5 value and SHA256 value of the current identifier;

[0021] The first 16 bits of the MD5 value are selected as H1, and the last 16 bits of the MD5 value are selected as H2; the first 16 bits of the SHA256 value are selected as H3, and the last 16 bits of the SHA256 value are selected as H4;

[0022] Subtract one from the data at positions H1, H2, H3, and H4, and return the cached data;

[0023] If the cache has a current identifier, delete the contents of the cache.

[0024] Optionally, the method also includes: storing the data of the chat system using a kv structure, using the project ID, session ID, and message type as strings to concatenate into an identifier for cached data, and using the homepage content of the session as the value; the identifier is calculated using a hash function and then placed in a hash bucket for storage.

[0025] In a second aspect, a distributed message caching device based on a chat system, the device comprising:

[0026] A calculation module, used for calculating the MD5 value and SHA256 value corresponding to the identifier of the home page message when the chat system queries the home page message;

[0027] A selection module is used to select the first 16 bits of data in the MD5 value as H1 and the last 16 bits of data as H2, and select the first 16 bits of data in the SHA256 value as H3 and the last 16 bits of data as H4;

[0028] The processing module is used to determine whether H1, H2, H3, and H4 are all greater than 0; if they are all greater than 0, continue to query whether the identifier in the cache exists; if it exists, directly return the cache data.

[0029] Optionally, after the processing module determines whether H1, H2, H3, and H4 are all greater than 0, the processing module further includes:

[0030] When H1, H2, H3, and H4 are less than or equal to 0, query whether the identifier exists in the database;

[0031] The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned.

[0032] Optionally, after the processing module continues to query whether the identifier in the cache exists, the process further includes:

[0033] When the identifier in the cache does not exist, the database is queried to see if the identifier exists;

[0034] The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned.

[0035] In a third aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the distributed message caching method described in the first aspect is implemented.

[0036] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the distributed message caching method described in the first aspect is implemented.

[0037] In the technical solution provided by the embodiment of the present application, when the chat system queries the homepage message, the MD5 value and SHA256 value corresponding to the identifier of the homepage message query are calculated; the first 16 bits and the last 16 bits of the MD5 value, and the first 16 bits and the last 16 bits of the SHA256 value are selected; and it is determined whether they are all greater than 0; if they are all greater than 0, it continues to query whether the identifier in the cache exists; if it exists, the cached data is directly returned. Compared with the prior art, the present application greatly improves the throughput of the system, and the transformation workload is small. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1This is a flowchart of the steps of the distributed message caching method based on the chat system of this application;

[0039] Figure 2 This is a flowchart of the steps of an optional embodiment of the present application;

[0040] Figure 3 A block diagram of a distributed message caching device provided in an embodiment of the present application;

[0041] Figure 4 Schematic diagram of the internal structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] In the description of the present application, expressions such as “including”, “comprising”, “having”, etc. also mean “not limited to” (certain units, components, materials, steps, etc.).

[0044] In the present invention, the cache exists as a separate service, and the message service is another service. The message service stores messages by calling the cache service and the database. The cache service uses hash sharding, and each shard uses multiple services for master-slave backup, so that the cache can be deployed in a cluster form on separate machines.

[0045] The cache system in the present invention caches the first page of messages of each session. The so-called first page of messages refers to the most recent N messages when the user obtains historical messages. N depends on the value passed in when the user queries the historical records. When the data is less than N, all current messages are cached.

[0046] The data in the present invention is stored using a kv structure, using the project ID, session ID, and message type as strings to concatenate into the key of the cached data, and the homepage content of the session as the value. The key can be calculated using a hash function and then stored in a hash bucket.

[0047] like Figure 1 As shown, a distributed message caching method based on a chat system is provided, that is, the message history record query logic of the system, including the following steps:

[0048] S11, when the chat system queries the home page message, the MD5 value and SHA256 value corresponding to the identifier (key) of the query home page message are calculated.

[0049] MD5 is an encryption algorithm used to generate message digests. SHA256 is another encryption algorithm that is also used to generate message digests. By calculating the MD5 and SHA256 values ​​of the corresponding key, two message digests of different lengths can be obtained.

[0050] S12, select the first 16 bits of the MD5 value as H1 and the last 16 bits as H2, select the first 16 bits of the SHA256 value as H3 and the last 16 bits as H4.

[0051] This step is to further divide the MD5 and SHA256 message digests into four 16-bit segments. These segments can be used individually for further processing.

[0052] The message cache is saved using serialized encoding. When the database query is empty, the content of no message can also be saved in the cache to improve the cache hit rate.

[0053] A simple multi-hash storage can be used on the message server to identify the existence of the corresponding message in the cache, that is, the existence status or status of multiple hashes is used to approximate whether a key exists in the cache. The first 16 bits of MD5 are recorded as H1, the last 16 bits of MD5 are recorded as H2, the first 16 bits of SHA256 are recorded as H3, and the last 16 bits of SHA256 are recorded as H4. The combination represents the status of a key in the cache. If the status of these four hashes are all present, then the key may exist in the cache. If any of the four hashes does not exist, then the key must not exist in the cache and can be directly accessed from the database, thereby reducing invalid access to the cache and improving the system response speed. The service locally uses an array to store the number of times these hashes are used. H1 is stored in A1, H2 is stored in A2, H3 is stored in A3, and H4 is stored in A4.

[0054] S13, determine whether H1, H2, H3, and H4 are all greater than 0;

[0055] S14, when all are greater than 0, continue to query whether the identifier in the cache exists;

[0056] S15, if it exists, directly return the cached data. That is, if the data corresponding to the key is found in the cache, it will be returned.

[0057] The above steps further include:

[0058] S16 If the cache does not exist, query the database.

[0059] S17 writes the data into the cache, and increases the number at the corresponding H1 position in A1 by one, increases the number at the corresponding H2 position in A2 by one, increases the number at the corresponding H3 position in A3 by one, and increases the number at the corresponding H4 position in A4 by one. The data is then returned.

[0060] In addition, if Figure 2 In another optional embodiment of the present application, the historical message storage logic of the system is as follows:

[0061] S21, saving the new message to the database.

[0062] S22, calculate the MD5 and SHA256 values ​​of the key.

[0063] S23, take the first 16 bits of MD5 as H1 and the last 16 bits as H2, take the first 16 bits of SHA256 as H3 and the last 16 bits as H4.

[0064] S24, decrement the corresponding H1, H2, H3, H4 positions in the local A1, A2, A3, A4 by one respectively.

[0065] S25: If the cache has the current key, delete the cached content.

[0066] Through the above methods, the cached data and the database are kept consistent eventually. If the cache service works abnormally, the database can still work normally.

[0067] The logic of fault handling is as follows:

[0068] When a cache service node fails, other cache service nodes will take over and continue to work. When the cache service fails as a whole, you can directly read the message data from the database. This is equivalent to not finding the data set in the cache, but it will fail when writing to the cache. When the cache service fails, you need to clear all the data in the cache, restore the cache's working logic, and restart the accumulation of cache data.

[0069] In summary, it can be seen that the present invention includes at least the following advantages:

[0070] 1. Easy to transform from a pure database solution, reducing transformation costs.

[0071] 2. It is convenient for different projects in the same service to switch between cache and database according to the situation.

[0072] 3. Empty data is also written into the cache to improve the hit rate and reduce database pressure.

[0073] 4. Partial caching is possible instead of full caching, which reduces the cost of the cache server.

[0074] In one embodiment, Figure 3 , provides a distributed message caching device, specifically including:

[0075] A calculation module, used for calculating the MD5 value and SHA256 value corresponding to the identifier of the home page message when the chat system queries the home page message;

[0076] A selection module is used to select the first 16 bits of data in the MD5 value as H1 and the last 16 bits of data as H2, and to select the first 16 bits of data in the SHA256 value as H3 and the last 16 bits of data as H4;

[0077] The processing module is used to determine whether H1, H2, H3, and H4 are all greater than 0; when they are all greater than 0, continue to query whether the identifier in the cache exists; if it exists, directly return the cache data.

[0078] In an optional embodiment of the present application, after the processing module determines whether H1, H2, H3, and H4 are all greater than 0, it further includes:

[0079] When H1, H2, H3, and H4 are less than or equal to 0, query whether the identifier exists in the database;

[0080] The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned.

[0081] In an optional embodiment of the present application, after the processing module continues to query whether the identifier in the cache exists, it also includes:

[0082] When the identifier in the cache does not exist, the database is queried to see if the identifier exists;

[0083] The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned.

[0084] For the specific definition of the distributed message caching device, please refer to the definition of the distributed message caching method above, which will not be repeated here.

[0085] In one embodiment, an electronic device is provided. The electronic device may be a computer, and its internal structure diagram may be as follows: Figure 4As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for will tamper-proof data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a will tamper-proof method is implemented.

[0086] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0087] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned will anti-tampering method are implemented.

[0088] The computer-readable storage medium provided in this embodiment has similar implementation principles and technical effects to those of the above method embodiments, and will not be described in detail here.

[0089] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A distributed message caching method based on a chat system, It is characterized in that The method comprises: When the chat system queries the home page message, the MD5 value and SHA256 value corresponding to the identifier of the home page message are calculated; The first 16 bits of the MD5 value are selected as H1, and the last 16 bits of the MD5 value are selected as H2; the first 16 bits of the SHA256 value are selected as H3, and the last 16 bits of the SHA256 value are selected as H4; Determine whether H1, H2, H3, and H4 are all greater than 0; If both are greater than 0, continue to query whether the identifier in the cache exists; If it exists, the cached data is returned directly; After determining whether H1, H2, H3, and H4 are all greater than 0, the method further includes: When H1, H2, H3, and H4 are less than or equal to 0, query whether the identifier exists in the database; The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned; After continuing to query whether the identifier in the cache exists, the method further includes: When the identifier in the cache does not exist, the database is queried to see if the identifier exists; The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned; The method further comprises: Update the new message into the database; Calculate the MD5 value and SHA256 value of the current identifier; The first 16 bits of the MD5 value are selected as H1, and the last 16 bits of the MD5 value are selected as H2; the first 16 bits of the SHA256 value are selected as H3, and the last 16 bits of the SHA256 value are selected as H4; Subtract one from the data at positions H1, H2, H3, and H4, and return the cached data; If the cache has the current identifier, delete the cache's contents; The method further includes: storing the chat system data using a kv structure, using a project ID, a session ID, and a message type as a string to concatenate into an identifier of cached data, and using the homepage content of the session as a value; The identifier is calculated using a hash function and placed into a hash bucket for storage.

2. A distributed message caching device based on a chat system, It is characterized in that The device comprises: A calculation module, used for calculating the MD5 value and SHA256 value corresponding to the identifier of the home page message when the chat system queries the home page message; A selection module is used to select the first 16 bits of data in the MD5 value as H1 and the last 16 bits of data as H2, and select the first 16 bits of data in the SHA256 value as H3 and the last 16 bits of data as H4; A processing module, used to determine whether H1, H2, H3, and H4 are all greater than 0; if they are all greater than 0, continue to query whether the identifier in the cache exists; if it exists, directly return the cache data; After the processing module determines whether H1, H2, H3, and H4 are all greater than 0, it further includes: When H1, H2, H3, and H4 are less than or equal to 0, query whether the identifier exists in the database; The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned; After the processing module continues to query whether the identifier in the cache exists, it also includes: When the identifier in the cache does not exist, the database is queried to see if the identifier exists; The data is written into the cache, and the data at positions H1, H2, H3, and H4 are incremented by one, and the cached data is returned; The device also includes: Update the new message into the database; Calculate the MD5 value and SHA256 value of the current identifier; The first 16 bits of the MD5 value are selected as H1, and the last 16 bits of the MD5 value are selected as H2; the first 16 bits of the SHA256 value are selected as H3, and the last 16 bits of the SHA256 value are selected as H4; Subtract one from the data at positions H1, H2, H3, and H4, and return the cached data; If the cache has the current identifier, delete the cache's contents; The device also includes: the chat system data is stored using a kv structure, the project ID, the session ID, and the message type are used as strings to concatenate into an identifier of the cached data, and the homepage content of the session is used as a value; The identifier is calculated using a hash function and placed into a hash bucket for storage.

3. An electronic device, It is characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the method according to claim 1 are implemented.

4. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to claim 1 are implemented.