Memory database persistence method and device, equipment and storage medium
By deploying multiple key spaces in the in-memory database and performing memory snapshots according to the update frequency timing, the data loss problem caused by insufficient data persistence frequency in the prior art is solved, and efficient data persistence and load reduction are achieved.
Patent Information
- Application Number
- CN202411737083.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-02-25
AI Technical Summary
In in-memory databases, it is difficult for the prior art to persist data at high frequency, resulting in the data updated during the last memory snapshot and the next memory snapshot that cannot be persisted, and there is a problem of data loss.
By pre-deploying multiple key spaces, each key space is used to store data keys whose update frequency is within the preset update frequency range, and to perform memory snapshots regularly based on the preset snapshot frequency of each key space.
The data keys for different update frequencies are persisted at different frequencies, ensuring that the data in each update frequency range can be persisted in time, reducing data loss, and reducing the load pressure of server memory snapshots.
Smart Images
Figure CN119512467A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and more specifically, to a method, device, equipment and storage medium for persisting a memory database. Background Art
[0002] In order to avoid the loss of memory data due to server downtime and affect normal business operations, the data in the memory database needs to be persisted to the disk so that data can be restored when the server fails. Taking Redis persistence as an example, the data persistence mode includes the memory snapshot mode (RDB), where the RDB mode refers to the persistence of the full amount of data in the memory at a specific point in time. However, since the RDB mode is the persistence of the full amount of data, considering the read and write pressure of the disk, it is impossible to achieve high-frequency persistence. This results in the data updated between the last memory snapshot and the next memory snapshot being unable to be persisted if the server crashes during the period between the last memory snapshot and the next memory snapshot, and the problem of data loss still exists.
[0003] Therefore, there is an urgent need for a memory database persistence method to solve the problem of data loss in the data persistence process of the memory database. Summary of the invention
[0004] In view of the above problems, the present application provides a method, device, equipment and storage medium for in-memory database persistence to solve the problem of data loss in the process of data persistence in in-memory database. The specific solution is as follows:
[0005] In a first aspect, the present application provides a method for persisting a memory database, which is applied to a memory database of a server, wherein the memory database is pre-deployed with at least two key spaces, the key spaces being used to store data keys whose update frequencies are within a preset update frequency range, and the preset update frequency ranges corresponding to each of the key spaces are different;
[0006] The memory database persistence method executed for each key space includes:
[0007] Determine the time interval between two adjacent memory snapshots of the key space according to a preset snapshot frequency corresponding to the key space, wherein the preset snapshot frequency is set according to the update frequency range corresponding to the key space;
[0008] When the time from the last memory snapshot reaches the time interval, a memory snapshot is taken of the data key and the key value corresponding to the data key stored in the key space.
[0009] In a possible implementation, the process of pre-deploying the key space includes:
[0010] Get the update frequency corresponding to each data key in the memory database;
[0011] According to the update frequency, dividing the update frequency range into a preset number;
[0012] Deploy a key space for each update frequency range respectively;
[0013] Migrate the data key whose update frequency belongs to the update frequency range to the key space corresponding to the update frequency range;
[0014] An RDB program and a Bloom filter for performing memory snapshots at a preset snapshot frequency are configured in the key space, and the Bloom filter is used to save the corresponding relationship between the data keys stored in the key space and the update frequency range.
[0015] In a possible implementation, the process of determining the preset snapshot frequency corresponding to the key space includes:
[0016] Determine an update frequency corresponding to each of the data keys stored in the key space;
[0017] The average value of the update frequencies corresponding to all the data keys in the key space is used as the preset snapshot frequency of the key space.
[0018] In a possible implementation, the method further includes:
[0019] Monitoring the update frequency of the data key corresponding to each of the key spaces;
[0020] When the update frequency changes and the update frequency is not within the preset update frequency range corresponding to the key space, determining the current update frequency of the data key;
[0021] Determine the target migration keyspace corresponding to the current update frequency according to the preset update frequency range corresponding to each keyspace;
[0022] Migrate the data key and the key value corresponding to the data key to the target migration key space.
[0023] In a possible implementation, the method further includes:
[0024] In response to receiving an edit request for a target data key in the in-memory database, determining a target key space storing the target data key according to the data key stored in each key space or the preset update frequency;
[0025] Based on the edit request, the target data key in the target key space is edited.
[0026] In a possible implementation, determining, based on the data keys stored in each of the key spaces, a target key space storing the target data key includes:
[0027] Querying the correspondence between the data key and the update frequency range stored in each of the Bloom filters, and determining a target Bloom filter that exists in the correspondence corresponding to the target data key;
[0028] The key space corresponding to the target Bloom filter is determined as the target key space.
[0029] In a possible implementation, determining the target key space storing the target data key according to the preset update frequency stored in each key space includes:
[0030] determining an update frequency of the target data key;
[0031] According to the update frequency range stored in each of the Bloom filters, determining the Bloom filter corresponding to the update frequency range including the update frequency as a target Bloom filter;
[0032] The key space corresponding to the target Bloom filter is determined as the target key space.
[0033] A second aspect of the present application provides a memory database persistence device, which is applied to a memory database of a server, wherein the memory database is pre-deployed with at least two key spaces, the key spaces are used to store data keys whose update frequency is within a preset update frequency range, and the preset update frequency range corresponding to each key space is different;
[0034] The memory database persistence device comprises:
[0035] An interval determination unit, configured to determine a time interval between two adjacent memory snapshots of the key space according to a preset snapshot frequency corresponding to the key space, wherein the preset snapshot frequency is set according to the update frequency range corresponding to the key space;
[0036] The memory snapshot unit is used to take a memory snapshot of the data key and the key value corresponding to the data key stored in the key space when the time length from the last memory snapshot reaches the time interval.
[0037] A third aspect of the present application provides a memory database persistence device, comprising at least one processor and a memory connected to the processor, wherein:
[0038] The memory is used to store computer programs;
[0039] The processor is used to execute the computer program so that the memory database persistence device can implement any one of the memory database persistence methods.
[0040] In a fourth aspect of the present application, a computer storage medium is provided, which carries one or more computer programs. When the one or more computer programs are executed by a memory database persistence device, the memory database persistence device can implement any one of the memory database persistence methods described.
[0041] It can be seen from the above technical solutions that the in-memory database persistence method provided by the embodiment of the present application, taking into account the different update frequencies of different data keys in the in-memory database, pre-deploys multiple key spaces, and sets the update frequency range of the data keys used to store for each key space, so that data keys with different update frequencies can be stored in different key spaces according to their ranges. Each key space regularly persists the data keys and key values stored in the key space according to the time interval corresponding to the preset snapshot frequency.
[0042] Since the memory snapshot operations between the key spaces of this application are independent of each other and the memory snapshot frequencies are different, compared with the memory snapshot of the full amount of data at a specific time point in the prior art, this application performs memory snapshots at different frequencies for data keys with different update frequencies to ensure that the data in each update frequency range can be persisted in time and minimize the loss of data. And since the memory snapshot operations between key spaces are independent of each other, compared with the memory snapshot of the full amount of data in the prior art, the amount of data in each memory snapshot of this application is reduced, thereby achieving the effect of reducing the load pressure of the server each time the memory snapshot is performed. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0044] Figure 1 A flowchart of a method for implementing in-memory database persistence provided in an embodiment of the present application;
[0045] Figure 2 A schematic diagram of a process for deploying a key space provided in an embodiment of the present application;
[0046] Figure 3 A schematic diagram of the structure of a memory database persistence device provided in an embodiment of the present application;
[0047] Figure 4 A schematic diagram of the structure of an in-memory database persistence device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0049] In the prior art, the Redis persistence mode may also include: a mode that records each modification command in sequence (AOF) and a hybrid mode. The AOF mode records each write operation performed by the Redis server and re-executes these operations when the server starts to reconstruct the original data. The hybrid mode is a persistence method that combines the RDB mode and the AOF mode.
[0050] AOF can save incremental commands at a higher frequency than RDB mode, and record the incremental commands between the last memory snapshot and the next memory snapshot, which is used to restore data updated after the last memory snapshot. However, the speed of data recovery is slower than that of RDB mode, and the higher the frequency of using AOF mode to persist memory, the greater the load burden on the server. In multi-concurrency scenarios, it may even cause the server to crash, and there is also the possibility of data loss. In hybrid mode, AOF can record the incremental commands between two RDB memory snapshots, and does not need to record all operations. Compared with the single AOF mode file, the file size is reduced. However, for data with a faster update frequency, a large number of incremental commands still need to be repeatedly executed during recovery to obtain updated data. The recovery speed is slow and it also puts a burden on the server. There is still a risk of crashes, resulting in data loss.
[0051] In order to solve the above-mentioned problem, an embodiment of the present application proposes a memory database persistence method, which is applied to the memory database of the server. The memory database is pre-deployed with at least two key spaces, wherein the key space is used to store data keys whose update frequency is within a preset update frequency range, and each key space corresponds to a different preset update frequency range.
[0052] It is understandable that the update frequency of each data key in the memory database is different. Regardless of RDB mode or hybrid mode, all data in the memory database is persisted. In order to reduce the burden on the server, data keys with high update frequency cannot be persisted at a high frequency. And considering that the update frequency of different data keys in the memory database is different, if all data keys are snapshotted at a specific time point, in the case of server downtime, data keys with high update frequency will inevitably fail to be saved.
[0053] Therefore, the embodiment of the present application deploys multiple key spaces in the memory database, wherein the key space of Redis is a data structure similar to a dictionary, which can store data keys and some metadata related thereto, and each key space can independently manage its own data. Based on this, the data keys in the memory database are divided into different key spaces, and persistence can be performed according to different key spaces. The persistence operation of each key space is relatively independent. Compared with the persistence of the entire data, persistence according to the key space can effectively reduce the amount of data persisted each time, thereby reducing the burden of each memory snapshot of the server.
[0054] At the same time, data keys whose update frequency is within the same preset update frequency range are assigned to the same key space. For example, data keys whose update frequency is within the frequency range of once every 10-30 seconds are divided into the same key space. In this key space, the data keys in the key space can be persisted at a faster memory snapshot frequency, such as setting the memory snapshot frequency of the key space to once every 30 seconds, so that the updated key value of each data key in the key space can be saved in time.
[0055] Reference Figure 2 , a flow chart of a method for implementing a memory database persistence provided by an embodiment of the present application, the specific process of the memory database persistence method for each key space may include:
[0056] Step S110: determining the time interval between two adjacent memory snapshots of the key space according to a preset snapshot frequency corresponding to the key space.
[0057] Step S120: When the time interval from the last memory snapshot reaches the time interval, a memory snapshot is taken for the data keys stored in the key space and the key values corresponding to the data keys.
[0058] After each memory snapshot of the key space is completed, the time point of each memory snapshot is recorded. Based on the snapshot frequency of the key space, the time interval between two adjacent memory snapshots is determined. When the length of time between the current time and the time point when the last memory snapshot was completed reaches the time interval, all data keys in the key space and the key values corresponding to each data key and other related data are snapshotted again to achieve persistence of all data in the key space.
[0059] It can be understood that the in-memory database contains multiple key spaces, each of which is persisted according to its corresponding snapshot frequency. The snapshot frequency of each key space can be determined based on the update frequency of the data keys stored in the key space, or it can be set based on the update frequency range corresponding to the key space to ensure that the persistence frequency is as close as possible to the update frequency of the data keys, thereby improving the timeliness of persistence and avoiding data loss.
[0060] In summary, the in-memory database persistence method provided by the embodiment of the present application takes into account the different update frequencies of different data keys in the in-memory database, deploys multiple key spaces in advance, and sets the update frequency range of the data keys used to store for each key space, so that data keys with different update frequencies can be stored in different key spaces according to their ranges. Each key space regularly persists the data keys and key values stored in the key space according to the time interval corresponding to the preset snapshot frequency.
[0061] Since the memory snapshot operations between the key spaces of this application are independent of each other and the memory snapshot frequencies are different, compared with the memory snapshot of the full amount of data at a specific time point in the prior art, this application performs memory snapshots at different frequencies for data keys with different update frequencies to ensure that the data in each update frequency range can be persisted in time and minimize the loss of data. And since the memory snapshot operations between key spaces are independent of each other, compared with the memory snapshot of the full amount of data in the prior art, the amount of data in each memory snapshot of this application is reduced, thereby achieving the effect of reducing the load pressure of the server each time the memory snapshot is performed.
[0062] Next, the embodiments of the present application illustrate other possible implementations of the in-memory database persistence method described above.
[0063] Reference Figure 2 , a schematic diagram of a process of deploying a key space provided in an embodiment of the present application, which illustrates the process of pre-deploying a key space, may specifically include the following steps:
[0064] Step S210, obtaining the update frequency corresponding to each data key in the memory database.
[0065] The update frequency of data keys can be obtained by parsing the update records of the memory database or files such as database logs, or by monitoring the memory database for a period of time and the update frequency of each data key during this period of time. For example, in units of days, the number of updates per day for each data key is used as the update frequency, and the memory database is continuously monitored for 10 days. If data key A is updated 7 times within 10 days, the update frequency of data key A is 7 times / day.
[0066] Optionally, other statistical methods such as machine learning, such as time series prediction models, can also be used to analyze the data source of the data keys stored in the memory database and estimate the update frequency of each data key based on the operating properties of the data source. It is understandable that there are many ways to obtain the update frequency of each data key, and the above is only for reference, and the embodiments of this application are not limited to it.
[0067] Step S220, dividing the update frequency range into a preset number according to the update frequency.
[0068] Step S230: deploy a key space for each update frequency range.
[0069] Step S240, migrate the data keys whose update frequency belongs to the update frequency range to the key space corresponding to the update frequency range.
[0070] According to the update frequency corresponding to all data keys in the memory database, a full frequency segment is determined, and the full frequency segment is divided into multiple update frequency ranges from low frequency to high frequency. For example, the unit of update frequency is times / day (how many times are updated per day), referring to the update frequency of all data keys in the memory database in Table 1, where the lowest update frequency is 0.2 and the highest update frequency is 1.5, then the full frequency segment is 0.2~1.5, or the full frequency segment is regarded as 0~1.5.
[0071] Table 1 Data key update frequency
[0072]
[0073] When the number of divisions is predetermined, the entire frequency segment is divided into a preset number of update frequency ranges. Optionally, considering the amount of data stored in each key space, the data amount can be simultaneously referenced during the division. For example, when the preset number is 3, one division method is to evenly divide the entire frequency segment into 3 segments to obtain 3 update frequency ranges, that is, the 3 update frequency ranges include: 0~0.5, 0.5~1, 1~1.5. Another division method is that the amount of data in the 3 update frequency ranges is the same, then the 3 update frequency ranges obtained by the division of Table 1 include: 0.2~0.6, 0.7~1, 1~1.5.
[0074] It is understandable that, when the amount of data in each update frequency range is basically the same, the key space with a high snapshot frequency will trigger persistent snapshots at a higher frequency than the key space with a low snapshot frequency, so that the data keys with a high update frequency can be persisted in time. When the amount of data in each update frequency range is different, even if all key spaces are persisted at the same time, since each key space has an independent memory snapshot, the amount of data in the memory snapshot is less than the full amount of data, which can reduce the burden on the server. Therefore, no matter what the data distribution in the frequency range is, the burden of the system snapshot can be effectively reduced from the perspective of data volume or from the perspective of update frequency.
[0075] Furthermore, key spaces having the same number as the divided update frequency ranges are configured, and data related to data keys belonging to the same update frequency range are migrated to corresponding key spaces to achieve isolation between data keys.
[0076] Step S250: configuring an RDB program and a Bloom filter in the key space for taking memory snapshots at a preset snapshot frequency.
[0077] According to the snapshot strategy currently supported by the in-memory database, set the corresponding RDB configuration in each key space. Optionally, in the redis.conf file, the default RDB persistence configuration is as follows: save 900 1. "Save" is called a checkpoint, and "900 1" means that when at least one key is modified within 900 seconds, Redis performs a persistence operation. It can also be understood that every 900 seconds, if at least one key is changed, a new dump.rdb file is generated. The dump.rdb file is a complete data snapshot in the redis memory, also called snapshotting.
[0078] According to the above RDB persistence configuration, set the snapshot frequency corresponding to each key space. Optionally, the snapshot frequency of each key space can be determined by the update frequency range corresponding to the key space, for example, taking the middle value or maximum value of the update frequency range, etc., to ensure that the update frequency of the key space is close to the update frequency of each data key in the key space.
[0079] Optionally, you can also configure an AOF program for each key space at the same time, that is, each key space is configured with a hybrid persistence mode of RDB and AOF. The following example illustrates the advantages of the hybrid persistence mode of RDB and AOF configured in the key space: In the prior art, the in-memory database persists the full amount of data in the in-memory database at a memory snapshot frequency of once every 2 hours, and AOF records the update instructions of all data keys within 2 hours between each two adjacent memory snapshots. For example, the two memory snapshot times of the in-memory database are: 8:00 and 10:00, and AOF records the update instructions of all data keys between 8:00 and 10:00. If the server goes down at 9:47, the data between 8:00 and 9:47 is not successfully persisted. If the data in this time period is restored, the server needs to execute all update instructions between 8:00 and 9:47 based on the data at 8:00 to obtain the original data. For data keys with a high update frequency, such as key1 updated twice a minute, when restoring data, the server only needs to execute hundreds of update instructions for key1, which has a heavy operating burden and takes a long time to restore data.
[0080] In the embodiment of the present application, each data key is divided into key spaces with different snapshot frequencies according to the update frequency, and key1 is divided into a key space with a snapshot frequency of once per minute. Then, when the server crashes at 9:47, it is only necessary to obtain the data of 9:46 and execute two update instructions based on the data of 9:46. Compared with the prior art, the embodiment of the present application can restore data faster and reduce the operating burden of the server.
[0081] In one possible implementation, the process of determining a preset snapshot frequency corresponding to a key space may include: determining an update frequency corresponding to each data key stored in the key space; and taking the average of the update frequencies corresponding to all data keys in the key space as the preset snapshot frequency of the key space.
[0082] If persistence is performed at the minimum frequency value of the update frequency range, the update frequency may be too fast, making multiple persistences invalid for a large number of data keys in the corresponding key space, which in turn increases power consumption. Therefore, in an embodiment of the present application, the average update frequency of all data keys stored in the key space is used as the snapshot frequency of the key space to ensure that the persistence frequency of the key space can keep up with the update frequency of most data keys in the key space.
[0083] While setting the RDB configuration in each key space, a Bloom filter for key space routing is established in the memory database. The Bloom algorithm in the Bloom filter is similar to a hash set, which is used to determine whether an element (key) is in a set. The difference is that the Bloom algorithm does not need to store the value of the key. For each key, only k bits are needed to store a flag for each key to determine whether the key is in the set. Based on this, the Bloom filter is used to save the correspondence between the data keys stored in the key space and the update frequency range, and can also be used to store the flag of each data key stored in the key space.
[0084] Taking into account the large amount of data to be persisted in the memory database, if each time the data key is updated, each key space is searched one by one to determine the key space to which the updated data key belongs, it will take a lot of time. Therefore, the embodiment of the present application introduces a Bloom filter, and each key space is configured with a Bloom filter. The Bloom filter is used to determine whether the updated data key belongs to the key space. It can be understood that a data key index list is established in the Bloom filter, and the data key index list can be queried without matching each data key one by one in the key space.
[0085] A Bloom filter can be established in the memory database according to the divided update frequency range. For example, if the full frequency segment is divided into 5 update frequency ranges, then 5 Bloom filters need to be established, and each Bloom filter corresponds to an update frequency range. The update frequency ranges are sequentially established from high to low. For each data key that needs to be stored in the Bloom filter, it is necessary to determine whether the data key or the key value of the data key already exists in the Bloom filter that has been constructed, so as to shield the subsequent Bloom filter from the situation where a data key belongs to multiple Bloom filters when judging which key space the data key belongs to. This situation can be called a false positive, which means that when the Bloom filter judges that an element exists, the element is not actually in the set, but the Bloom filter returns the result that the element exists. The embodiment of the present application shields the influence of false positives when establishing the Bloom filter by pre-judging whether there are false positives, thereby improving the accuracy of subsequent judgments by the Bloom filter.
[0086] Furthermore, the data keys and update frequency ranges corresponding to the key space are inserted into the Bloom filter corresponding to the update frequency range to obtain the Bloom filter corresponding to each key space. For the data keys with position update frequency and the data keys that may be added later, a new key space with unknown update frequency and the corresponding Bloom filter with unknown update frequency can be established. The key space can be persisted according to the update frequency of the data keys therein, that is, as long as there is data update, it will be persisted. A snapshot frequency can also be pre-set, and the data keys with unknown update frequency can be persisted according to the snapshot frequency.
[0087] In summary, the full frequency segment where the update frequency of the data key in the memory database is located is divided into multiple update frequency ranges, and each data key belongs to one of the update frequency ranges according to its update frequency. A memory database key space is established for each update frequency range, and each key space is set to RDB mode, or RDB and AOF mixed persistence mode can be set. According to the set persistence mode, each key space is persisted independently of each other. Compared with the existing technology, from the perspective of reducing the amount of persistent data, or from the perspective of reducing the number of persistence times, it can reduce the burden of the sub-process persistence process on the system. In addition, different snapshot frequencies are arranged for data keys with different update frequencies, so that data keys can be persisted in time, reducing the possibility of data loss.
[0088] The division of key spaces does not affect any processing of the data stored in the in-memory database. In response to receiving an edit request for a target data key in the in-memory database, the target key space storing the target data key is determined based on the data keys stored in each key space or a preset update frequency; based on the edit request, the target data key in the target key space is edited.
[0089] When receiving an edit request to the in-memory database, the edit request may include: insert, delete, update, and other operation requests for the target data key. According to the target data key or the key value of the target data key in the edit request, the Bloom filter corresponding to each key space is searched in turn. If the target data key exists in one of the Bloom filters, based on the operation request in the edit request, the target data key in the key space corresponding to the Bloom filter is deleted or updated, or the target data key is inserted into the key space. If the target data key does not exist in any key space, the target data key is stored in the unknown update frequency key space, and the Bloom filter of the unknown update frequency key space is updated synchronously.
[0090] When a read request to the in-memory database is received, similar to the above-mentioned edit request, the Bloom filter corresponding to each key space is searched in turn according to the target data key or the key value of the target data key in the read request. If the target data key does not exist in all Bloom filters, including the Bloom filter with unknown update frequency, the result that the data does not exist is directly returned. If the target data key exists in one of the Bloom filters, the key value and other data corresponding to the target data key are read from the key space corresponding to the Bloom filter and returned as the request result of the read request.
[0091] Optionally, the Bloom filter stores the correspondence between the data key and the update frequency range. Therefore, when searching for the Bloom filter corresponding to each key space, the search can be performed based on the data key or the update frequency of the data key. In one possible implementation, the correspondence between the data key and the update frequency range stored in each Bloom filter is queried to determine the target Bloom filter that exists in the correspondence corresponding to the target data key; the key space corresponding to the target Bloom filter is determined as the target key space. Or in another possible implementation, the update frequency of the target data key is determined; based on the update frequency range stored in each Bloom filter, the Bloom filter corresponding to the update frequency range containing the update frequency is determined as the target Bloom filter; the key space corresponding to the target Bloom filter is determined as the target key space.
[0092] In a possible implementation, after the key space is divided and deployed, it can also include: monitoring the update frequency of the data key corresponding to each key space; when the update frequency changes and the update frequency is not within the preset update frequency range corresponding to the key space, determining the current update frequency of the data key; based on the preset update frequency range corresponding to each key space, determining the target migration key space corresponding to the current update frequency; migrating the data key and the key value corresponding to the data key to the target migration key space.
[0093] It is understandable that due to factors such as system updates, upgrades, and business changes, the data keys in the in-memory database are not static, and their update frequency will also change with changes in the system. Therefore, the key space to which the data keys belong also needs to be changed.
[0094] Optionally, the update frequency of each data key is monitored at a preset frequency or in real time, and in this process, the update frequency of data keys with unknown update frequencies can also be updated simultaneously when the key space is deployed. If the full frequency range of the update frequency corresponding to all data keys is expanded relative to the full frequency range determined in steps S220-S240, a new key space, Bloom filter, etc. can be expanded.
[0095] For data keys whose update frequency has changed and whose updated frequency after the change does not belong to the update frequency range of the currently corresponding key space, it is necessary to modify the affiliation between the data key and the key space. Therefore, the corresponding relationship stored in the Bloom filter also needs to be updated synchronously. In the embodiment of the present application, the Bloom filter to which the data key originally belonged is called the old version Bloom filter, and the Bloom filter to which the data key belongs after the update frequency changes is called the new version Bloom filter.
[0096] During the parallel period of the old and new Bloom filter versions, data keys with changed update frequencies will exist in both key spaces at the same time. It is necessary to determine whether the data key exists in both the old and new versions of the Bloom filter to pre-screen the problems that may be caused by false positives of the Bloom filter. If it exists in both key spaces, no processing is required; otherwise, the key value and other related data corresponding to the data key are migrated and inserted from the key space corresponding to the old version of the Bloom filter to the key space corresponding to the new version of the Bloom filter.
[0097] For transactions during the coexistence period of the old and new versions, query transactions are still queried in the key space corresponding to the old version of the Bloom filter, and write transactions are written in the key space corresponding to the old and new versions of the Bloom filter at the same time.
[0098] If all data keys with changed update frequencies have completed the insertion of data in the new key space, the new version of the Bloom filter will take effect, and the corresponding relationship stored in the old version of the Bloom filter for the data keys with changed update frequencies will become invalid. For data keys with changed update frequencies, check whether the data key exists in both the old and new versions of the Bloom filter. If it exists in both versions of the Bloom filter, delete the data key from the key space of the old version. Conversely, if the key space only exists in the new version of the Bloom filter, no processing is required.
[0099] Based on this, the embodiment of the present application can track the changes of data keys in real time, and adjust the persistence strategy for each data key according to the changes, thereby improving the accuracy and flexibility of the persistence of the in-memory database.
[0100] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of personal information in the memory database involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0101] The following is a description of an in-memory database persistence device provided in an embodiment of the present application. The in-memory database persistence device described below and the in-memory database persistence method described above can refer to each other.
[0102] First, combine Figure 3 , an in-memory database persistence device applied to the in-memory database of the above server is introduced, wherein the in-memory database is pre-deployed with at least two key spaces, the key spaces are used to store data keys with update frequencies within a preset update frequency range, and the preset update frequency ranges corresponding to each of the key spaces are different. Figure 3 As shown, the memory database persistence device may include:
[0103] An interval determination unit 100, configured to determine a time interval between two adjacent memory snapshots of the key space according to a preset snapshot frequency corresponding to the key space, wherein the preset snapshot frequency is set according to the update frequency range corresponding to the key space;
[0104] The memory snapshot unit 200 is used to take a memory snapshot of the data key and the key value corresponding to the data key stored in the key space when the time length from the last memory snapshot reaches the time interval.
[0105] In summary, the embodiment of the present application takes into account the different update frequencies of different data keys in the memory database, deploys multiple key spaces in advance, and sets the update frequency range of the data keys used to store for each key space, so that data keys with different update frequencies can be stored in different key spaces according to their ranges. Each key space regularly persists the data keys and key values stored in the key space according to the time interval corresponding to the preset snapshot frequency.
[0106] Since the memory snapshot operations between the key spaces of this application are independent of each other and the memory snapshot frequencies are different, compared with the memory snapshot of the full amount of data at a specific time point in the prior art, this application performs memory snapshots at different frequencies for data keys with different update frequencies to ensure that the data in each update frequency range can be persisted in time and minimize the loss of data. And since the memory snapshot operations between key spaces are independent of each other, compared with the memory snapshot of the full amount of data in the prior art, the amount of data in each memory snapshot of this application is reduced, thereby achieving the effect of reducing the load pressure of the server each time the memory snapshot is performed.
[0107] In a possible implementation, the functional unit for pre-deploying the key space includes:
[0108] An update frequency acquisition unit, used to acquire the update frequency corresponding to each data key in the memory database;
[0109] A frequency range division unit, used to divide a preset number of update frequency ranges according to the update frequency;
[0110] A key space deployment unit, used to deploy a key space for each update frequency range respectively;
[0111] A data migration unit, configured to migrate the data keys whose update frequency belongs to the update frequency range to the key space corresponding to the update frequency range;
[0112] A key space configuration unit is used to configure an RDB program and a Bloom filter in the key space for performing memory snapshots at a preset snapshot frequency, wherein the Bloom filter is used to store the correspondence between the data keys stored in the key space and the update frequency range.
[0113] In a possible implementation, the functional unit in the key space configuration unit for determining the preset snapshot frequency corresponding to the key space includes:
[0114] A frequency determination subunit, used to determine the update frequency corresponding to each of the data keys stored in the key space;
[0115] The snapshot frequency determination subunit is used to use the average value of the update frequencies corresponding to all the data keys in the key space as the preset snapshot frequency of the key space.
[0116] In a possible implementation, the method further includes:
[0117] A frequency monitoring unit, used to monitor the update frequency of the data key corresponding to each of the key spaces;
[0118] a current frequency determination unit, configured to determine a current update frequency of the data key when the update frequency changes and the update frequency is not within the preset update frequency range corresponding to the key space;
[0119] A target space determination unit, configured to determine a target migration key space corresponding to the current update frequency according to the preset update frequency range corresponding to each key space;
[0120] The data migration unit is used to migrate the data key and the key value corresponding to the data key to the target migration key space.
[0121] In a possible implementation, the method further includes:
[0122] a target key space determination unit, configured to, in response to receiving an edit request for a target data key in the memory database, determine a target key space storing the target data key according to the data key stored in each key space or the preset update frequency;
[0123] An editing unit is used to edit the target data key in the target key space based on the editing request.
[0124] In a possible implementation, the target key space determination unit includes:
[0125] A key query subunit, configured to query the correspondence between the data key stored in each of the Bloom filters and the update frequency range, and determine a target Bloom filter that exists in the correspondence corresponding to the target data key;
[0126] The target determination subunit is used to determine the key space corresponding to the target Bloom filter as a target key space.
[0127] In a possible implementation, the target key space determination unit includes:
[0128] A frequency query subunit, used to determine the update frequency of the target data key;
[0129] a target filter determination subunit, configured to determine, according to the update frequency range stored in each of the Bloom filters, the Bloom filter corresponding to the update frequency range including the update frequency as a target Bloom filter;
[0130] The target determination subunit is used to determine the key space corresponding to the target Bloom filter as a target key space.
[0131] The in-memory database persistence apparatus provided in the embodiment of the present application can be applied to an in-memory database persistence device.
[0132] Figure 4 The schematic diagram of the structure of the memory database persistence device is shown. Figure 4 The structure of the memory database persistence device may include: at least one processor 10, at least one memory 20, at least one communication bus 30 and at least one communication interface 40.
[0133] In the embodiment of the present application, the number of the processor 10 , the memory 20 , the communication bus 30 , and the communication interface 40 is at least one, and the processor 10 , the memory 20 , and the communication interface 40 communicate with each other through the communication bus 30 .
[0134] The processor 10 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.
[0135] The memory 20 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory.
[0136] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement each processing flow in the aforementioned in-memory database persistence method.
[0137] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs suitable for execution by a processor. When the one or more computer programs are executed by a memory database persistence device, the memory database persistence device can implement each processing flow in the aforementioned memory database persistence method.
[0138] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0139] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0140] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for persisting a memory database, characterized in that: A memory database applied to a server, wherein the memory database is pre-deployed with at least two key spaces, the key spaces are used to store data keys whose update frequency is within a preset update frequency range, and the preset update frequency range corresponding to each key space is different; The memory database persistence method executed for each key space includes: Determine the time interval between two adjacent memory snapshots of the key space according to a preset snapshot frequency corresponding to the key space, wherein the preset snapshot frequency is set according to the update frequency range corresponding to the key space; When the time from the last memory snapshot reaches the time interval, a memory snapshot is taken of the data key and the key value corresponding to the data key stored in the key space.
2. The method for persisting a memory database according to claim 1, characterized in that: The process of pre-deploying the keyspace includes: Get the update frequency corresponding to each data key in the memory database; According to the update frequency, dividing the update frequency range into a preset number; Deploy a key space for each update frequency range respectively; Migrate the data key whose update frequency belongs to the update frequency range to the key space corresponding to the update frequency range; An RDB program and a Bloom filter for performing memory snapshots at a preset snapshot frequency are configured in the key space, and the Bloom filter is used to save the corresponding relationship between the data keys stored in the key space and the update frequency range.
3. The method for persisting a memory database according to claim 2, characterized in that: The process of determining the preset snapshot frequency corresponding to the key space includes: Determine an update frequency corresponding to each of the data keys stored in the key space; The average value of the update frequencies corresponding to all the data keys in the key space is used as the preset snapshot frequency of the key space.
4. The method for persisting a memory database according to claim 2, characterized in that: Also includes: Monitoring the update frequency of the data key corresponding to each of the key spaces; When the update frequency changes and the update frequency is not within the preset update frequency range corresponding to the key space, determining the current update frequency of the data key; Determine the target migration key space corresponding to the current update frequency according to the preset update frequency range corresponding to each key space; Migrate the data key and the key value corresponding to the data key to the target migration key space.
5. The method for persisting a memory database according to claim 2, characterized in that: Also includes: In response to receiving an edit request for a target data key in the in-memory database, determining a target key space storing the target data key according to the data key stored in each key space or the preset update frequency; Based on the edit request, the target data key in the target key space is edited.
6. The method for persisting a memory database according to claim 5, characterized in that: Determining a target key space storing the target data key according to the data key stored in each key space includes: Querying the correspondence between the data key and the update frequency range stored in each of the Bloom filters, and determining a target Bloom filter that exists in the correspondence corresponding to the target data key; The key space corresponding to the target Bloom filter is determined as the target key space.
7. The method for persisting a memory database according to claim 5, characterized in that: The step of determining a target key space storing the target data key according to the preset update frequency stored in each key space includes: determining an update frequency of the target data key; According to the update frequency range stored in each of the Bloom filters, determining the Bloom filter corresponding to the update frequency range including the update frequency as a target Bloom filter; The key space corresponding to the target Bloom filter is determined as the target key space.
8. A memory database persistence device, characterized in that: A memory database applied to a server, wherein the memory database is pre-deployed with at least two key spaces, the key spaces are used to store data keys whose update frequency is within a preset update frequency range, and the preset update frequency range corresponding to each key space is different; The memory database persistence device comprises: An interval determination unit, configured to determine a time interval between two adjacent memory snapshots of the key space according to a preset snapshot frequency corresponding to the key space, wherein the preset snapshot frequency is set according to the update frequency range corresponding to the key space; The memory snapshot unit is used to take a memory snapshot of the data key and the key value corresponding to the data key stored in the key space when the time length from the last memory snapshot reaches the time interval.
9. A memory database persistence device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the in-memory database persistence device can implement the in-memory database persistence method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that: The computer storage medium carries one or more computer programs, and when the one or more computer programs are executed by the memory database persistence device, the memory database persistence device can implement the memory database persistence method as described in any one of claims 1 to 7.