Memory snapshot generation method and device, electronic equipment and storage medium

Abstract

The application discloses a memory snapshot generation method and device, electronic equipment and a storage medium, and relates to the technical field of memory storage. The specific implementation scheme is as follows: a first sub-process is created when a system starts; a merge compression task is sent to the first sub-process in the case that a memory snapshot generation process is triggered, the merge compression task containing the file name of a first snapshot file and the file name of an incremental file; wherein the first snapshot file is used for recording the memory snapshot at the moment when the first snapshot file is generated, and the incremental file is used for recording operation instructions from the moment when the first snapshot file is generated to the current moment; and the execution result of the merge compression task is received. The embodiment of the application can improve system availability and reduce memory usage.

Description

TECHNICAL FIELD

[0001] The present application relates to the computer technology field, and in particular to the memory storage technology field. BACKGROUND

[0002] Redis is a high-performance key-value memory storage system, which supports multiple data structures, memory data persistence and master-slave synchronization function between nodes, and is widely used. The necessary step for Redis to complete memory data persistence and master-slave synchronization is to generate a memory snapshot. The current way to generate a Redis memory snapshot is: through a system call fork function, a child process is generated by the fork function, which completely contains all the memory data of the main process at this moment, and the child process saves this part of memory data to the hard disk to realize the persistence of memory data. This way is simple and effective, and the main process can generate a memory snapshot without additional operations, but there are two problems: the system availability decreases and the memory usage increases greatly. The specific reasons are as follows: first, the fork function call is a lightweight system function call, but when the process memory is larger, the call time is longer, which leads to a decrease in system availability. Second, the child process generated by the fork function itself does not consume a lot of memory, but the main process is always processing user requests, which leads to changes in memory data, triggering copy-on-write, and thus consuming a large amount of memory; usually, in order to ensure the successful generation of a memory snapshot, a part of memory needs to be allocated redundantly, and in the extreme case, the same size of memory as the Redis process memory is allocated redundantly, thereby increasing the memory cost by one time, leading to a great increase in memory usage. SUMMARY

[0003] In a first aspect, an embodiment of the present application provides a method for generating a memory snapshot, comprising:

[0004] When the system starts, a first child process is created;

[0005] In the case that the memory snapshot generation process is triggered, a merge and compression task is sent to the first child process, and the merge and compression task contains the file name of a first snapshot file and the file name of an incremental file; wherein the first snapshot file is used to record the memory snapshot at the time when the first snapshot file is generated, and the incremental file is used to record the operation instructions from the time when the first snapshot file is generated to the current time;

[0006] The execution result of the merge and compression task is received.

[0007] In an embodiment, the first snapshot file is a Redis database file, and the incremental file is a file for performing an append operation.

[0008] In an embodiment, in the case that the memory snapshot generation process is triggered, further comprising:

[0009] record a first mapping relationship between a file name of the incremental file and memory snapshot information, and record the file name of the incremental file; the memory snapshot information comprises relevant information of a memory snapshot generation process being triggered.

[0010] In an implementation, after receiving the execution result of the merging and compressing task, the method further comprises:

[0011] According to the file name of the incremental file corresponding to the execution result, the first mapping relationship is searched to obtain the memory snapshot information corresponding to the file name of the incremental file;

[0012] According to the memory snapshot information, a corresponding operation is performed.

[0013] In an implementation, the condition of the memory snapshot generation process being triggered comprises at least one of the following:

[0014] receiving a triggering instruction from outside;

[0015] triggering when a slave node is performing master-slave synchronization;

[0016] the data amount of the incremental file reaching a predetermined size.

[0017] In an implementation, the merging and compressing task further comprises master-slave replication information.

[0018] In an implementation, the method further comprises:

[0019] in a case where the execution result is correct compression, deleting the incremental file, generating a new incremental file, and the new incremental file being used to record operation instructions after the current time.

[0020] In a second aspect, an embodiment of the present application provides a memory snapshot generation method, comprising:

[0021] a first sub-process receives a merging and compressing task, the merging and compressing task comprising a file name of a first snapshot file and a file name of an incremental file; wherein the first sub-process is created when a system is started; the first snapshot file is used to record a memory snapshot at a time when the first snapshot file is generated, and the incremental file is used to record operation instructions from the time when the first snapshot file is generated to a current time;

[0022] the merging and compressing task is executed, the first snapshot file and the incremental file are merged and compressed to obtain a second snapshot file used to record a memory snapshot at the current time;

[0023] an execution result of the merging and compressing task is sent.

[0024] In an embodiment, the first snapshot file or the second snapshot file is a Redis database file, and the incremental file is a file on which an append operation is performed.

[0025] In an embodiment, the performing the merge and compression task comprises:

[0026] loading operation instructions recorded in the incremental file and storing the operation instructions in sequence according to keys in key-value pairs as a dimension;

[0027] reading the key-value pairs in the snapshot file;

[0028] for each key-value pair, performing an operation instruction corresponding to a key in the key-value pair to obtain a latest data state of the key, updating the key-value pair according to the latest data state, and deleting the key-value pair in the memory;

[0029] saving the updated key-value pair into the second snapshot file.

[0030] In an embodiment, the performing the merge and compression task further comprises:

[0031] performing remaining operation instructions in the incremental file and saving the performing result into the second snapshot file.

[0032] In an embodiment, the performing the merge and compression task further comprises:

[0033] in a case where the key in the key-value pair has no corresponding operation instruction, saving the key-value pair into the second snapshot file.

[0034] In an embodiment, the merge and compression task further comprises master-slave replication information.

[0035] The method further comprises saving the master-slave replication information into the second snapshot file.

[0036] In a third aspect, an embodiment of the present application provides a memory snapshot generation apparatus, comprising:

[0037] a creating module configured to create a first sub-process when a system is started;

[0038] a task sending module configured to send a merge and compression task to the first sub-process in a case where a memory snapshot generation process is triggered, the merge and compression task comprising a file name of a first snapshot file and a file name of an incremental file; wherein the first snapshot file is used to record a memory snapshot at a first snapshot file generation time, and the incremental file is used to record operation instructions from the first snapshot file generation time to a current time.

[0039] A result receiving module is configured to receive an execution result of the merged compression task.

[0040] In an embodiment, the apparatus further comprises:

[0041] A recording module is configured to record a first mapping relationship between a file name of the incremental file and memory snapshot information, and record the file name of the incremental file, if the memory snapshot generation process is triggered.

[0042] In an embodiment, the apparatus further comprises:

[0043] A searching module is configured to search the first mapping relationship according to the file name of the incremental file corresponding to the execution result, to obtain the memory snapshot information corresponding to the file name of the incremental file.

[0044] An operation executing module is configured to execute a corresponding operation according to the memory snapshot information.

[0045] In an embodiment, the condition under which the memory snapshot generation process is triggered comprises at least one of the following:

[0046] A trigger instruction from outside is received;

[0047] A slave node triggers when performing master-slave synchronization;

[0048] A data amount of the incremental file reaches a predetermined size.

[0049] In a fourth aspect, an embodiment of the present application provides a memory snapshot generation apparatus, comprising:

[0050] A task receiving module is configured to receive a merged compression task by using a first sub-process, the merged compression task comprising a file name of a first snapshot file and a file name of an incremental file; the first sub-process is created when a system is started; the first snapshot file is used to record a memory snapshot at a time when the first snapshot file is generated; and the incremental file is used to record operation instructions from the time when the first snapshot file is generated to a current time.

[0051] A task executing module is configured to execute the merged compression task by using the first sub-process, to perform merged compression on the first snapshot file and the incremental file, and to obtain a second snapshot file used to record a memory snapshot at the current time.

[0052] A result sending module is configured to send an execution result of the merged compression task by using the first sub-process.

[0053] In an implementation, the task execution module is configured to load the operation instructions recorded in the incremental file and store the operation instructions in sequence according to the keys in the key-value pairs; read the key-value pairs in the snapshot file; for each key-value pair, execute the operation instruction corresponding to the key in the key-value pair to obtain the latest data state of the key, update the key-value pair according to the latest data state, and delete the operation instruction corresponding to the key; and save the updated key-value pair into the second snapshot file.

[0054] In an implementation, the task execution module is further configured to execute the remaining operation instructions in the incremental file and save the execution result into the second snapshot file.

[0055] In an implementation, the task execution module is further configured to, in the case that the key in the key-value pair does not correspond to an operation instruction, save the key-value pair into the second snapshot file.

[0056] The above embodiment has the following advantages or beneficial effects: since the first sub-process for merging the first snapshot file and the incremental file is created at the system startup, the first sub-process does not contain the data of the main process, which is different from the prior art in which a sub-process is generated by calling the fork function when the main process generates a memory snapshot, and the memory snapshot is generated by the sub-process, thereby fundamentally eliminating the impact of the fork call on the system, improving the system availability, and reducing the memory usage. The embodiment of the present application uses the RDB file and the AOF file in the prior art, and realizes the memory snapshot by merging the RDB file and the AOF file. In the case that the memory snapshot is triggered, the main process can also record the mapping relationship between the AOF file name and the memory snapshot information, and record the AOF file name, to realize the recording of the memory snapshot information, thereby ensuring that after the memory snapshot is completed, the necessary operations can be performed according to the recording. The conditions for triggering the memory snapshot generation process in the embodiment of the present application can include receiving an external instruction, performing master-slave synchronization from a node, and the data amount of the AOF file reaching a predetermined size. Among them, the first two triggering conditions can be used to realize data saving and master-slave synchronization, and the last triggering condition can cut the AOF file to improve the execution efficiency of the merging and compression operation and save memory. After cutting the AOF file, a new AOF file can be generated to record the Redis operation instructions from the current time (including the current time). In addition, the merging and compression task sent by the main process to the first sub-process can also include master-slave replication information, so that the first sub-process writes the master-slave replication information into the new RDB file, thereby meeting the demand of master-slave synchronization.

[0057] Other effects of the above optional mode will be described in the following with specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present application. Among them:

[0059] Figure 1 Implementation flow of a memory snapshot generation method according to an embodiment of the present application Figure One ;

[0060] Figure 2 Implementation flow of a memory snapshot generation method according to an embodiment of the present application Figure Two ;

[0061] Figure 3 Implementation flow of a memory snapshot generation method according to an embodiment of the present application Figure Three ;

[0062] Figure 4 Implementation flow of step S302 in a memory snapshot generation method according to an embodiment of the present application

[0063] Figure 5 Implementation schematic diagram of loading and storing Redis operation instructions in a memory snapshot generation method according to an embodiment of the present application

[0064] Figure 6 Implementation flow of a memory snapshot generation method according to an embodiment of the present application

[0065] Figure 7 Interaction process schematic diagram of a Redis master process and a Compact child process in a memory snapshot generation method according to an embodiment of the present application

[0066] Figure 8A Comparison schematic diagram of response delay time using the method proposed in the present application and using an existing method

[0067] Figure 8B Comparison schematic diagram of reserved memory size using the method proposed in the present application and using an existing method

[0068] Figure 9 Structure schematic diagram of a memory snapshot generation device according to an embodiment of the present application Figure One ;

[0069] Figure 10 Structure schematic diagram of a memory snapshot generation device according to an embodiment of the present application Figure Two ;

[0070] Figure 11 Structure schematic diagram of a memory snapshot generation device according to an embodiment of the present application Figure Three;

[0071] Figure 12 is a block diagram of an electronic device for implementing the memory snapshot generation method according to an embodiment of the present application. DETAILED DESCRIPTION

[0072] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, in which various details are set forth to assist in an understanding of the embodiments of the present application. It will be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the spirit and scope of the present application. Thus, the present application is not intended to be limited to the embodiments described herein but is to be accorded the widest scope of the appended claims. Also, for the sake of brevity, descriptions of functions and constructions that are well known to those skilled in the art can be omitted.

[0073] The present application provides a memory snapshot generation method, Figure 1 The present application provides a memory snapshot generation method, Figure One , comprising:

[0074] Step S101: When the system is started, a first sub-process is created.

[0075] Step S102: When the memory snapshot generation process is triggered, a merge and compression task is sent to the first sub-process, the merge and compression task containing a file name of a first snapshot file and a file name of an incremental file; wherein the first snapshot file is used to record a memory snapshot at a time when the first snapshot file is generated, and the incremental file is used to record operation instructions from the time when the first snapshot file is generated to a current time.

[0076] Step S103: An execution result of the merge and compression task is received.

[0077] The memory snapshot generation method according to the present application can be used to generate a Redis memory snapshot; accordingly, the system described above can refer to a Redis storage system, and the operation instructions described above can refer to Redis operation instructions.

[0078] The above process can be applied to a master process of a Redis system. The first sub-process is used to execute the merge and compression task, and the first sub-process can be named as a Compact sub-process. In step S103, the master process receives an execution result fed back by the Compact sub-process.

[0079] In a possible implementation, the first snapshot file is a Redis database (RDB, Redis Datebase) file, and the incremental file is an append operation file (AOF, Append Only File).

[0080] The present application provides another memory snapshot generation method, Figure 2A method for generating a memory snapshot according to an embodiment of the present application Figure Two As shown in Figure 2 the embodiment, the embodiment further includes, relative to the above embodiment:

[0081] Step S201: In a case where a memory snapshot generation process is triggered, record a first mapping relationship between a file name of a delta file and memory snapshot information, and record the file name of the delta file; wherein the memory snapshot information includes related information of the memory snapshot generation process being triggered.

[0082] The above step S103 can further include:

[0083] Step S202: Find the first mapping relationship according to a file name of a delta file corresponding to an execution result, to obtain memory snapshot information corresponding to the file name of the delta file;

[0084] Step S203: Perform a corresponding operation according to the memory snapshot information.

[0085] Step S204: In a case where the execution result is correct compression, delete the delta file, and generate a new delta file, the new delta file being used to record operation instructions after a current time.

[0086] The above steps S201 to S204 can be performed by a Redis master process.

[0087] In a possible implementation, the condition under which the memory snapshot generation process is triggered includes at least one of the following:

[0088] A trigger instruction is received from outside;

[0089] A slave node triggers when performing master-slave synchronization;

[0090] A data amount of the delta file reaches a predetermined size.

[0091] In the above embodiment, the merge compression task can further include master-slave replication information, which can be used for saving when the first sub-process generates a new snapshot file.

[0092] The embodiment of the present application further provides a method for generating a memory snapshot, which can be applied to the above first sub-process. As shown in Figure 3 A method for generating a memory snapshot according to an embodiment of the present application Figure Three , including:

[0093] Step S301: The first sub-process receives a merge and compression task, the merge and compression task including a file name of a first snapshot file and a file name of an incremental file; wherein the first sub-process is created when the system starts; the first snapshot file is used to record a memory snapshot at a time when the first snapshot file is generated, and the incremental file is used to record operation instructions from the time when the first snapshot file is generated to a current time;

[0094] Step S302: The merge and compression task is executed to merge and compress the first snapshot file and the incremental file to obtain a second snapshot file used to record a memory snapshot at the current time;

[0095] Step S303: An execution result of the merge and compression task is sent.

[0096] In the step S301, the first sub-process can receive the merge and compression task from a Redis master process; and in the step S303, the first sub-process can send the execution result of the compression task to the Redis master process. The operation instructions can be Redis operation instructions.

[0097] In a possible implementation, the first snapshot file or the second snapshot file is an RDB file, and the incremental file is an AOF file.

[0098] Figure 4 For a method for generating a memory snapshot according to an embodiment of the present application, a flowchart for implementing the step S302 includes:

[0099] Step S401: Operation instructions recorded in the incremental file are loaded and stored in sequence according to a key in a key-value pair as a dimension;

[0100] Step S402: The key-value pair in the snapshot file is read;

[0101] Step S403: For each key-value pair, an operation instruction corresponding to a key in the key-value pair is executed to obtain a latest data state of the key, the key-value pair is updated according to the latest data state, and the key-value pair is deleted in the memory;

[0102] Step S404: The updated key-value pair is saved into the second snapshot file.

[0103] As shown in the step S402, the step S402 can further include: Figure 4

[0104] Step S405: In a case where the key in the key-value pair does not have a corresponding operation instruction, the key-value pair is saved into the second snapshot file.

[0105] After the operations on all the key-value pairs are completed, the method can further include: ​

[0106] Step S406: execute the remaining operation instructions in the incremental file, and save the execution result into the second snapshot file.

[0107] In a possible implementation, the merging and compressing task further comprises master-slave replication information.

[0108] Correspondingly, the method further comprises saving the master-slave replication information into the second snapshot file.

[0109] The above is the introduction of the embodiments of the application from the perspective of the Redis master process and the first slave process. The following introduces the embodiments of the application from the perspective of the interaction between the Redis master process and the first slave process.

[0110] For ease of description, in the following embodiments, the first slave process is referred to as a Compact slave process, the merging and compressing task is referred to as a Compact task, the first snapshot file is referred to as a stock RDB file, the second snapshot file is referred to as a new RDB file, and the incremental file is referred to as an incremental AOF file.

[0111] The RDB file is a persistent disk file of the Redis memory data, and records the memory snapshot at a certain moment. The incremental AOF file records the Redis operation instructions, that is, the operation records of the Redis database. The application adopts the stock RDB file and the incremental AOF file to construct the memory snapshot at a certain moment, that is, generates a new RDB file, and the new RDB file records the memory snapshot at the current moment. Then, a new incremental AOF file is generated, and the new incremental AOF file records the Redis operation instructions starting from the current moment. Next time the memory snapshot is generated, the RDB file (which is the new RDB file when it is generated, but its identity has been converted into the stock RDB file) and the incremental AOF file generated before are used to construct the memory snapshot, and a new RDB file is generated. This is repeatedly executed. The application creates a Compact slave process when the Redis is started, and the role of the Compact slave process is to generate the Redis memory snapshot.

[0112] Since the Compact slave process in the application is created when the Redis is started, rather than when the master process is triggered to generate the memory snapshot, the Compact slave process does not contain the data of the master process; this is different from the prior art in which the slave process containing all the data of the master process is generated by calling the fork function when the master process is triggered to generate the memory snapshot. Therefore, the application fundamentally eliminates the impact of the fork call on the system, and can improve the system availability and reduce the memory usage.

[0113] The role of the Compact sub-process is to receive the Compact task sent by the main process, and to combine and compress the specified inventory RDB file and the incremental AOF file into a new RDB file. The main work of the Compact sub-process is divided into two parts, one part is the processing of the incremental AOF file, and the other part is to perform the combination and compression of the inventory RDB file and the incremental AOF file.

[0114] Specifically, when processing the incremental AOF file, the Compact sub-process loads the Redis operation instructions in the incremental AOF file, and stores the Redis operation instructions in sequence according to the Key. Figure 5 For the implementation schematic diagram of loading and storing Redis operation instructions in the generation method of a memory snapshot according to an embodiment of the present application. Redis supports multiple DBs, and the SELECT instruction in the incremental AOF file records the DB corresponding to the current Redis operation instruction. As shown in Figure 5 After the Compact sub-process loads the Redis operation instructions in the incremental AOF file, it first distinguishes which DB the Redis operation instructions are directed to, and for each DB, records the Redis operation instructions for each Key. The Redis operation instruction record for each Key is arranged in time order to ensure that the results after subsequent operations on the Key are consistent with the state of the main process. Among them, the Redis operation instructions are all determined, and this part of the instructions can be in the form of shared objects to save memory. In addition, for special instructions such as the SCRIPT instruction, the LUA instruction, the FLUSHALL instruction, the FLUSHDB instruction, the SWAPDB instruction, and the like, semantic-level analysis needs to be performed, and the Redis operation instructions stored are processed accordingly according to the analysis result. As shown in Figure 5 The delete operation of the Redis operation instructions for Key1 and Key2 in database DB3 is shown, and the exchange operation of operate_3 for Key1 in database DB1 and operate_3 for Key3 in database DB2 is shown. These operations are only examples, and the processing method of the Redis operation instructions is not limited by the embodiments of the present application.

[0115] Compared with the sub-process invoked by the fork function in the prior art, the Compact sub-process can effectively save memory consumption. In the prior art, 50-100% of memory redundancy is generally reserved for the sub-process invoked by the fork function, while the Compact sub-process in the present application uses memory basically equal to the size of the AOF file. For example, for a Redis instance with a memory size of 10G, 5G-10G of memory needs to be reserved for the sub-process invoked by the fork function in the prior art; while in the present application, the AOF file is generally 1G at most, plus the data structure inside the Compact sub-process, the Compact sub-process consumes about 1.5G of memory in total, which saves 70-85% of memory compared with the prior art; for a Redis instance with a larger memory, the present application saves more memory.

[0116] The core process of the Compact sub-process is to merge and compress the inventory RDB file and the incremental AOF file. In order to save the use of memory by the Compact sub-process, the present application can read in a piece of information from the inventory RDB file in a streaming manner. For information of non-data type, such as database information, LUA script and auxiliary field, the Compact sub-process directly records. For information of data type, i.e. content recorded in the Key-Value structure, the Compact sub-process executes all Redis operation records of the Key in the incremental AOF file, so as to obtain the latest data state of the Key, then persistently stores the updated Key-Value (i.e. writes into the new RDB file), and deletes the Key-Value in the memory to save the use of memory; if there is no change record (i.e. no corresponding Redis operation record) of the Key in the incremental AOF file, the Key-Value read from the inventory RDB file can be directly saved into the new RDB file, so as to improve the generation efficiency of the new RDB file.

[0117] Figure 6 For the generation method of the memory snapshot according to the embodiment of the present application, the process flow chart of the merging and compressing process of the inventory RDB file and the incremental AOF file comprises:

[0118] Step S601: judge whether the inventory RDB file is read completely, if yes, execute step S602; if no, execute step S603.

[0119] Step S602: execute the remaining instructions in the incremental AOF file, and perform persistence. After step S602 is executed, the current process is ended.

[0120] Step S603: read a piece of information from the inventory RDB file.

[0121] Step S604: Determine the type of the information. If it is a non-data type, proceed to step S605; if it is a data type, proceed to step S606.

[0122] Step S605: Record this information. For example, this information may be database information or LUA information. After step S605 is completed, return to step S601.

[0123] Step S606: Determine whether the Redis operation command corresponding to the Key in the incremental AOF file exists. If it exists, proceed to step S607; otherwise, proceed to step S608.

[0124] Step S607: Obtain the Redis operation instructions corresponding to the key in the incremental AOF file, execute these Redis operation instructions in sequence to obtain the updated key-value, and persist the updated key-value (that is, save the updated key-value into a new RDB file). Continue to execute step S609.

[0125] Step S608: Append the data of the Key in the existing RDB file to the new RDB file. Proceed to step S609.

[0126] Step S609: Delete the Key-Value pair from memory. Return to step S601.

[0127] pass Figure 6 The process described above achieves the merging and compression of existing RDB files and incremental AOF files.

[0128] In one possible implementation, when Redis receives a trigger command from an external source (such as the BGSAVE command) or performs a full synchronization with a slave node, Redis needs to generate a memory snapshot to achieve data preservation or master-slave synchronization. After the Compact subprocess completes the merging and compression, it notifies the master process and completes subsequent operations. Furthermore, to improve the execution efficiency of the Compact subprocess and save memory, the generation of a memory snapshot can also be triggered when the incremental AOF file reaches a predetermined size. When the incremental AOF file reaches the predetermined size, a memory snapshot is triggered, merging the existing RDB file with the incremental AOF file, then generating a new incremental AOF file, and saving the newly generated Redis operation commands into the new incremental AOF file. This method achieves the splitting of the incremental AOF file. It is important to note that when splitting the incremental AOF file, it is necessary to ensure that all Redis operation commands generated before the splitting point have been saved into the incremental AOF file.

[0129] The Redis master process and the Compact child process can communicate in a pipe manner. The main content of the communication includes the file name of the inventory RDB file that needs to be compressed and merged and the file name of the incremental AOF file. In addition, in order to be compatible with the master-slave replication information recorded in the inventory RDB file by Redis, the communication content can also include the master-slave replication information, such as the database ID of the master-slave replication, the master-slave replication identification ID, the offset of the master-slave replication, and the like. The Compact child process can write the part of the master-slave replication information into the new RDB file.

[0130] Figure 7 For the generation method of the memory snapshot according to the embodiment of the application, a schematic diagram of the interaction process between the Redis master process and the Compact child process is shown in FIG. 7. The process includes the following steps.

[0131] Step S701: The master process is triggered to generate a memory snapshot.

[0132] Since the master process needs to perform subsequent operations (such as master-slave synchronization) according to the condition triggered at that time, it is necessary to record the memory snapshot information. Since the file name of the incremental AOF file can uniquely identify a memory snapshot information, the memory snapshot information can be recorded in the following step S702. The memory snapshot information is different from the memory snapshot described above, and can be the trigger condition, trigger time, trigger subject, and the like of the memory snapshot generation.

[0133] Step S702: The master process records the mapping relationship between the file name of the incremental AOF file and the memory snapshot information, and records the file name of the current incremental AOF file. Then, the master process returns a confirmation of the memory snapshot generation.

[0134] In addition, since Redis supports multi-Key operations, and the Compact child process needs to sort the Key, the Redis operation instruction written in the incremental AOF file must be a single-Key operation, so it is necessary to convert the multi-Key operation to a single-Key operation in the Redis master process.

[0135] Step S703: The master process sends a Compact task to the Compact child process.

[0136] Step S704: The Compact child process performs the merging and compression of the inventory RDB file and the incremental AOF file.

[0137] Step S705: The Compact child process returns the execution result of the Compact task to the master process.

[0138] Step S706: The main process finds the mapping relationship according to the file name of the current incremental AOF file, obtains the corresponding memory snapshot information, and performs necessary operations according to the memory snapshot information.

[0139] In addition, for the case of triggering memory snapshot generation by master-slave replication, after the slave node completes full data replication, the Compact task of the current Compact child process should be terminated to avoid logical conflicts between the two processes due to the modification of the RDB file when the child process completes the merging and compression task.

[0140] As can be seen from the above embodiments, the present application eliminates the influence of the existing fork child process for generating a memory snapshot, thereby improving system availability and reducing memory usage. Figure 8A and Figure 8B The performance comparison chart when generating a memory snapshot using the present application and the existing method is shown. Figure 8A The response delay time comparison chart when using the method proposed in the present application and using the existing method is shown. As shown in Figure 8A , when the Redis instance memory size is 10GB, the request response delay caused by using the existing open source solution is 127ms, and the request response delay caused by using the method proposed in the present application is 1ms; when the Redis instance memory size is 20GB, the request response delay caused by using the existing open source solution is 287ms, and the request response delay caused by using the method proposed in the present application is 1ms; when the Redis instance memory size is 30GB, the request response delay caused by using the existing open source solution is 422ms, and the request response delay caused by using the method proposed in the present application is 1ms. (Due to the display ratio problem, the bar chart of the request response delay caused by using the method proposed in the present application in the above three cases cannot be clearly displayed in Figure 8A ). Figure 8B The reserved memory size comparison chart when using the method proposed in the present application and using the existing method is shown. As shown in Figure 8B , when the Redis instance memory size is 10GB, the reserved memory required by using the existing open source solution is 5GB, and the reserved memory required by using the method proposed in the present application is 1.5GB; when the Redis instance memory size is 20GB, the reserved memory required by using the existing open source solution is 10GB, and the reserved memory required by using the method proposed in the present application is 2.58GB; when the Redis instance memory size is 30GB, the reserved memory required by using the existing open source solution is 15GB, and the reserved memory required by using the method proposed in the present application is 2.6GB.

[0141] The present application also proposes a memory snapshot generation device, which can be applied to a Redis master process.Figure 9 Structure diagram of a memory snapshot generation device according to an embodiment of the present application Figure One , comprising:

[0142] The creating module 901 is configured to create a first sub-process when the system starts;

[0143] The task sending module 902 is configured to send a merge and compression task to the first sub-process in a case where a memory snapshot generation process is triggered, the merge and compression task containing a file name of a first snapshot file and a file name of an incremental file; the first snapshot file is used to record a memory snapshot at a time when the first snapshot file is generated, and the incremental file is used to record operation instructions from the time when the first snapshot file is generated to a current time;

[0144] The result receiving module 903 is configured to receive an execution result of the merge and compression task.

[0145] Figure 10 Structure diagram of a memory snapshot generation device according to an embodiment of the present application Figure Two , comprising: the creating module 901, the task sending module 902, the result receiving module 903, a recording module 1004, a searching module 1005 and an operation executing module 1006; the creating module 901, the task sending module 902 and the result receiving module 903 have the same functions as the corresponding modules in the above embodiment, and will not be described again.

[0146] The recording module 1004 is configured to record a first mapping relationship between a file name of an incremental file and memory snapshot information and record the file name of the incremental file in a case where a memory snapshot generation process is triggered; the memory snapshot information includes related information of the memory snapshot generation process being triggered.

[0147] The searching module 1005 is configured to search the first mapping relationship according to a file name of an incremental file corresponding to an execution result, to obtain memory snapshot information corresponding to the file name of the incremental file.

[0148] The operation executing module 1006 is configured to execute corresponding operations according to the memory snapshot information.

[0149] In a possible implementation, the condition under which the memory snapshot generation process is triggered includes at least one of the following:

[0150] A trigger instruction from outside is received;

[0151] A slave node triggers when performing master-slave synchronization;

[0152] A data amount of the incremental file reaches a predetermined size.

[0153] The embodiment of the present application further provides another memory snapshot generation device, which can be applied to the first sub-process (for example, the Compact sub-process). Figure 11 The memory snapshot generation device according to the embodiment of the present application has the structure as shown in the figure Figure Three , and includes:

[0154] The task receiving module 1101 is configured to receive a merge and compression task by using the first sub-process, wherein the merge and compression task contains the file name of the first snapshot file and the file name of the incremental file; the first sub-process is created when the system is started; the first snapshot file is used to record the memory snapshot at the time when the first snapshot file is generated, and the incremental file is used to record the operation instruction from the time when the first snapshot file is generated to the current time.

[0155] The task execution module 1102 is configured to execute the merge and compression task by using the first sub-process, and perform merge and compression on the first snapshot file and the incremental file to obtain a second snapshot file used to record the memory snapshot at the current time.

[0156] The result sending module 1103 is configured to send the execution result of the merge and compression task by using the first sub-process.

[0157] In a possible implementation, the task execution module 1102 is configured to load the operation instruction recorded in the incremental file, and store the operation instruction in sequence according to the key in the key-value pair as the dimension; read the key-value pair in the snapshot file; for each key-value pair, execute the operation instruction corresponding to the key in the key-value pair to obtain the latest data state of the key, update the key-value pair according to the latest data state, and delete the operation instruction corresponding to the key; and save the updated key-value pair into the second snapshot file.

[0158] In a possible implementation, the task execution module 1102 is further configured to execute the remaining operation instruction in the incremental file, and save the execution result into the second snapshot file.

[0159] In a possible implementation, the task execution module 1102 is further configured to, in the case that the key in the key-value pair does not have a corresponding operation instruction, save the key-value pair into the second snapshot file.

[0160] The functions of each module in each device of the embodiment of the present application can be referred to the corresponding description in the above method, which will not be repeated here.

[0161] According to the embodiment of the present application, the present application further provides an electronic device and a readable storage medium.

[0162] As Figure 12As shown in FIG. 1, a block diagram of an electronic device for implementing the method for generating a memory snapshot according to an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown in the FIG. 1, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0163] As shown in FIG. 1, Figure 12 The electronic device includes one or more processors 1201, memory 1202, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected using different buses, and can be mounted on a common main board or otherwise mounted as desired. The processor can process instructions executed within the electronic device, including graphical information stored in the memory or on the memory to display a graphical user interface (GUI) on an external input / output device, such as a display device coupled to the interface. In other embodiments, multiple processors and / or buses can be used with multiple memories and multiple memory, if desired. Also, multiple electronic devices can be connected, each device providing part of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Figure 12 The processor 1201 is taken as an example in the embodiment.

[0164] The memory 1202 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by at least one processor, so that the at least one processor executes the method for generating a memory snapshot provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the method for generating a memory snapshot provided by the present application.

[0165] The memory 1202 is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method of XXX in the embodiment of the present application (for example, the creation module 901, the task sending module 902 and the result receiving module 903 shown in FIG. 1). Figure 9 The processor 1201 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 1202, that is, implements the method for generating a memory snapshot in the above method embodiments.

[0166] The memory 1202 can include a program storage area and a data storage area, where the program storage area can store an operating system, at least one application required by a function, and the data storage area can store data created based on the generation of the memory snapshot, etc. In addition, the memory 1202 can include a high-speed random access memory, and can further include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid state memory device. In some embodiments, the memory 1202 can optionally include a memory disposed remotely with respect to the processor 1201, which can be connected to the generation of the memory snapshot by the electronic device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0167] The electronic device of the generation method of the memory snapshot can further include an input device 1203 and an output device 1204. The processor 1201, the memory 1202, the input device 1203, and the output device 1204 can be connected by a bus or other means, Figure 12 for example, by a bus connection.

[0168] The input device 1203 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device of the generation method of the memory snapshot, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 1204 can include a display device, an auxiliary lighting device (e.g., an LED), a tactile feedback device (e.g., a vibration motor), etc. The display device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0169] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0170] These computer programs (also known as programs, software, software applications or code) include machine instructions for the programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0171] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0172] The systems and techniques described here can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0173] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0174] According to the technical scheme of the embodiment of the present application, the Compact child process for performing the merging and compression of the inventory RDB file and the incremental AOF file is created when Redis starts. Since the Compact child process is created when Redis starts, rather than being created when the main process generates a memory snapshot is triggered, the Compact child process does not contain the data of the main process; this is different from the prior art in which a child process containing all the data of the main process is generated by calling the fork function when the main process generates a memory snapshot is triggered. Therefore, the present application fundamentally eliminates the impact of the fork call on the system, and can improve system availability and reduce memory usage.

[0175] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical scheme disclosed in the present application can be achieved, and the present application is not limited herein.

[0176] The specific embodiments described above are not meant to limit the scope of the present application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives of the specific embodiments described above can be made depending on design requirements and other factors. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A method for generating a memory snapshot, the method comprising: The method comprises the following steps: When the Redis system is started, a first child process is created; The first child process is used to execute a merge and compression task; In a case where a memory snapshot generation process is triggered, a merge and compression task is sent to the first child process, the merge and compression task containing a file name of a first snapshot file and a file name of an incremental file; wherein the first snapshot file is used to record a memory snapshot at a time when the first snapshot file is generated, and the incremental file is used to record operation instructions from the time when the first snapshot file is generated to a current time; An execution result of the merge and compression task is received; the receiving of the execution result of the merge and compression task comprises that a master process of the Redis system receives an execution result fed back by the first child process; An execution process of the merge and compression task is applied to the first child process, and the execution process of the merge and compression task comprises the following steps: Loading the operation instructions recorded in the incremental file, and storing the operation instructions in sequence according to keys in key-value pairs as a dimension; Reading the key-value pairs in the first snapshot file; For each key-value pair, executing the operation instruction corresponding to the key in the key-value pair to obtain a latest data state of the key, updating the key-value pair according to the latest data state, and deleting the key-value pair in the memory; Saving the updated key-value pair into a second snapshot file.

2. The method of claim 1, wherein, The first snapshot file is a Redis database file, and the incremental file is a file on which an append operation is performed.

3. The method according to claim 1 or 2, characterized in that, In a case where the memory snapshot generation process is triggered, the following steps are further included: Recording a first mapping relationship between the file name of the incremental file and memory snapshot information, and recording the file name of the incremental file; the memory snapshot information comprises related information of the memory snapshot generation process being triggered.

4. The method of claim 3, wherein, After the execution result of the merge and compression task is received, the following steps are further included: Finding the first mapping relationship according to the file name of the incremental file corresponding to the execution result to obtain memory snapshot information corresponding to the file name of the incremental file; Performing a corresponding operation according to the memory snapshot information.

5. The method according to claim 1 or 2, characterized in that, The condition under which the memory snapshot generation process is triggered comprises at least one of the following: Receiving a trigger instruction from the outside; Triggering when a slave node performs master-slave synchronization; The data amount of the incremental file reaches a predetermined size.

6. The method of claim 1 or 2, wherein, The merge and compression task further contains master-slave replication information.

7. The method according to claim 1 or 2, characterized in that, Further comprising the following steps: In a case where the execution result is correct compression, the incremental file is deleted, and a new incremental file is generated, the new incremental file being used to record operation instructions after the current time.

8. A method of generating a memory snapshot, the method comprising: The method comprises the following steps: The first child process receives a merge and compression task, the merge and compression task containing a file name of a first snapshot file and a file name of an incremental file; wherein the first child process is created when a Redis system is started, and the first child process is used to execute a merge and compression task; the first snapshot file is used to record a memory snapshot at a time when the first snapshot file is generated, and the incremental file is used to record operation instructions from the time when the first snapshot file is generated to a current time; The merge and compression task is executed, and the first snapshot file and the incremental file are merged and compressed to obtain a second snapshot file used to record a memory snapshot at the current time; sending an execution result of the merge and compression task; the sending of the execution result of the merge and compression task comprises feeding back the execution result to a master process of a Redis system; the execution of the merge and compression task comprises: loading operation instructions recorded in the incremental file and storing the operation instructions in sequence according to keys in key-value pairs as dimensions; reading the key-value pairs in the first snapshot file; for each key-value pair, executing an operation instruction corresponding to a key in the key-value pair to obtain a latest data state of the key, updating the key-value pair according to the latest data state, and deleting the key-value pair in the memory; saving the updated key-value pair into the second snapshot file.

9. The method of claim 8, wherein, The first snapshot file or the second snapshot file is a Redis database file, and the incremental file is a file for performing an append operation.

10. The method of claim 8, wherein, The execution of the merge and compression task further comprises: executing the remaining operation instructions in the incremental file and saving the execution result into the second snapshot file.

11. The method of claim 8, wherein, The execution of the merge and compression task further comprises: in the case where the key in the key-value pair does not have a corresponding operation instruction, saving the key-value pair into the second snapshot file.

12. The method of claim 8 or 9, wherein, The merge and compression task further comprises master-slave replication information. The method further comprises saving the master-slave replication information into the second snapshot file.

13. An apparatus for generating a memory snapshot, the apparatus comprising: comprises: a creating module configured to create a first child process when a Redis system starts; the first child process is configured to execute a merge and compression task; a task sending module configured to send a merge and compression task to the first child process in the case where a memory snapshot generation process is triggered, the merge and compression task comprising a file name of a first snapshot file and a file name of an incremental file; the first snapshot file is configured to record a memory snapshot at a time when the first snapshot file is generated, and the incremental file is configured to record operation instructions from the time when the first snapshot file is generated to a current time; a result receiving module configured to receive an execution result of the merge and compression task; the result receiving module is specifically configured to receive an execution result fed back by the first child process by a master process of the Redis system; an execution process of the merge and compression task is applied to the first child process, and the execution process of the merge and compression task comprises: loading operation instructions recorded in the incremental file and storing the operation instructions in sequence according to keys in key-value pairs as dimensions; reading the key-value pairs in the snapshot file; for each key-value pair, executing an operation instruction corresponding to a key in the key-value pair to obtain a latest data state of the key, updating the key-value pair according to the latest data state, and deleting the key-value pair in the memory; saving the updated key-value pair into the second snapshot file.

14. The apparatus of claim 13, wherein, further comprises: a recording module configured to record a first mapping relationship between a file name of the incremental file and memory snapshot information and record the file name of the incremental file in the case where the memory snapshot generation process is triggered; the memory snapshot information comprises related information of the memory snapshot generation process.

15. The apparatus of claim 14, wherein, the apparatus further comprises: The searching module is configured to search the first mapping relationship according to the file name of the incremental file corresponding to the execution result, and obtain the memory snapshot information corresponding to the file name of the incremental file. The operation execution module is configured to execute a corresponding operation according to the memory snapshot information.

16. The apparatus of claim 13, wherein, The condition under which the memory snapshot generation process is triggered includes at least one of the following: a trigger instruction is received from outside; the slave node triggers when performing master-slave synchronization; a data amount of the incremental file reaches a predetermined size.

17. An apparatus for generating a memory snapshot, the apparatus comprising: The method comprises: The task receiving module is configured to receive a merge and compression task by using a first sub-process, the merge and compression task including a file name of a first snapshot file and a file name of an incremental file; the first sub-process is created when a Redis system is started, and the first sub-process is configured to execute the merge and compression task; the first snapshot file is configured to record a memory snapshot at a time when the first snapshot file is generated, and the incremental file is configured to record operation instructions from the time when the first snapshot file is generated to a current time; The task execution module is configured to execute the merge and compression task by using the first sub-process, and perform merge and compression on the first snapshot file and the incremental file to obtain a second snapshot file configured to record a memory snapshot at the current time; The result sending module is configured to send an execution result of the merge and compression task by using the first sub-process; and the result sending module is specifically configured to feed back the execution result to a master process of the Redis system. The task execution module is configured to load the operation instructions recorded in the incremental file, and store the operation instructions in sequence according to a key in a key-value pair as a dimension; read the key-value pairs in the snapshot file; for each key-value pair, execute the operation instruction corresponding to the key in the key-value pair to obtain a latest data state of the key, update the key-value pair according to the latest data state, and delete the operation instruction corresponding to the key; and save the updated key-value pair into the second snapshot file.

18. The apparatus of claim 17, wherein The task execution module is further configured to execute the remaining operation instructions in the incremental file, and save the execution result into the second snapshot file.

19. The apparatus of claim 17, wherein The task execution module is further configured to, in a case where the key in the key-value pair does not have a corresponding operation instruction, save the key-value pair into the second snapshot file.

20. An electronic device, comprising: The apparatus comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-12.

21. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are configured to enable the computer to execute the method of any one of claims 1-12.

22. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions enable the processor to execute the method of any one of claims 1-12 when executed by the processor.

Images