Data processing method, system, equipment and medium
By executing snapshot tasks on the slave nodes of the in-memory database system, the master node avoids the execution of snapshot generation tasks, solves the memory bloat problem when memory data is modified, and efficient memory resource utilization and rapid response to user requests are achieved.
Patent Information
- Application Number
- CN202311014909.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-11
- Publication Date
- 2025-02-18
AI Technical Summary
In the in-memory database system, the log files generated by the master node when performing data processing need to be persisted, resulting in memory bloating and wasting resources when modifying memory data.
The snapshot task is performed by selecting a target slave node among multiple slave nodes, without the master node performing the snapshot generation task. When the master node receives a request for memory data modification, it can directly modify it in-situ in memory to avoid creating a copy of the data.
It effectively avoids memory bloat problem, improves memory resource utilization, and responds to user data modification requests in a timely manner.
Smart Images
Figure CN119474037A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to data processing methods, systems, devices and media. Background Art
[0002] In-memory database systems, they are usually composed of a master node (Master) and multiple slave nodes (Slave). Among them, the slave node (also called the standby node) is used to keep data synchronized with the master node. When the master node cannot execute a task, the standby node replaces the master node to execute the task.
[0003] In actual applications, when the master node performs data processing, it will generate log files in real time and perform persistent processing on the log files (for example, storing them in the form of snapshots). In order to keep the master node and the slave node synchronized, each node will store data in its own local file system. However, in the process of executing the persistent processing task, there may be a need to modify some memory data. In order not to affect the normal execution of the persistent processing task, a copy of the corresponding memory data is usually created. When creating a copy, it is easy to cause memory expansion and waste memory resources. Summary of the invention
[0004] In order to solve or improve the problems existing in the prior art, the embodiments of the present application provide a data processing method, system, device and medium.
[0005] In a first aspect, in one embodiment of the present application, a data processing method is provided. Applied to a master node, the method comprises:
[0006] When the first temporary log file obtained based on the first memory data meets the snapshot generation condition, selecting a target slave node for executing the snapshot task from the plurality of slave nodes;
[0007] Sending a snapshot generation request for the first temporary log file to the target slave node;
[0008] When the target slave node executes the task corresponding to the snapshot generation request, if a modification request for the first memory data is received, modifying the first memory data;
[0009] Receive the snapshot generation feedback information of the first temporary log file fed back by the target slave node.
[0010] In a second aspect, in one embodiment of the present application, a data processing method is provided, which is applied to a slave node, and the method includes:
[0011] If a snapshot generation request provided by the master node is received, determining a first temporary log file corresponding to the snapshot generation request; wherein the first temporary log file is a log file generated based on the first memory data;
[0012] If the master node executes a modification request for the first memory data and generates a second temporary log file, then loads and executes a task corresponding to the snapshot generation request for the first temporary log file;
[0013] A first temporary snapshot file corresponding to the first temporary log file is generated.
[0014] In a third aspect, in one embodiment of the present application, a data processing system is provided, including:
[0015] A master node, configured to execute the method described in any one of the first aspects;
[0016] A slave node, configured to execute any one of the methods in the second aspect;
[0017] The shared file system is used to store the snapshot files persisted by the plurality of slave nodes.
[0018] In a fourth aspect, in one embodiment of the present application, an electronic device is provided, including a memory and a processor; wherein:
[0019] The memory is used to store programs;
[0020] The processor is coupled to the memory, and is used to execute the program stored in the memory to implement the method described in the first aspect, or to implement the method described in the second aspect.
[0021] In the fifth aspect, in one embodiment of the present application, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the data processing method described in the first aspect or the data processing method described in the second aspect.
[0022] The technical solution provided by the embodiment of the present application is that in a memory database system composed of a master node and multiple slave nodes, when it is necessary to perform a snapshot generation task for a first temporary log file obtained based on the first memory data, a node is selected from the memory database system as a target slave node, and the target slave node performs the snapshot generation task without the master node performing the snapshot generation task. While executing the task of the snapshot generation request for the first temporary log file, the master node receives the modification for the first memory data, and the master node can directly modify the first memory data without creating a copy of the first memory data. Because the first temporary log file corresponding to the first memory data has been loaded to the target slave node at this time, and the snapshot task is being executed by the target slave node, the target slave node will not load other newly obtained temporary log files. Therefore, directly modifying the first memory data in situ in the memory will not affect the snapshot task, and will not occupy too much memory to cause memory expansion. Moreover, since the master node does not need to perform the snapshot generation task, it can respond to the user's modification request for the first memory data in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0024] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;
[0025] Figure 2 A schematic diagram of an in-memory database system for illustrating an embodiment of the present application;
[0026] Figure 3 A flowchart of another data processing method provided in an embodiment of the present application;
[0027] Figure 4 A schematic diagram of a data processing system provided in an embodiment of the present application;
[0028] Figure 5 A schematic diagram of the interaction of the in-memory database system for illustrating an embodiment of the present application;
[0029] Figure 6a A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0030] Figure 6b A schematic diagram of the structure of another data processing device provided in an embodiment of the present application;
[0031] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0033] In some processes described in the specification, claims and the above-mentioned figures of the present application, multiple operations appearing in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. In addition, the embodiments described below are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.
[0034] In-memory database systems, there is generally a master node and multiple slave nodes. When the in-memory database is running, the data is stored in the running memory, which will lose the data if the power is off. Therefore, in order to ensure the security of the in-memory data, the in-memory data will be persisted in the local file system of each node under certain conditions. When the in-memory data is saved, incremental data modifications will be generated as the user's data write and modification requests, and temporary log files (binlog files) will be written in real time. When the temporary log files grow to a certain number, the in-memory database will generate a full data snapshot (snsapshot) to replace all current temporary log files, thereby effectively reducing the storage burden of persistent storage. When traditional in-memory databases generate data snapshots, most of them use a copy on write mechanism (Copy on write, COW) or a copy on write mechanism implemented by the user program itself (such as (Multi-Version Concurrent Control, MVCC) and other protocols). However, this type of copy-on-write mechanism has some defects: the former is based on a one-time jam that causes a snapshot file to be created. When the master node is persisting the snapshot file, if a user wants to modify the memory data, in order not to affect the normal execution of the snapshot task, a copy of the memory data will be created. When the amount of data to be modified is large, more copies will be created, which will cause a large memory expansion. Therefore, a solution is needed to execute the snapshot task without causing memory expansion.
[0035] Terminology explanation:
[0036] In-memory database: A database in which all data is stored in memory, generally consisting of a master node and multiple slave nodes.
[0037] Master node: the main database, accepting user read and write requests.
[0038] Slave node: also known as standby node, standby database, usually does not accept user requests, the data of the master node will be continuously synchronized to the slave node, under normal circumstances, the data in the master node and the slave node are consistent.
[0039] Temporary log file: Binlog is used to record the write operation (excluding query) information performed by the database, and is saved in the disk in binary form. Binlog is the logical log of MySQL and is recorded by the Server layer. MySQL databases using any storage engine will record binlog logs.
[0040] The technical solution implemented in this application will be explained below in conjunction with specific embodiments.
[0041] like Figure 1The flowchart of the data processing method provided by the embodiment of the present application is shown in FIG. The execution subject of the method may be a master node in a memory database system. Figure 1 You can see that the specific steps include the following:
[0042] 101: When a first temporary log file obtained based on the first memory data meets a snapshot generation condition, a target slave node for executing a snapshot task is selected from a plurality of slave nodes.
[0043] 102: Send a snapshot generation request for the first temporary log file to the target slave node.
[0044] 103: When the target slave node executes the task corresponding to the snapshot generation request, if a modification request for the first memory data is received, modify the first memory data.
[0045] 104: Receive snapshot generation feedback information of the first temporary log file fed back by the target slave node.
[0046] In the technical solution of the present application, the in-memory database system includes a master node and multiple slave nodes. The master node performs read and write tasks. When the master node can work normally, the slave node is a read-only node. When the master node cannot work normally, the slave node replaces the master node to perform data interaction with the user. At this time, the slave node has read and write capabilities.
[0047] The first temporary log file mentioned here is a binary file binlog of the incremental data in the memory (that is, the first memory data). Each time the memory data is modified, a corresponding temporary log file (binlog file) will be generated. It should be noted that the first temporary log file mentioned here is only used as an example. In actual applications, there may be multiple temporary log files corresponding to multiple memory data. For example, there are multiple first temporary log files that meet the snapshot generation conditions and need to generate snapshot files.
[0048] When selecting a slave node for executing a snapshot task, it can be selected randomly or according to a slave node list. It should be noted that only one slave node is selected to execute a snapshot task each time, so that the snapshot task execution requirements are met while avoiding multiple slave nodes from executing repeated snapshot tasks.
[0049] After determining the target slave node, a snapshot generation request is sent to the target slave node. After receiving the snapshot generation request, the target slave node executes the snapshot generation task based on the information carried in the snapshot generation request. When executing the snapshot generation task, the slave node loads the first temporary log file and does not load other temporary log files currently being generated (the latest generated or generated binlog files).
[0050] Generally speaking, it takes a certain amount of time (for example, several hundred milliseconds or several minutes) to execute a snapshot generation task, during which the memory database system continues to work, which means that a user may want to modify the first memory data during the execution of the snapshot generation task. Since the first temporary log file obtained before the first memory data is modified has been loaded into the target slave node, it means that the modification of the first memory data at this time will not affect the first temporary log file, nor will it affect the normal execution of the snapshot task. Therefore, in the technical solution of the present application, when executing a snapshot task, if there is a new memory data modification requirement, the master node can modify the first memory data directly in the memory without making a copy of the memory data, thereby effectively avoiding memory expansion when the memory data is modified.
[0051] In one or more embodiments of the present application, after receiving the snapshot generation feedback information of the first temporary log file fed back by the target slave node, it also includes: when the snapshot generation feedback information is successful, the master node renames the first temporary snapshot file to obtain a first persistent snapshot file; and deletes the first temporary log file.
[0052] The snapshot generation feedback information mentioned here refers to feedback information provided by the target slave node according to the snapshot generation result, and the feedback information may be success or failure.
[0053] When the snapshot generation feedback information is generation failure, that is, the target slave node has not successfully generated the snapshot file corresponding to the first temporary log file, indicating that the snapshot file is incomplete or an error occurs, the temporary snapshot file that failed to generate is deleted. If the reason for the snapshot generation failure is the failure of the target slave node, the target slave node needs to be replaced. If it is other failures, corresponding measures need to be taken to solve it.
[0054] When the snapshot generation feedback information is generated successfully, since the first temporary snapshot file generated by the target slave node is temporarily named, it needs to be renamed if it is to be actually stored. When renaming, it needs to be renamed according to the naming rules. For example, it can be renamed to SNAPSHOT-2 according to the sequential number of the persistent snapshot.
[0055] However, when storing snapshot files, there are multiple storage methods. One is that when the master node and the slave node share the same shared file system, the first temporary log file and the corresponding snapshot file can be stored in the shared file system, so that both the master node and the slave node can use the shared file system to store files, and there is no need to transfer files between the master node and the slave node.
[0056] Of course, it is also possible not to use a shared storage method, that is, the slave node and the master node each have their own storage space. That is, after the target slave node generates the first temporary snapshot file, it will be stored locally in the target slave node. Then, the target slave node sends the generated first temporary snapshot file to the master node, and the master node renames the first temporary snapshot and sends it to each slave node.
[0057] After completing the renaming of the first temporary log file and obtaining the first persistent snapshot file, the master node deletes the first temporary snapshot file, thereby releasing more storage space.
[0058] The above two situations will be respectively described below by means of specific embodiments.
[0059] In one or more embodiments of the present application, the master node and multiple slave nodes can share the same shared file system, that is, the first temporary log file and the generated snapshot file are stored in the shared file system, and each file in the shared file system is visible to each node (including the master node and multiple slave nodes). After the target slave node generates the first temporary snapshot file based on the snapshot generation request, the first temporary snapshot file stored in the shared file system is renamed to obtain a first persistent snapshot file; and the first temporary log file in the shared file system is deleted.
[0060] In actual applications, since the master node and multiple slave nodes share the same shared file system, it can be seen that the first temporary log file obtained based on the first memory data is itself stored in the shared file system. When the master node specifies the target slave node to perform a snapshot task, the master node does not need to send the first temporary log file to the target slave node through continuous data protection (CDP). After receiving the snapshot generation request, the target slave node determines the first temporary log file in the shared file system based on the information carried in the request. Since the master node and multiple slave nodes share the file system in the memory database system, the time occupied by data transmission can be effectively avoided, and the execution speed of the snapshot task can be effectively improved.
[0061] When the target slave node executes the snapshot generation task, the first temporary snapshot file is obtained. The snapshot is temporary, so the name of the first temporary snapshot file is also a temporary name, which does not comply with the master node's data management rules. Therefore, after the master node learns that the first temporary snapshot file is generated, if the first temporary snapshot file is accurate, it will be renamed and the renamed first persistent snapshot file will be stored in the shared file system. Since the first persistent snapshot file in the shared file system is obtained based on the first temporary log file at this time, in order to reduce the occupation of storage resources in the shared file system by temporary log files, it is necessary to delete the first temporary log file to release more storage space. It should be noted that the storage space occupied by the first temporary log file mentioned here is greater than the storage space occupied by the first persistent snapshot file.
[0062] Through the above method, the temporary log files and persistent snapshot files of the master node and multiple slave nodes are stored in the shared file system at the same time. There is no need to transfer temporary log files or persistent snapshot files between different nodes, which can effectively reduce or avoid the time occupied by data transmission, thereby effectively improving the efficiency of snapshot generation.
[0063] It should be noted here that although the master node and multiple slave nodes share the same shared file system to store temporary log files and persistent snapshot files, in the actual physical storage, the first persistent snapshot file in the shared file system is backed up and stored. In other words, at the logical level, the master node and multiple slave nodes share the same temporary log file or persistent snapshot file, but these temporary log files or persistent snapshot files are stored in different disks, which can ensure data security.
[0064] In one or more embodiments of the present application, it also includes: receiving a first temporary snapshot file generated by the target slave node based on the snapshot generation request, the first temporary snapshot file being a snapshot file generated by the target slave node based on the snapshot generation request; renaming the first temporary snapshot file to obtain a first persistent snapshot file; sending the first persistent snapshot file to the multiple slave nodes; and deleting the first temporary log file.
[0065] In actual applications, the master node and slave node in the memory database system can have their own local file system. Each node stores the temporary log files or persistent snapshot files that it is responsible for in its own local file system. If you want to share the temporary log files or persistent snapshot files that you are responsible for with other nodes, you need to transfer the temporary log files or persistent snapshot files through the continuous backup method (CDP). After the transmission through the continuous backup method, each node stores the received temporary log files or persistent snapshot files in the local file system.
[0066] When the target slave node is designated to be responsible for generating the first temporary snapshot file, the master node sends the first temporary log file to the target slave node through continuous backup, and then the target slave node executes the snapshot generation task. In the actual snapshot generation process, it often takes a certain amount of time, during which time the user may want to modify the first memory data. Since the first temporary log file has been sent to the target slave node, even if the first memory data is modified directly in the memory during the snapshot generation, it will not affect the target slave node's execution of the snapshot generation task. Therefore, there is no need to create a copy of the first memory data in the memory, effectively avoiding the problem of memory expansion.
[0067] After the target slave node generates the first temporary snapshot file, it is also necessary to send the first temporary snapshot file to the master node through a time series backup method. Then, after the master node confirms that it is correct, the first temporary snapshot file is renamed as the first persistent snapshot file. The first persistent snapshot file is sent to multiple slave nodes, and stored in the local file system by multiple slave nodes. Since the persistent snapshot file corresponding to the first temporary log file has been obtained, in order to save storage space, the master node can delete the first temporary log file. It should be noted that when deleting the first temporary log file, you can choose a time when the master node is relatively idle (that is, when no data processing request from the user is received), so as to avoid affecting the untimely response to user requests.
[0068] For example, if Figure 2 A schematic diagram of an in-memory database system is provided as an example for illustrating an embodiment of the present application. Figure 2 In the figure, we can see that there is one master node and three slave nodes. The master node and each slave node have their own local file system, which is used to store temporary log files or persistent snapshot files. Since the local file system of each node is managed independently, each node cannot read the data in the local file system of other nodes.
[0069] from Figure 2 As can be seen in the figure, the master node has read-write capabilities, while the slave node has read-only capabilities. Further assume that the first slave node is selected as the target slave node. After the first slave node receives the first temporary log file from the master node as the target slave node, it stores it in the local file system and executes the snapshot generation task. Thereafter, the first slave node executes the snapshot task to obtain the first temporary snapshot file, and sends the first temporary snapshot file to the master node. The master node renames the first temporary snapshot file to the first persistent snapshot file, sends the first persistent snapshot file to the slave node, and deletes the first temporary log file in the local file system of the master node.
[0070] In one or more embodiments of the present application, when the first temporary log file is stored in a shared file system, when the first temporary log file obtained based on the first memory data meets the snapshot generation condition, a target slave node for executing the snapshot task is selected from multiple slave nodes, including: when the storage space occupied by the first temporary log file is greater than a preset threshold, or when the remaining storage space of the shared file system is not greater than the preset space, it is determined that the first temporary log file meets the snapshot generation condition; and a target slave node for executing the snapshot task is selected from multiple slave nodes.
[0071] In actual applications, since only the master node has read and write capabilities, when the storage space occupied by the first temporary log file in the local file system of the master node is greater than the preset threshold, or when the storage space occupied by the first temporary log file in the shared file system is greater than the preset threshold, it means that the first temporary log file meets the snapshot generation conditions, and a snapshot needs to be generated based on the temporary log file to save storage space.
[0072] Alternatively, the storage space of the shared file system is compared with the preset space of the memory. There are many ways to compare, such as comparing the remaining space, such as comparing the remaining storage space of the shared file system with the occupied space in the memory, etc. This is just an example, and other solutions can be selected to determine whether a snapshot needs to be generated.
[0073] In the above manner, it is determined whether the first temporary log file meets the snapshot generation condition. In other words, if the master node and the slave node have their own independent local file systems, the master node determines whether there is a need to generate a snapshot, thereby timely reducing the storage burden of the master node. If the master node and the slave node have a common shared file system, the master node or the slave node can determine whether there is a need to generate a snapshot.
[0074] When selecting the target slave node, you can use a random selection method or a sequential selection method (for example, the first slave node is selected as the target node when the snapshot task is executed for the first time, the second slave node is selected as the target node when the snapshot task is executed for the second time, and so on). If the current nodes are all busy, an external coordination node can help complete the snapshot generation task.
[0075] Through the above solution, snapshot processing needs can be discovered in a timely manner to avoid occupying too many storage resources and affecting the normal operation of the in-memory database system.
[0076] In one or more embodiments of the present application, sending a snapshot generation request to the target slave node includes: sending a snapshot generation request carrying a timestamp to the target slave node, so that the target slave node determines the first temporary log file based on the timestamp.
[0077] In actual applications, when the master node creates a snapshot generation request, the task name, that is, the task ID, and the temporary snapshot file name NEW_SNAPSHOT are written in the request. When the target slave node generates a snapshot, the temporary snapshot file name NEW_SNAPSHOT is used as the name of the first temporary snapshot file.
[0078] In addition, the snapshot generation request also includes a timestamp, which can be used to mark the generation time of the current request and can also help the target slave node accurately find the temporary log file that needs to generate a snapshot. Specifically, after the target slave node receives the snapshot generation request, it parses the timestamp and checks the generation time of each temporary log file in the shared file system according to the timestamp. If the generation time is earlier than the timestamp, it means that these temporary log files do not need to be updated and snapshots can be generated; conversely, if the generation time is later than the timestamp, it means that these temporary log files may still be updated and snapshots cannot be generated.
[0079] For example, if the timestamp is 15:12 on March 23, the target slave node or master node will search for the first temporary log file generated before this timestamp based on this timestamp, while other temporary log files generated after this timestamp are not within the processing scope of this snapshot generation task. In other words, the modification of the data in the memory after the timestamp will not affect the execution of the snapshot generation task by the slave node.
[0080] In one or more embodiments of the present application, after sending a snapshot generation request to the target slave node, it also includes: based on a check cycle, sending a status check request to the target slave node to check the status of the target slave node; if the check fails, replacing the target slave node and deleting the temporary snapshot file generated by the target slave node.
[0081] As mentioned above, when executing the snapshot generation task, the target slave node may take a certain amount of time. During this period, it is necessary to ensure that the target slave node is executing the snapshot generation task smoothly and normally. Therefore, the master node will interact with the target slave node at a certain period to inquire about the progress or check the working status. If the working status of the target node is normal or the working progress is normal, the master node waits for the target slave node to complete the snapshot generation task.
[0082] If the check result is that the target slave node is faulty, or the target slave node has suspended the snapshot generation task, it means that the target slave node will not be able to complete the snapshot generation task, and it is necessary to reselect other slave nodes to replace the target slave node to perform the snapshot generation task. If some temporary snapshot files have been generated (that is, temporary snapshot files that are being generated and have not completed the generation task) when the target slave node fails, but the final temporary snapshot file has not been generated, it is necessary to delete the temporary snapshot file so that other slave nodes can be selected to re-execute the snapshot generation task to avoid the generation of half of the temporary snapshot files affecting the re-execution of the snapshot generation task by other slave nodes.
[0083] For example, the master node checks whether the target slave node A is executing the snapshot generation task normally according to the check cycle. If the target node A fails and the generated temporary snapshot file B is also suspended, the temporary snapshot file B is deleted. The master node reselects another slave node C as the target slave node C and re-executes the snapshot generation task to obtain the temporary snapshot file B.
[0084] In addition, to ensure the security of the first persistent snapshot file, the first persistent snapshot file can be backed up and stored using a shared file system. For example, the backup can be performed in different storage spaces in the shared file system, or the backup can be stored in the local file system of the master node or the slave node.
[0085] In one or more embodiments of the present application, it also includes: generating a second temporary log file corresponding to the modified first memory data; when the second temporary log file meets the persistence condition, sending a snapshot generation request to the target slave node.
[0086] After the user completes the modification of the first memory data, a second temporary log file will be generated, which records the modification information of the first memory data. When it is determined that the second temporary log file meets the persistence condition, a new snapshot corresponding to the second temporary log file will be generated. And the newly generated snapshot is stored in the shared file system. The new snapshot also needs to be renamed by the master node to distinguish it from the first persistent snapshot file. When other slave nodes continue to execute later, the persistent snapshot file in the shared file system can be shared with other slave nodes. In the above manner, multiple slave nodes share the same shared file system with the master node, saving the time spent on data transmission between different nodes.
[0087] Based on the same idea, the embodiment of the present application also provides another data processing method, which is applied to the slave node. Figure 3 A flow chart of another data processing method provided in an embodiment of the present application. Figure 3 You can see that the specific steps include the following:
[0088] 301: If a snapshot generation request provided by a master node is received, determine a first temporary log file corresponding to the snapshot generation request; wherein the first temporary log file is a log file generated based on first memory data.
[0089] 302: If the master node executes a modification request for the first memory data and generates a second temporary log file, then load and execute a task corresponding to a snapshot generation request for the first temporary log file.
[0090] 303: Generate a first temporary snapshot file corresponding to the first temporary log file.
[0091] If the slave node receives the snapshot generation request sent by the master node, it means that the slave node is selected by the master node as a target slave node for executing the task of generating a snapshot for the first temporary log file. After receiving the snapshot generation request, the target slave node further parses the snapshot generation request to obtain the timestamp contained therein, and determines the first temporary log file generated before the timestamp.
[0092] As mentioned above, when the slave node executes the snapshot generation task, in order not to affect the normal operation of the master node and to prevent the master node data modification from affecting the snapshot task, the slave node only executes the snapshot generation task for the temporary log file obtained before the timestamp. During the snapshot generation task execution, the master node can modify any memory data directly in memory. Since the slave node is responsible for executing the snapshot generation task, the master node is responsible for receiving the user's data modification request and directly modifying the data in memory, there is no need to create a copy of the memory data, and there will be no memory expansion problem.
[0093] In one or more embodiments of the present application, after generating the first temporary snapshot file corresponding to the first temporary log file, it also includes: storing the first temporary snapshot file in a shared file system so that the master node renames the first temporary snapshot file to obtain a first persistent snapshot file, and deletes the first temporary log file in the shared file system.
[0094] As an optional method, the master node and multiple slave nodes can share the same shared file system, and use the shared file system to store temporary log files and temporary snapshot files. The files in the shared file system can be read by the master node and multiple slave nodes. Therefore, when the target slave node executes the snapshot generation task, the required file can be found from the shared file system according to the timestamp and the file name of the specified temporary log file, and the master node does not need to send the temporary log file to the slave node. Similarly, after the slave node generates the first temporary snapshot file, the first temporary snapshot file is stored in the shared file system, and does not need to be sent to the master node separately, which can effectively avoid the time spent on data transmission.
[0095] Based on the same idea, the embodiment of the present application also provides a data processing system. Figure 4 A schematic diagram of a data processing system provided in an embodiment of the present application. Figure 4 As can be seen in the figure, the system is a distributed system, which includes:
[0096] The master node 41 is configured to select a target slave node for executing a snapshot task from a plurality of slave nodes when a first temporary log file obtained based on the first memory data meets a snapshot generation condition;
[0097] Sending a snapshot generation request for the first temporary log file to the target slave node;
[0098] When the target slave node executes the task corresponding to the snapshot generation request, if a modification request for the first memory data is received, modifying the first memory data;
[0099] A second temporary log file corresponding to the modified first memory data is generated.
[0100] The master node 41 is configured to rename the first temporary snapshot file to obtain a first persistent snapshot file when the snapshot generation feedback information indicates that the snapshot generation is successful;
[0101] Delete the first temporary log file.
[0102] The master node 41 is configured to rename a first temporary snapshot file stored in the shared file system to obtain a first persistent snapshot file if the master node and the slave node share the same shared file system, wherein the first temporary snapshot file is a snapshot file generated by the target slave node based on the snapshot generation request;
[0103] Delete the first temporary log file in the shared file system.
[0104] The master node 41 is configured to receive a request from the target slave node to generate a first temporary snapshot file based on the snapshot generation request, where the first temporary snapshot file is a snapshot file generated by the target slave node based on the snapshot generation request;
[0105] Rename the first temporary snapshot file to obtain a first persistent snapshot file;
[0106] Sending the first persistent snapshot file to the multiple slave nodes;
[0107] Delete the first temporary log file.
[0108] The master node 41 is used for determining that the first temporary log file meets the snapshot generation condition when the storage space occupied by the first temporary log file is greater than a preset threshold, or when the remaining storage space of the shared file system is not greater than a preset space, when the first temporary log file is stored in the shared file system;
[0109] A target slave node for executing the snapshot task is selected from multiple slave nodes.
[0110] The master node 41 is configured to send a snapshot generation request carrying a timestamp to the target slave node, so that the target slave node determines the first temporary log file based on the timestamp.
[0111] The master node 41 is used to send a status check request to the target slave node based on a check period to check the status of the target slave node;
[0112] If the check fails, the target slave node is replaced, and the temporary snapshot file generated by the target slave node is deleted.
[0113] The master node 41 is used for backing up and storing the first persistent snapshot file in the shared file system.
[0114] The master node 41 is used to generate a second temporary log file corresponding to the modified first memory data; when the second temporary log file meets the persistence condition, send a snapshot generation request to the target slave node.
[0115] The slave node 42 is configured to determine a first temporary log file corresponding to a snapshot generation request provided by the master node if the snapshot generation request is received; wherein the first temporary log file is a log file generated based on the first memory data;
[0116] If the master node executes a modification request for the first memory data and generates a second temporary log file, then a snapshot generation task for the first temporary log file is loaded and executed;
[0117] A first temporary snapshot file corresponding to the first temporary log file is generated.
[0118] The slave node 42 is configured to determine, according to the timestamp included in the snapshot generation request, the first temporary log file generated before the timestamp.
[0119] The slave node 42 is used to store the first temporary snapshot file in a shared file system, so that the master node renames the first temporary snapshot file to obtain a first persistent snapshot file, and deletes the first temporary log file in the shared file system.
[0120] The shared file system 43 is used to store the snapshots persisted by the plurality of slave nodes.
[0121] For ease of understanding, the following will be described by way of specific examples. Figure 5 The following is a schematic diagram of the interaction of the in-memory database system for illustrating an embodiment of the present application. In this system, the master node and the slave node share a shared file system to store temporary log files and snapshot files. Specifically,
[0122] Assume that the original snapshot file in the shared file system is SNAPSHOT-1, and the temporary log file binlog file list at this time is binlog.1, binlog.2, binlog.3, a total of 3 files.
[0123] Further assume that the storage space occupied by the temporary log file exceeds the threshold and a snapshot file needs to be generated for storage. Therefore, the master node sends a snapshot generation request to the selected target slave node. The request may be, for example, M(ID:1,NEW_SNAPSHOT:SNAPSHOT.{timestamp}.tmp). The {timestamp} in the request refers to the timestamp when the task is issued, ID is the task name, and NEW_SNAPSHOT is the temporary name of the new temporary snapshot file.
[0124] When the target slave node has finished loading the binlog.3 file, the master node just receives a data modification request from the client and waits for the master node to create the binlog.4 file (but has not yet loaded the content of this file). The target slave node starts to persist a new snapshot file as SNAPSHOT.{timestamp}.tmp, and after completion, notifies the master node M (ID: 1, NEW_SNAPSHOT: SNAPSHOT-2.tmp, LAST_BINLOG_NAME: binlog.3). Among them, LAST_BINLOG_NAME indicates the name of the last binlog file corresponding to the new snapshot file generated by the target slave node. The snapshot file corresponding to LAST_BINLOG_NAME is stored in the shared file system.
[0125] After receiving the notification that the snapshot generation task is completed, the master node renames SNAPSHOT.{timestamp}.tmp to the first persistent snapshot file SNAPSHOT-2, and then deletes binlog.3 and earlier binlog files according to the information of LAST_BINLOG_NAME. The renamed persistent snapshot file is stored in the shared file system.
[0126] Based on the same idea, the present embodiment also provides a data processing device. Figure 6a A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The device is applied to a master node, from Figure 6a As can be seen in the figure, the device comprises:
[0127] The selection module 601 is used to select a target slave node for executing a snapshot task from a plurality of slave nodes when a first temporary log file obtained based on the first memory data meets a snapshot generation condition.
[0128] The sending module 602 is used to send a snapshot generation request for the first temporary log file to the target slave node.
[0129] A modification module 603 is configured to modify the first memory data if a modification request for the first memory data is received when the target slave node executes the task corresponding to the snapshot generation request;
[0130] The generating module 604 is used to generate a second temporary log file corresponding to the modified first memory data.
[0131] Optionally, a renaming module 605 is further included, which is used for, when the snapshot generation feedback information indicates that the generation is successful, the master node renames the first temporary snapshot file to obtain a first persistent snapshot file;
[0132] Delete the first temporary log file.
[0133] Optionally, a renaming module 605 is configured to rename a first temporary snapshot file stored in the shared file system to obtain a first persistent snapshot file if the master node and the slave node share the same shared file system, wherein the first temporary snapshot file is a snapshot file generated by the target slave node based on the snapshot generation request;
[0134] Delete the first temporary log file in the shared file system.
[0135] A renaming module 605 is configured to receive a first temporary snapshot file generated by the target slave node based on the snapshot generation request, where the first temporary snapshot file is a snapshot file generated by the target slave node based on the snapshot generation request;
[0136] Rename the first temporary snapshot file to obtain a first persistent snapshot file;
[0137] Sending the first persistent snapshot file to the multiple slave nodes;
[0138] Delete the first temporary log file.
[0139] A selection module 601 is used to determine that the first temporary log file meets the snapshot generation condition when the storage space occupied by the first temporary log file is greater than a preset threshold, or when the remaining storage space of the shared file system is not greater than a preset space, in the case where the first temporary log file is stored in a shared file system;
[0140] A target slave node for executing the snapshot task is selected from multiple slave nodes.
[0141] The sending module 602 is used to send a snapshot generation request carrying a temporary snapshot file name and a timestamp to the target slave node, so that the target slave node determines the first temporary log file based on the timestamp.
[0142] Optionally, a checking module 606 is further included, which is used to send a status checking request to the target slave node based on a checking period to check the status of the target slave node;
[0143] If the check fails, the target slave node is replaced, and the first temporary snapshot file generated by the target slave node is deleted.
[0144] Optionally, a backup module 607 is further included, which is used to back up and store the first persistent snapshot file in the shared file system.
[0145] A sending module 602 is used to generate a second temporary log file corresponding to the modified first memory data;
[0146] When the second temporary log file meets the persistence condition, a snapshot generation request is sent to the target slave node.
[0147] Based on the same idea, the present embodiment also provides another data processing device. Figure 6b A schematic diagram of the structure of another data processing device provided in an embodiment of the present application. The device is applied to a slave node. Figure 6b As can be seen in the figure, the device comprises:
[0148] The determination module 611 is used to determine a first temporary log file corresponding to a snapshot generation request provided by a master node if the snapshot generation request is received; wherein the first temporary log file is a log file generated based on the first memory data.
[0149] The loading module 612 is used to load and execute the task corresponding to the snapshot generation request for the first temporary log file if the master node executes the modification request for the first memory data and generates a second temporary log file.
[0150] The generating module 613 is configured to generate a first temporary snapshot file corresponding to the first temporary log file.
[0151] The determination module 611 is configured to determine, according to the timestamp included in the snapshot generation request, the first temporary log file generated before the timestamp.
[0152] Optionally, a deletion module 614 is further included, which is used to store the first temporary snapshot file in a shared file system so that the master node renames the first temporary snapshot file to obtain a first persistent snapshot file and deletes the first temporary log file in the shared file system.
[0153] An embodiment of the present application also provides an electronic device. The electronic device is a master node electronic device in a computing unit. Figure 7 The electronic device includes a memory 701, a processor 702 and a communication component 703;
[0154] The memory 701 is used to store programs;
[0155] The processor 702 is coupled to the memory and is configured to execute the program stored in the memory to:
[0156] When the first temporary log file obtained based on the first memory data meets the snapshot generation condition, selecting a target slave node for executing the snapshot task from the plurality of slave nodes;
[0157] Sending a snapshot generation request for the first temporary log file to the target slave node;
[0158] When the target slave node executes the task corresponding to the snapshot generation request, if a modification request for the first memory data is received, modifying the first memory data;
[0159] Receive the snapshot generation feedback information of the first temporary log file fed back by the target slave node.
[0160] Processor 702, configured to, when the snapshot generation feedback information indicates that the generation is successful, rename the first temporary snapshot file by the master node to obtain a first persistent snapshot file;
[0161] Delete the first temporary log file.
[0162] Processor 702, configured to rename a first temporary snapshot file stored in the shared file system to obtain a first persistent snapshot file if the master node and the slave node share the same shared file system, where the first temporary snapshot file is a snapshot file generated by the target slave node based on the snapshot generation request;
[0163] Delete the first temporary log file in the shared file system.
[0164] Processor 702, configured to receive a first temporary snapshot file generated by the target slave node based on the snapshot generation request, where the first temporary snapshot file is a snapshot file generated by the target slave node based on the snapshot generation request;
[0165] Rename the first temporary snapshot file to obtain a first persistent snapshot file;
[0166] Sending the first persistent snapshot file to the multiple slave nodes;
[0167] Delete the first temporary log file.
[0168] Processor 702, configured to determine that the first temporary log file meets a snapshot generation condition when the storage space occupied by the first temporary log file is greater than a preset threshold, or when the remaining storage space of the shared file system is not greater than a preset space, when the first temporary log file is stored in a shared file system;
[0169] A target slave node for executing the snapshot task is selected from multiple slave nodes.
[0170] The processor 702 is configured to send a snapshot generation request carrying a timestamp to the target slave node, so that the target slave node determines the first temporary log file based on the timestamp.
[0171] The processor 702 is configured to send a status check request to the target slave node based on a check period to check the status of the target slave node;
[0172] If the check fails, the target slave node is replaced, and the temporary snapshot file generated by the target slave node is deleted.
[0173] Processor 702 is configured to back up and store the first persistent snapshot file in the shared file system.
[0174] The processor 702 is configured to send a snapshot generation request to the target slave node when the second temporary log file meets the persistence condition.
[0175] When applied to a slave node, the processor 702 is configured to, if receiving a snapshot generation request provided by a master node, determine a first temporary log file corresponding to the snapshot generation request; wherein the first temporary log file is a log file generated based on the first memory data;
[0176] If the master node executes a modification request for the first memory data and generates a second temporary log file, then loads and executes a task corresponding to the snapshot generation request for the first temporary log file;
[0177] A first temporary snapshot file corresponding to the first temporary log file is generated.
[0178] The processor 702 is configured to determine, according to the timestamp included in the snapshot generation request, the first temporary log file generated before the timestamp.
[0179] The processor 702 is configured to store the first temporary snapshot file in a shared file system so that the master node renames the first temporary snapshot file to obtain a first persistent snapshot file, and delete the first temporary log file in the shared file system.
[0180] The memory 701 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0181] Furthermore, the processor 702 in this embodiment may specifically be: a programmable switching processing chip, in which a data replication engine is configured to replicate received data.
[0182] When the processor 702 executes the program in the memory, in addition to the above functions, it can also realize other functions, which can be specifically referred to the description of the above embodiments. Figure 7 As shown, the electronic device also includes: a power supply component 704 and other components.
[0183] The present application also provides a non-transitory machine-readable storage medium, wherein the non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes Figure 1 The method described in the corresponding embodiment.
[0184] Based on the above embodiment, in a memory database system composed of a master node and multiple slave nodes, when it is necessary to perform a snapshot generation task for a first temporary log file obtained based on the first memory data, a node is selected from the memory database system as a target slave node, and the target slave node performs the snapshot generation task, without the need for the master node to perform the snapshot generation task. While executing the snapshot generation task for the first temporary log file, the master node receives a modification to the first memory data, and the master node can directly modify the first memory data without creating a copy of the first memory data. Because the first temporary log file corresponding to the first memory data has been loaded into the target slave node at this time, and the snapshot task is being executed by the target slave node, the target slave node will no longer load other newly obtained temporary log files. Therefore, directly modifying the first memory data in situ in the memory will not affect the snapshot task, and will not occupy too much memory to cause memory expansion. Moreover, since the master node does not need to perform the snapshot generation task, it can respond to the user's modification request for the first memory data in a timely manner.
[0185] In addition, if a shared file system is used in the memory database system to provide file storage services for the master node and multiple slave nodes at the same time, there is no need to transfer files between the master node and the slave nodes during the execution of the snapshot generation task, which can effectively improve the work efficiency of the snapshot generation task.
[0186] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A data processing method, characterized in that: Applied to the master node, the method comprises: When the first temporary log file obtained based on the first memory data meets the snapshot generation condition, selecting a target slave node for executing the snapshot task from the plurality of slave nodes; Sending a snapshot generation request for the first temporary log file to the target slave node; When the target slave node executes the task corresponding to the snapshot generation request, if a modification request for the first memory data is received, modifying the first memory data; Receive the snapshot generation feedback information of the first temporary log file fed back by the target slave node.
2. The method according to claim 1, characterized in that After receiving the snapshot generation feedback information of the first temporary log file fed back by the target slave node, the method further includes: When the snapshot generation feedback information indicates that the generation is successful, the master node renames the first temporary snapshot file to obtain a first persistent snapshot file; Delete the first temporary log file.
3. The method according to claim 2, characterized in that The master node renaming the first temporary snapshot file to obtain a first persistent snapshot file, and deleting the first temporary log file, includes: If the master node and the slave node share the same shared file system, rename the first temporary snapshot file stored in the shared file system to obtain a first persistent snapshot file, where the first temporary snapshot file is a snapshot file generated by the target slave node based on the snapshot generation request; Delete the first temporary log file in the shared file system.
4. The method according to claim 2, characterized in that: The master node renaming the first temporary snapshot file to obtain a first persistent snapshot file, and deleting the first temporary log file, includes: receiving a first temporary snapshot file generated by the target slave node based on the snapshot generation request, wherein the first temporary snapshot file is a snapshot file generated by the target slave node based on the snapshot generation request; Rename the first temporary snapshot file to obtain a first persistent snapshot file; Sending the first persistent snapshot file to the multiple slave nodes; Delete the first temporary log file.
5. The method according to claim 3, characterized in that: When the first temporary log file is stored in a shared file system, when the first temporary log file obtained based on the first memory data meets the snapshot generation condition, selecting a target slave node for executing the snapshot task from a plurality of slave nodes includes: When the storage space occupied by the first temporary log file is greater than a preset threshold, or when the remaining storage space of the shared file system is not greater than a preset space, determining that the first temporary log file meets the snapshot generation condition; A target slave node for executing a snapshot task is selected from multiple slave nodes.
6. The method according to any one of claims 1 to 5, characterized in that The sending a snapshot generation request to the target slave node includes: A snapshot generation request carrying a timestamp is sent to the target slave node, so that the target slave node determines the first temporary log file based on the timestamp.
7. The method according to any one of claims 1 to 5, characterized in that After sending the snapshot generation request to the target slave node, the method further includes: Based on the check period, sending a status check request to the target slave node to check the status of the target slave node; If the check fails, the target slave node is replaced, and the temporary snapshot file generated by the target slave node is deleted.
8. The method according to claim 1, characterized in that Also includes: Generate a second temporary log file corresponding to the modified first memory data; When the second temporary log file meets the persistence condition, a snapshot generation request is sent to the target slave node.
9. A data processing method, characterized in that: Applied to a slave node, the method comprises: If a snapshot generation request provided by the master node is received, determining a first temporary log file corresponding to the snapshot generation request; wherein the first temporary log file is a log file generated based on the first memory data; If the master node executes a modification request for the first memory data and generates a second temporary log file, then loads and executes a task corresponding to the snapshot generation request for the first temporary log file; A first temporary snapshot file corresponding to the first temporary log file is generated.
10. The method according to claim 9, characterized in that If a snapshot generation request provided by the master node is received, determining a first temporary log file corresponding to the snapshot generation request includes: The first temporary log file generated before the timestamp is determined according to the timestamp included in the snapshot generation request.
11. The method according to claim 9, characterized in that After generating the first temporary snapshot file corresponding to the first temporary log file, the method further includes: The first temporary snapshot file is stored in a shared file system, so that the master node renames the first temporary snapshot file to obtain a first persistent snapshot file, and deletes the first temporary log file in the shared file system.
12. A database system, characterized in that: The system comprises: A master node, configured to execute the method according to any one of claims 1 to 8; A slave node, configured to execute the method according to any one of claims 9 to 11; The shared file system is used to store the snapshot files persisted by the plurality of slave nodes.
13. An electronic device comprising a memory and a processor; wherein: The memory is used to store programs; The processor is coupled to the memory, and is configured to execute the program stored in the memory, so as to implement the method according to any one of claims 1 to 11.
14. A non-transitory machine-readable storage medium having executable code stored thereon, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 11.