Memory fragmentation management method, device and system

Abstract

The present invention discloses a memory fragmentation management method, device, and system, relating to the field of computer technology. A specific implementation of the method includes: upon receiving a management request issued by any node in a distributed system when memory fragmentation management conditions are met, determining whether a leader replica exists in the node; if a leader replica exists in the node, determining the follower replicas corresponding to each leader replica in the node on other nodes, and sending an election instruction to the follower replica; the election instruction instructs the follower replica and the leader replica to perform a leader election, and restricts the candidates for the leader election to be among the follower replicas; after obtaining the election result of the leader election, the node is restarted. This implementation can effectively manage memory fragmentation in distributed system nodes without affecting business.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a memory fragmentation management method, device and system. Background Art

[0002] In a distributed system, each node may experience memory fragmentation due to differences in the length of stored data, frequent data updates, and other reasons. Memory fragmentation refers to the waste of memory space caused by the operating system allocating memory to a process but the process being unable to use it due to reasons such as limited continuous storage space. Excessive memory fragmentation will occupy memory space, reduce process execution efficiency, and affect the performance of the corresponding application. In the prior art, there are two main methods to solve the problem of memory fragmentation: first, when a node is found to have excessive memory fragmentation, the node is directly restarted. Second, when a node has excessive memory fragmentation, a thread is started in the background to move memory data, trying to make the used memory continuous and thus reduce memory fragmentation.

[0003] In the process of implementing the present invention, the inventors found that the existing technology has at least the following problems: the first method can indeed solve the memory fragmentation problem, but if the restarted node stores a leader copy (leader), the entire consistency group including the leader copy and the corresponding follower copy (follower) will be unavailable. Re-electing the leader requires at least three special election cycles (special election cycles refer to election cycles applied in fault repair scenarios). If each special election cycle is 500 milliseconds, it will cause related services to be unavailable for more than 1.5 seconds, thereby causing greater jitter to the business. The second method requires a large number of locking operations, which will have a more serious impact on the business and cannot be used in actual production systems. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a memory fragmentation management method, device, and system, which can effectively manage memory fragmentation in distributed system nodes without affecting business.

[0005] To achieve the above object, according to one aspect of the present invention, a method for managing memory fragmentation is provided.

[0006] The memory fragmentation management method of an embodiment of the present invention includes: after receiving a management request issued by any node in a distributed system when the memory fragmentation management conditions are met, determining whether there is a leader copy in the node; wherein each node in the distributed system stores a copy of at least one shard of the target data, and the copy includes a leader copy and a follower copy; when there is a leader copy in the node, determining that each leader copy in the node corresponds to a follower copy in other nodes, and sending an election instruction to the follower copy; the election instruction instructs the follower copy and the leader copy to perform a leader election, and limits the candidates for the leader election to the follower copy; after obtaining the election result of the leader election, the node is restarted.

[0007] Optionally, the method further includes: receiving a heartbeat signal emitted by each node in the distributed system, the heartbeat signal carrying the identifier of the leader copy in the node and the identifier of the follower copy corresponding to the leader copy and the node identifier; judging whether there is a leader copy in the node includes: judging whether there is a leader copy in the node based on the heartbeat signal emitted by the node.

[0008] Optionally, after obtaining the election result of the leader election, the node is restarted, including: in response to the election result indicating that there is currently no leader copy on the node, sending a restart instruction to the node; and the node performing a restart after receiving the restart instruction.

[0009] Optionally, the method further includes: when it is determined that there is no leader copy in the node that issues the governance request, sending a restart instruction to the node; and the node performing a restart after receiving the restart instruction.

[0010] Optionally, the memory fragmentation management condition includes: a memory fragmentation ratio greater than a threshold.

[0011] Optionally, each copy of a shard of the target data is stored in a different node of the distributed system.

[0012] Optionally, the distributed system is a memory-based distributed storage system; the leader replica is used to interact with an application to implement data reading and / or writing, and the follower replica maintains data consistency with the corresponding leader replica.

[0013] To achieve the above object, according to another aspect of the present invention, a memory fragmentation management device is provided.

[0014] The memory fragmentation management device of an embodiment of the present invention may include: a leader judgment unit, which is used to: after receiving a management request issued by any node in a distributed system when the memory fragmentation management conditions are met, determine whether there is a leader copy in the node; wherein each node in the distributed system stores a copy of at least one shard of the target data, and the copy includes a leader copy and a follower copy; an election instruction unit, which is used to: when a leader copy exists in the node, determine that each leader copy in the node corresponds to a follower copy in other nodes, and send an election instruction to the follower copy; the election instruction instructs the follower copy and the leader copy to perform a leader election, and limits the candidates for the leader election to the follower copy; an execution unit, which is used to obtain the election result of the leader election; and after obtaining the election result, the node is restarted.

[0015] To achieve the above objectives, according to another aspect of the present invention, a memory fragmentation management system is provided.

[0016] The memory fragmentation management system of an embodiment of the present invention may include multiple nodes and a scheduling module, each node stores a copy of at least one shard of the target data, and the copies include a leader copy and a follower copy; wherein the scheduling module receives a management request issued by any node when the memory fragmentation management conditions are met, and determines whether there is a leader copy in the node; when it is determined that there is a leader copy in the node, the scheduling module determines that each leader copy in the node corresponds to a follower copy in other nodes, and sends an election instruction to the follower copy; the election instruction instructs the follower copy and the leader copy to perform a leader election, and limits the candidates for the leader election to the follower copy; the scheduling module obtains the election result of the leader election; and, after obtaining the election result, the node is restarted.

[0017] Optionally, the scheduling module receives a heartbeat signal sent by each node in the system, and determines whether there is a leader copy in the node that issues the governance request based on the heartbeat signal; wherein, the heartbeat signal sent by any node carries the identifier of the leader copy in the node and the identifier of the follower copy corresponding to the leader copy and the node identifier.

[0018] Optionally, the memory fragmentation management conditions include: the memory fragmentation ratio is greater than a threshold; each copy of a shard of the target data is stored on different nodes of the system; the leader copy is used to interact with the application to realize data reading and / or writing, and the follower copy maintains data consistency with the corresponding leader copy.

[0019] To achieve the above objective, according to another aspect of the present invention, an electronic device is provided.

[0020] An electronic device of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the memory fragmentation management method provided by the present invention.

[0021] To achieve the above objective, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0022] A computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the memory fragmentation management method provided by the present invention.

[0023] According to the technical solution of the present invention, the embodiments of the above invention have the following advantages or beneficial effects: the scheduling module in the distributed system uses the heartbeat signals sent by each node to keep track of the current replica status of each node in real time. When a node meets the memory fragmentation management conditions and needs to be restarted, the node sends a management request to the scheduling module. The scheduling module first determines whether there is a leader replica in the node. If not, it can directly instruct the node to restart (for example, displaying information that there is no leader replica in the node, so that the staff can restart the node based on this information); if so, it sends an election instruction to the follower replicas corresponding to each leader replica in the node on other nodes, so that the leader replica and the follower replica elect a new leader replica from the follower replicas. Afterwards, when it is determined that there is no leader replica in the node to be restarted, the node can be instructed to restart. Through the above settings, nodes that meet the memory fragmentation management conditions can perform a normal election process (this election process can take only a few milliseconds) before restarting, so that the leader replica in them is scheduled to other nodes. After that, restarting the node will not affect the business and user experience, thereby achieving effective memory fragmentation management without affecting the business.

[0024] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0026] Figure 1 This is a schematic diagram of the main steps of the memory fragmentation management method according to an embodiment of the present invention;

[0027] Figure 2 Schematic diagram of the composition of a distributed system according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of data reading and writing of a replica in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of specific execution steps of the memory fragmentation management method according to an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the main parts of the memory fragmentation management device according to an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the structure of an electronic device used to implement the memory fragmentation management method in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] It should be pointed out that, in the absence of conflict, the embodiments of the present invention and the technical features therein may be combined with each other.

[0034] Figure 1 2 is a schematic diagram of the main steps of the memory fragmentation management method according to an embodiment of the present invention.

[0035] like Figure 1 As shown, the memory fragmentation management method according to the embodiment of the present invention can be specifically performed according to the following steps:

[0036] Step S101: After receiving a governance request from any node in the distributed system when the memory fragmentation governance conditions are met, it is determined whether there is a leader copy in the node.

[0037] The memory fragmentation management method of the embodiments of the present invention can be applied in distributed systems. It is understood that for a computer system, when computing or storage requirements exceed the performance of a single server and require the coordinated operation of multiple servers, the computer system is called a distributed system. Generally, a distributed system can be composed of multiple nodes, each of which can correspond to a process or a service. In some practical scenarios, each node can also correspond to a real server.

[0038] Preferably, the memory fragmentation management method of the embodiment of the present invention can be applied to a memory-based distributed storage system (such as Redis, Tachyon), and can manage the memory fragmentation of any node in the system. It can be understood that in a memory-based distributed storage system, most of the data is stored in the memory, and data updates and memory releases are very frequent, which may generate a large amount of memory fragments, resulting in a high memory fragmentation rate and a waste of memory resources. Therefore, for a memory-based distributed storage system, a memory fragmentation management method that does not affect the business is more urgently needed. The following will mainly introduce the memory-based distributed storage system as an example. It should be noted that the memory fragmentation management method of the embodiment of the present invention is not limited to application in a memory-based distributed storage system, but can be applied to the memory fragmentation management process of any distributed system.

[0039] In some embodiments, the memory fragmentation rate can be used to measure the memory fragmentation of a corresponding node. The memory fragmentation rate of a node can be the ratio of the memory size requested by the node's process from the operating system to the memory size occupied by the node's data. Generally, when the memory fragmentation rate of a node exceeds a preset threshold, it indicates that the node has excessive memory fragmentation and requires treatment. The threshold can be set to 1.5 or 1.8, etc., depending on the actual scenario.

[0040] Figure 2 FIG. 1 is a schematic diagram of the composition of a distributed system according to an embodiment of the present invention. Figure 2 As shown, the distributed system includes multiple nodes ( Figure 2 Four nodes are shown as an example in the figure) and a scheduling module that can communicate with each node. The distributed system can be used to store the required data of the target application. These data can be data in one or more data tables required for the target application to run. The data required by the target application will be referred to as target data below.

[0041] In the above-mentioned distributed system, each node can store a copy of one or more shards of the target data. It can be understood that in the field of distributed computing, since the magnitude of the target data is generally large and difficult to store and read and write, the target data is divided into multiple shards for storage. For each shard, in order to avoid data loss caused by a single point of failure, each shard can be copied into multiple copies (the original shard can also be used as a copy) and stored in different nodes. In an embodiment of the present invention, each copy of the same shard of the target data can be stored in different nodes of the distributed system, that is, a node can only store one copy of a shard at most.

[0042] For any shard, its multiple replicas can be divided into leader replicas (leader) and follower replicas (follower) according to the replica roles. The leader replica and follower replica of the same shard form a consistency group. In the consistency group, the leader replica can interact with the application to realize data reading (i.e. providing the replica data to the application) and / or writing (i.e. writing the data provided by the application to the replica), and the follower replica can maintain consistency with the leader replica data according to the consistency algorithm (such as raft, paxos, etc.); when the leader replica fails, the follower replica can be re-elected to generate a new leader replica. In some embodiments, in order to disperse the data reading pressure of the leader replica, the follower replica can also receive external data reading requests and return the corresponding data. It should be noted that when the above leader replica or follower replica communicates with the outside world, it is essentially the node where the leader replica or follower replica is located that communicates with the outside world.

[0043] exist Figure 2 In this example, the target data is divided into three shards, and each shard has four copies. Figure 2 As shown, node 1 stores the leader copy of shard 1 ( Figure 2 In the figure, the boxes filled with high grayscale represent the leader replica, and the boxes of the same size that are not filled with high grayscale represent the follower replica), the follower replica of shard 2, and the follower replica of shard 3, node 2 stores the follower replica of shard 1, the leader replica of shard 2, and the follower replica of shard 3, node 3 stores the follower replica of shard 1, the follower replica of shard 2, and the leader replica of shard 3, and node 4 stores the follower replica of shard 1, the follower replica of shard 2, and the follower replica of shard 3.

[0044] Figure 3 This is a schematic diagram of data reading and writing of a replica in an embodiment of the present invention. Figure 3 In the figure, the solid lines represent actions or steps related to data reading, and the dotted lines represent actions or steps related to data writing. Figure 3As shown, when an application needs to read the replica data of a consistency group, it can directly communicate with the leader replica to read. When an application needs to write data to a consistency group, the following steps can be specifically performed: Step S301, the application sends a data write request to the leader replica of the consistency group, and the leader replica writes the relevant data locally. Step S302, the leader replica sends a data synchronization instruction to each follower replica of the consistency group. Step S303, after the follower replica synchronizes the relevant data locally, it returns a response to the leader replica indicating successful data synchronization. Step S304, the leader replica returns a response to the application indicating successful data write. It can be understood that the above process does not limit the memory fragmentation management method of the embodiment of the present invention. In actual applications, the leader replica can return a response to the application when some follower nodes successfully synchronize data, or it can directly return a response to the application when local writing is successful, and then instruct the follower nodes to synchronize data.

[0045] In specific applications, the scheduling module can be a process or a server, which can be used to monitor the operating status of each node, report monitoring data, and manage related metadata (such as the name of the data table corresponding to the target data). It can also interact with each node to achieve the addition and deletion of shards or copies. In particular, the memory fragmentation management method of the embodiment of the present invention can be executed by the scheduling module. As a preferred solution, the scheduling module can periodically receive the heartbeat signal sent by each node in the distributed system. The heartbeat signal can carry the identifier of the leader copy in the node, the identifier of the follower copy corresponding to the leader copy, and the identifier of the node where the follower copy is located. Through the above mechanism, the scheduling module can grasp the copy status of each node in real time.

[0046] In step S101, if any node in the distributed system determines that it meets the preset memory fragmentation management conditions (for example, the memory fragmentation rate of the node is higher than the preset threshold), a management request is sent to the scheduling module. After receiving the management request, the scheduling module can determine whether the node currently stores a leader copy based on the heartbeat signal most recently sent by the node: if not, it means that restarting the node will not affect business operations, and the node can be instructed to restart. It should be noted that there are two ways to instruct the node to restart: in the first way, the scheduling module can locally display relevant information that the node currently does not store a leader copy, and the staff can manually restart the node after seeing the information. In the second way, the scheduling module can send a restart instruction to the node, and the node automatically restarts after receiving the restart instruction.

[0047] Step S102: When a leader replica exists in the node, determine the follower replicas on other nodes corresponding to each leader replica in the node, and send an election instruction to the follower replica; the election instruction instructs the follower replica and the leader replica to perform a leader election, and limits the candidates for the leader election to the follower replica.

[0048] In this step, if the scheduling module determines that there is currently a leader copy in the node that issues the governance request (hereinafter referred to as the node to be restarted), it first determines the follower copy corresponding to each leader copy in the node based on the heartbeat signal most recently sent by the node to be restarted (that is, the follower copy in the same consistency group as the leader copy). It can be understood that the follower copies in the same consistency group as any leader copy of the node to be restarted can be distributed in other nodes (other nodes refer to nodes in the distributed system other than the node to be restarted). The following will be explained using this consistency group as an example.

[0049] After determining each follower replica of the consistency group, the scheduling module can send an election instruction to each determined follower replica to instruct the above follower replica to initiate a new leader election. As a preferred solution, the above election instruction sent by the scheduling module to each follower replica can indicate that the leader election can be performed by the leader replica in the consistency group (that is, the leader replica in the node to be restarted) and each follower replica in the consistency group (that is, the leader replica and each follower replica can vote). At the same time, the above election instruction also limits the candidates for the leader election to the follower replicas in the consistency group, that is, excludes the possibility of the leader replica in the node to be restarted winning the leader election, so that the leader replica in the node to be restarted can be scheduled outside the node to be restarted. It can be understood that the leader replica in the node to be restarted will become a follower replica after the election.

[0050] In another alternative solution, after the scheduling module sends an election instruction to each follower replica in the consistency group, these follower replicas perform the leader election together with the leader replica in the node to be restarted according to a preset algorithm (i.e., the leader replica also has voting rights). Because only the follower replicas receive the election instruction sent by the scheduling module, according to the constraints of the preset algorithm, the candidates for the leader election can only be among the follower replicas. In this way, the leader replica in the node to be restarted can also be scheduled outside the node to be restarted.

[0051] Step S103: After obtaining the election result of the leader election, the node is restarted.

[0052] In this step, after the leader election for each leader replica and its corresponding consistency group in the node to be restarted is complete, the newly elected leader replica can send the election results to the scheduling module. It is understood that the election results may include the identifier and node identifier of the current leader replica in the corresponding consistency group, as well as the identifier and node identifier of the current follower replica. Furthermore, under normal circumstances, the leader replica in the election results is not stored in the node to be restarted. As an alternative, the newly elected leader replica can also send the election results to the scheduling module via a heartbeat signal.

[0053] After receiving the election results, if the scheduling module determines from the election results that the node to be restarted no longer has a leader replica, it can display the election results locally. A staff member can then manually restart the node to be restarted to manage memory fragmentation. In some embodiments, after determining that the node to be restarted no longer has a leader replica, the scheduling module can send a restart instruction to the node to be restarted. Upon receiving the restart instruction, the node to be restarted can automatically execute the restart.

[0054] Through the above steps, before the node to be restarted that meets the memory fragmentation management conditions is restarted, its stored leader copy can be scheduled to other nodes through election, and then restarting the node will not affect business operations and user responses. The above election is a normal election process (that is, it does not correspond to a fault scenario), which generally only takes a few milliseconds, which is much less than the special election time caused by directly restarting the node in the prior art (generally a few seconds). Therefore, the memory fragmentation management method of the embodiment of the present invention can achieve effective management of distributed system node memory fragmentation without affecting business operations and user responses.

[0055] Figure 4 FIG. 1 is a schematic diagram of specific execution steps of the memory fragmentation management method according to an embodiment of the present invention. Figure 4 As shown, in step S401, the scheduling module receives a governance request sent by any node in the distributed system. In step S402, the scheduling module determines whether the node currently stores a leader copy based on the heartbeat signal sent by the node: if so, execute step S403; otherwise, execute step S407. In step S403, the scheduling module determines the follower copy corresponding to each leader copy in the node. In step S404, the scheduling module sends an election instruction to each follower copy of the same consistency group. In step S405, the scheduling module obtains the election result sent by the newly elected leader copy. In step S406, the scheduling module determines whether the node currently has a leader copy based on the election result: if so, return to step S403; otherwise, execute step S407. In step S407, the scheduling module instructs the node to restart, thereby ending the process.

[0056] In the technical solution of the embodiment of the present invention, the scheduling module in the distributed system grasps the current copy status of each node in real time through the heartbeat signal sent by each node. When a node meets the memory fragmentation management conditions and needs to be restarted, the node sends a management request to the scheduling module. The scheduling module first determines whether there is a leader copy in the node. If not, it can directly instruct the node to restart; if so, it sends an election instruction to the follower copy of each leader copy in the node corresponding to the follower copy in other nodes, so that the leader copy and the follower copy elect a new leader copy from the follower copy. Afterwards, when it is determined that there is no leader copy in the node to be restarted, the node can be instructed to restart. Through the above settings, the nodes that meet the memory fragmentation management conditions can execute the normal election process before restarting, so that the leader copy therein can be scheduled to other nodes. After that, restarting the node will not affect the business and user experience, thereby achieving effective memory fragmentation management without affecting the business.

[0057] It should be noted that for the aforementioned method embodiments, for ease of description, they are expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited to the order of the actions described, and certain steps can actually be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required to implement the present invention.

[0058] In order to better implement the above solutions of the embodiments of the present invention, relevant devices for implementing the above solutions are also provided below.

[0059] See also Figure 5 As shown, the memory fragmentation management device 500 provided by the embodiment of the present invention may include: a leader judgment unit 501, an election instruction unit 502 and an execution unit 503.

[0060] Among them, the leader judgment unit 501 can be used to: after receiving a governance request issued by any node in the distributed system when the memory fragmentation governance conditions are met, determine whether there is a leader copy in the node; wherein each node in the distributed system stores a copy of at least one shard of the target data, and the copy includes a leader copy and a follower copy; the election instruction unit 502 can be used to: when there is a leader copy in the node, determine that each leader copy in the node corresponds to a follower copy in other nodes, and send an election instruction to the follower copy; the election instruction instructs the follower copy and the leader copy to perform a leader election, and limits the candidates for the leader election to the follower copy; the execution unit 503 can be used to obtain the election result of the leader election; and, after obtaining the election result, the node is restarted.

[0061] In an embodiment of the present invention, the leader judgment unit 501 can be further used to: receive a heartbeat signal sent by each node in the distributed system, and judge whether there is a leader copy in the node that issues the governance request based on the heartbeat signal; wherein the heartbeat signal sent by any node carries the identifier of the leader copy in the node and the identifier of the follower copy corresponding to the leader copy and the node identifier.

[0062] In a specific application, the execution unit 503 may be further configured to: in response to the election result indicating that the node currently has no leader replica, send a restart instruction to the node; and the node restarts after receiving the restart instruction.

[0063] In actual applications, the execution unit 503 can be further used to: when the leader judgment unit 501 determines that there is no leader copy in the node that issues the governance request, send a restart instruction to the node; and the node executes a restart after receiving the restart instruction.

[0064] In an optional solution, the memory fragmentation management condition includes: a memory fragmentation ratio is greater than a threshold.

[0065] Preferably, each copy of a shard of the target data is stored in a different node of the distributed system.

[0066] In addition, in an embodiment of the present invention, the distributed system is a memory-based distributed storage system; the leader replica is used to interact with the application to realize data reading and / or writing, and the follower replica maintains data consistency with the corresponding leader replica.

[0067] The embodiment of the present invention also provides a memory fragmentation management system, such as Figure 2 shown.

[0068] The memory fragmentation management system may include multiple nodes and a scheduling module, wherein each node in the system stores a copy of at least one shard of the target data, wherein the copy includes a leader copy and a follower copy; wherein the scheduling module receives a management request issued by any node when the memory fragmentation management conditions are met, and determines whether there is a leader copy in the node; when it is determined that there is a leader copy in the node, the scheduling module determines that each leader copy in the node corresponds to a follower copy in other nodes, and sends an election instruction to the follower copy; the election instruction instructs the follower copy and the leader copy to perform a leader election, and limits the candidates for the leader election to the follower copy; the scheduling module obtains the election result of the leader election; and, after obtaining the election result, the node is restarted.

[0069] In an embodiment of the present invention, the scheduling module receives a heartbeat signal sent by each node in the system, and determines whether there is a leader copy in the node that issues the governance request based on the heartbeat signal; wherein, the heartbeat signal sent by any node carries the identifier of the leader copy in the node and the identifier of the follower copy corresponding to the leader copy and the node identifier.

[0070] In a specific application, when the scheduling module determines that there is no leader copy in the node that issues the governance request, it sends a restart instruction to the node; and the node executes the restart after receiving the restart instruction.

[0071] In actual applications, in response to the election result indicating that the node that issued the governance request currently has no leader copy, the scheduling module sends a restart instruction to the node; and the node executes a restart after receiving the restart instruction.

[0072] In addition, in an embodiment of the present invention, the memory fragmentation management conditions include: the memory fragmentation ratio is greater than a threshold; each copy of a shard of the target data is stored on different nodes of the system; the system is a memory-based distributed storage system; the leader copy is used to interact with the application to realize data reading and / or writing, and the follower copy maintains data consistency with the corresponding leader copy.

[0073] In the technical solution of the embodiment of the present invention, the scheduling module in the distributed system grasps the current copy status of each node in real time through the heartbeat signal sent by each node. When a node meets the memory fragmentation management conditions and needs to be restarted, the node sends a management request to the scheduling module. The scheduling module first determines whether there is a leader copy in the node. If not, it can directly instruct the node to restart; if so, it sends an election instruction to the follower copy of each leader copy in the node corresponding to the follower copy in other nodes, so that the leader copy and the follower copy elect a new leader copy from the follower copy. Afterwards, when it is determined that there is no leader copy in the node to be restarted, the node can be instructed to restart. Through the above settings, the nodes that meet the memory fragmentation management conditions can execute the normal election process before restarting, so that the leader copy therein can be scheduled to other nodes. After that, restarting the node will not affect the business and user experience, thereby achieving effective memory fragmentation management without affecting the business.

[0074] The present invention also provides an electronic device. The electronic device in an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the memory fragmentation management method provided by the present invention.

[0075] Reference below Figure 6 , which shows a schematic structural diagram of a computer system 600 of an electronic device suitable for implementing an embodiment of the present invention. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0076] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the computer system 600 are also stored in the RAM 603. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0077] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed in the storage section 608 as needed.

[0078] In particular, according to embodiments disclosed herein, the processes described in the main step diagrams above can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via the communication section 609 and / or installed from a removable medium 611. When the computer program is executed by the central processing unit 601, the above-described functions defined in the system of the present invention are performed.

[0079] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0081] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as: a processor including a leader determination unit, an election instruction unit, and an execution unit. The names of these units do not, in some cases, limit the units themselves. For example, the leader determination unit may also be described as "providing the election instruction unit with a determination result of whether a leader replica exists in the corresponding node."

[0082] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the device, the device performs the following steps: after receiving a governance request issued by any node in the distributed system when the memory fragmentation governance condition is met, determining whether there is a leader copy in the node; wherein each node in the distributed system stores a copy of at least one shard of the target data, the copy includes a leader copy and a follower copy; when there is a leader copy in the node, determining that each leader copy in the node corresponds to a follower copy in other nodes, and sending an election instruction to the follower copy; the election instruction instructs the follower copy and the leader copy to perform a leader election, and limits the candidates for the leader election to the follower copy; after obtaining the election result of the leader election, the node is restarted.

[0083] In the technical solution of the embodiment of the present invention, the scheduling module in the distributed system grasps the current copy status of each node in real time through the heartbeat signal sent by each node. When a node meets the memory fragmentation management conditions and needs to be restarted, the node sends a management request to the scheduling module. The scheduling module first determines whether there is a leader copy in the node. If not, it can directly instruct the node to restart; if so, it sends an election instruction to the follower copy of each leader copy in the node corresponding to the follower copy in other nodes, so that the leader copy and the follower copy elect a new leader copy from the follower copy. Afterwards, when it is determined that there is no leader copy in the node to be restarted, the node can be instructed to restart. Through the above settings, the nodes that meet the memory fragmentation management conditions can execute the normal election process before restarting, so that the leader copy therein can be scheduled to other nodes. After that, restarting the node will not affect the business and user experience, thereby achieving effective memory fragmentation management without affecting the business.

[0084] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A memory fragmentation management method, characterized in that: include: After receiving a governance request from any node in the distributed system when a memory fragmentation governance condition is met, determining whether a leader replica exists in the node; wherein each node in the distributed system stores a replica of at least one shard of target data, the replicas including a leader replica and a follower replica; the memory fragmentation governance condition includes: a memory fragmentation ratio greater than a threshold; When a leader replica exists in the node, determine the follower replicas on other nodes corresponding to each leader replica in the node, and send an election instruction to the follower replica; the election instruction instructs the follower replica and the leader replica to perform a leader election, and restricts candidates for the leader election to the follower replica; After obtaining the election result of the leader election, the node is restarted; after the node completes the restart, the replica stored in the node is restored and added to the corresponding consistency group as a follower replica.

2. The method according to claim 1, characterized in that The method further includes: receiving a heartbeat signal sent by each node in the distributed system, the heartbeat signal carrying an identifier of a leader replica in the node, an identifier of a follower replica corresponding to the leader replica, and an identifier of the node where the follower replica is located; Determining whether there is a leader copy in the node includes: determining whether there is a leader copy in the node according to a heartbeat signal sent by the node.

3. The method according to claim 1, characterized in that After obtaining the leader election result, the node is restarted, including: In response to the election result indicating that no leader replica currently exists at the node, a restart instruction is sent to the node; and the node performs a restart after receiving the restart instruction.

4. The method according to claim 1, wherein The method further comprises: When it is determined that there is no leader copy in the node that issues the governance request, a restart instruction is sent to the node; and the node executes a restart after receiving the restart instruction.

5. The method according to claim 1, wherein Replicas of a shard of the target data are stored on different nodes of the distributed system.

6. The method according to any one of claims 1 to 5, characterized in that: The distributed system is a memory-based distributed storage system; The leader replica is used to interact with the application to implement data reading and / or writing, and the follower replica maintains data consistency with the corresponding leader replica.

7. A memory fragmentation management device, characterized in that: include: A leader determination unit is configured to: upon receiving a governance request from any node in the distributed system when a memory fragmentation governance condition is met, determine whether a leader replica exists in the node; wherein each node in the distributed system stores a replica of at least one shard of target data, the replicas including a leader replica and a follower replica; and the memory fragmentation governance condition includes: a memory fragmentation ratio greater than a threshold; an election instruction unit, configured to: when a leader replica exists in the node, determine the follower replicas on other nodes corresponding to each leader replica in the node, and send an election instruction to the follower replica; the election instruction instructs the follower replica and the leader replica to perform a leader election, and restricts candidates for the leader election to the follower replica; An execution unit is used to obtain an election result of a leader election; and after obtaining the election result, the node is restarted; after the node completes the restart, the replica stored in the node is restored to the corresponding consistency group as a follower replica.

8. A memory fragmentation management system, characterized in that: The system includes multiple nodes and a scheduling module, each node stores a copy of at least one shard of target data, the copy including a leader copy and a follower copy; wherein, The scheduling module receives a governance request from any node when the memory fragmentation governance condition is met, and determines whether there is a leader copy in the node; the memory fragmentation governance condition includes: the memory fragmentation ratio is greater than a threshold; When it is determined that a leader replica exists in the node, the scheduling module determines the follower replicas on other nodes corresponding to each leader replica in the node, and sends an election instruction to the follower replica; the election instruction instructs the follower replica and the leader replica to perform a leader election, and limits the candidates for the leader election to the follower replica; The scheduling module obtains the election result of the leader election; and after obtaining the election result, the node is restarted; after the node completes the restart, the copy stored in the node is restored to the corresponding consistency group as a follower copy.

9. The system according to claim 8, characterized in that The scheduling module receives a heartbeat signal from each node in the system, and determines whether there is a leader copy in the node that issues the governance request based on the heartbeat signal; The heartbeat signal sent by any node carries the identifier of the leader replica in the node, the identifier of the follower replica corresponding to the leader replica, and the node identifier.

10. The system according to claim 8 or 9, characterized in that The memory fragmentation management conditions include: the memory fragmentation ratio is greater than a threshold; Replicas of a shard of the target data are stored in different nodes of the system; The leader replica is used to interact with the application to implement data reading and / or writing, and the follower replica maintains data consistency with the corresponding leader replica.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images