A distributed storage packet online upgrading method and device, a terminal and a medium

Abstract

The application relates to a distributed storage packet online upgrading method and device, a terminal and a medium. The application groups nodes to be upgraded according to a preset grouping rule, the grouping rule makes the nodes to be upgraded in different groups not involve the same storage pool; an upgrading thread is configured for each group to control the upgrading process of the nodes to be upgraded in the group, each upgrading thread is started at a set first time interval, and each group upgrades in a concurrent staggered manner. The application can compress the total time length of online upgrading through concurrent upgrading among multiple groups. The nodes to be upgraded in different groups do not involve the same storage pool, when the nodes to be upgraded in the group are upgraded in sequence one by one, the nodes to be upgraded corresponding to the same storage pool will not be upgraded at the same time, and the staggered upgrading among the groups avoids the situation that the nodes to be upgraded restart services at the same time on a large scale, leading to service interruption.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage cluster grouping online upgrade, and in particular to a distributed storage grouping online upgrade method and device, a terminal and a medium. BACKGROUND

[0002] The distributed storage device has the characteristics of high scalability, and can be composed of several nodes to thousands of nodes. Each node can contain several storage devices, such as solid state disks, magnetic disks, etc. Each storage device is managed by an OSD service process. Data is mapped to PG and OSD by the CRUSH algorithm. PG and OSD can be divided into different storage pools. In addition to the OSD service, the cluster also has a MON service to monitor the cluster state and save osdmap and other cluster metadata information. The MON service does not need to be deployed to all nodes, and generally needs to deploy an odd number of MON services.

[0003] The online upgrade of the distributed storage generally issues an upgrade instruction from the control node to the upgrade node, the upgrade node executes the installation package of the new version, replaces the executable files, related library files and configuration files of the MON, OSD and other services of the distributed cluster, and then restarts the service to take effect. Since online upgrade often requires uninterrupted business, only one node is upgraded each time. For large-scale clusters, the upgrade time will be very long. Assuming that each node upgrade takes 10 minutes, for a 100-node cluster, it takes 1000 minutes, more than 16 hours of upgrade time. This is unacceptable for users who need to fix the upgrade time to a few hours of low business period. In order to optimize the length of online upgrade, the current scheme adopts the grouping method of upgrade nodes, and nodes in different storage pools that do not interfere with each other are divided into several groups, and a node is selected from each group for upgrade each time. However, the current scheme has multiple groupings, such as multiple storage pool divisions, and multiple nodes are upgraded at the same time each time. Although these nodes are located in different storage pools and do not interfere with each other in terms of failure domains, data reconstruction, PG Peering, etc., restarting multiple nodes at the same time will cause OSD to report a large number of messages to MON, which cannot be processed in time, causing the PG to be stuck in the Peering stage, thereby affecting the front-end business. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a distributed storage grouping online upgrade method, device, terminal and medium.

[0005] In a first aspect, the present application provides a distributed storage grouping online upgrade method, comprising: grouping the nodes to be upgraded according to a preset grouping rule, the grouping rule causing the nodes to be upgraded in different groups not to involve the same storage pool;

[0006] An upgrade thread is configured for each group to control the upgrade process of the nodes to be upgraded in the group, and each upgrade thread is started at a first time interval to enable the groups to upgrade concurrently and staggered in time, thereby avoiding a large-scale restart of the services of the nodes to be upgraded at the same time.

[0007] Further, the upgrade thread invokes the upgrade script on each node to be upgraded in the group via ssh, and the upgrade script executes the upgrade package, verifies the restart condition, and restarts the service when the restart condition is met or when the time since the execution of the upgrade package exceeds a set time threshold and the restart condition is still not met, and performs data reconstruction after the service is restarted.

[0008] Further, the verification of the restart condition includes: after the upgrade script executes the upgrade package, it is determined whether all OSDs of the node to be upgraded are in the Up stage and no placement group is in the peering state, and if so, the restart condition is met and the service is restarted, otherwise, the verification of the restart condition is performed again after a second time interval, until the time since the execution of the upgrade package reaches the set time threshold or the restart condition is met and the verification of the restart condition is terminated.

[0009] Further, the upgrade thread iterates through each node to be upgraded in the corresponding group, and invokes the upgrade script on the iterated node to be upgraded via ssh, and after the upgrade script of the iterated node to be upgraded performs data reconstruction, it continues to iterate through the next node to be upgraded in the group until the iteration through the group is completed.

[0010] Further, the nodes to be upgraded that meet the restart condition and restart the service and the nodes to be upgraded that reach the set time threshold and restart the service are counted and corresponding logs are output.

[0011] Further, the group rule includes: counting the storage pools to which the OSDs of each node to be upgraded belong, and when all OSDs of each node to be upgraded belong to one storage pool, the nodes to be upgraded are grouped according to the storage pools; when the OSDs of each node to be upgraded belong to different storage pools, the nodes to be upgraded that do not cross the storage pools are grouped into different groups, and the nodes to be upgraded that cross the storage pools are grouped into one group, and if there is a node to be upgraded that crosses the nodes to be upgraded in different groups, the different groups are merged into the same group.

[0012] In a second aspect, the application provides a distributed storage group online upgrade device, comprising: a grouping module that groups nodes to be upgraded according to a preset grouping rule, and the grouping rule enables the nodes to be upgraded in different groups to not involve the same storage pool.

[0013] The upgrade control module configures an upgrade thread for each group to control the upgrade process of the nodes to be upgraded in the group. The upgrade control module starts each upgrade thread at a set first time interval, so that each group can upgrade concurrently and at different times, avoiding restarting the service at the same time.

[0014] The upgrade execution module is configured with an upgrade script. The upgrade script executes the upgrade package, verifies the restart conditions, and restarts the service if the restart conditions are met or if the time since the execution of the upgrade package exceeds a set time threshold and the restart conditions are not met. After restarting the service, data reconstruction is performed.

[0015] Furthermore, the distributed storage group online upgrade device also includes a log module, which is used to statistically output the nodes to be upgraded that meet the restart conditions and restart the service, and output the corresponding logs.

[0016] Thirdly, the present invention provides a distributed storage group online upgrade terminal, comprising: a processing unit, a bus unit, and a storage unit, wherein the bus unit is connected to the storage unit and the processing unit, the storage unit stores a computer program, and the computer program is executed by the processing unit to implement the distributed storage group online upgrade method.

[0017] Fourthly, the present invention provides a storage medium for implementing a distributed storage group online upgrade method, wherein the storage medium stores a computer program, and the computer program, when executed by a processor, implements the distributed storage group online upgrade method.

[0018] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art:

[0019] This invention groups nodes to be upgraded according to preset grouping rules. Nodes in different groups do not share the same storage pool. Each group is configured with an upgrade thread to control the upgrade process of the nodes within that group. Each upgrade thread is started at a set first time interval, allowing for concurrent and staggered upgrades across groups. This invention enables concurrent upgrades across multiple groups, reducing the total online upgrade time. Since nodes in different groups do not share the same storage pool, and upgrades are performed sequentially, nodes within the same storage pool will not be upgraded simultaneously (two or more nodes will not be upgraded at the same time). Furthermore, the staggered upgrades between groups prevent service interruptions caused by a large number of nodes restarting simultaneously. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating an online upgrade method for distributed storage groups provided in an embodiment of the present invention;

[0023] Figure 2 This is a flowchart illustrating the process of grouping nodes to be upgraded using grouping rules, as provided in an embodiment of the present invention.

[0024] Figure 3 A flowchart illustrating the upgrade process of a node to be upgraded in an upgrade thread control group, provided in an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram illustrating the effect of the upgrade thread starting at a preset first time interval, as provided in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of a distributed storage group online upgrade device provided in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of a distributed storage group online upgrade terminal provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0030] The English terms or abbreviations used in this application have the following meanings: OSD (Object-based Storage Device) refers to an object storage device. PG stands for Placement Group. Peering is one of the storage device processes, which generally consists of four stages: Getinfo, Getlog, Getmissing, and Active. A waitupThru may be performed after Getmissing if necessary. Getinfo: The primary OSD of the PG collects pg_info_t information from other secondary OSDs. GetLog: The OSD with the authoritative log is selected. If this OSD is not the primary OSD, the authoritative log is pulled from that OSD and sent to the primary OSD. GetMissing: The primary OSD pulls pg_entry_t from other OSDs and compares it with the authoritative log to determine the missing object information on each OSD.

[0031] Example 1

[0032] See Figure 1 As shown, the present invention provides a method for online upgrade of distributed storage groups, comprising:

[0033] S100: The nodes to be upgraded are grouped according to a preset grouping rule, which ensures that the nodes to be upgraded in different groups do not involve the same storage pool. For specific implementation details, please refer to [link to relevant documentation]. Figure 2As shown, the grouping rules include: Firstly, identifying the storage pool to which each OSD of a node to be upgraded belongs. If all OSDs of a node to be upgraded belong to one storage pool, then the nodes are grouped according to their respective storage pools. For example, if all OSDs of node A to be upgraded belong to storage pool 1, all OSDs of node B to be upgraded belong to storage pool 2, all OSDs of node C to be upgraded belong to storage pool 1, and all OSDs of node D to be upgraded belong to storage pool 2, then nodes A and C are grouped together, and nodes B and D are grouped together. When the OSDs of each node to be upgraded belong to different storage pools, it is checked whether there is any overlap in storage pools. Nodes with overlapping storage pools are grouped into the same group, while nodes without overlapping storage pools are grouped into different groups. If, for different groups, there is an overlap between a node to be upgraded and a node in each group, then the different groups are merged into the same group. For example: If some OSDs of node A to be upgraded belong to storage pool 1 and some OSDs belong to storage pool 2; if some OSDs of node B to be upgraded belong to storage pool 3 and some OSDs belong to storage pool 4, then node A and node B do not have overlapping storage pools and are divided into two groups. If some OSDs of node C to be upgraded belong to storage pool 1 and some OSDs belong to storage pool 2; if some OSDs of node D to be upgraded belong to storage pool 1 and some OSDs belong to storage pool 4, then node C and node D overlap in storage pool 1 and are divided into one group. If some OSDs of node E to be upgraded belong to storage pool 1 and some OSDs belong to storage pool 4, then node E overlaps in storage pools with both node A and node B. In this case, the two groups of node A and node B to be upgraded are merged into one group, and node E to be upgraded is added to the group.

[0034] S200 configures an upgrade thread for each group to control the upgrade process of the nodes to be upgraded in the group. For specific implementation details, please refer to [link / reference needed]. Figure 3 As shown, the upgrade process for the nodes to be upgraded in the upgrade thread control group includes:

[0035] S201: The upgrade thread iterates through each node to be upgraded in the corresponding group and calls the upgrade script on the visited node via SSH. Each group is stored as a list. The upgrade thread iterates through the nodes to be upgraded in the list, and for each visited node, the upgrade thread calls the upgrade script via SSH.

[0036] S202, the upgrade script executes the upgrade package, verifies the restart conditions, and restarts the service if the restart conditions are met or if the time since executing the upgrade package exceeds a set time threshold without meeting the restart conditions. Data reconstruction is then performed after the service restarts. The upgrade script executes the upgrade package to install the program to be upgraded. The verification of the restart conditions includes: after executing the upgrade package, determining whether all OSDs on the node to be upgraded are in the Up phase and no placement groups are in the peering state; if so, the restart conditions are met and the service is restarted; otherwise, the restart conditions are verified again after a second time interval until the time since executing the upgrade package reaches the set time threshold or the restart conditions are met, at which point the verification of the restart conditions is terminated. One feasible second time interval is 4 seconds, and another feasible set time threshold is 1 minute.

[0037] S203: Check whether the upgrade script in the traversed upgrade node has completed data reconstruction. If the upgrade script in the traversed upgrade node has completed data reconstruction, then execute S204.

[0038] S204, check if the group has been traversed. Otherwise, continue to execute S201, traverse the next node to be upgraded in the group, and end if it is.

[0039] S300 starts each upgrade thread at a pre-defined first time interval, allowing each group to upgrade concurrently and at staggered times, thus avoiding a large-scale simultaneous service restart of nodes awaiting upgrade. For specific implementation details, please refer to [link / reference]. Figure 4 As shown, the upgrade threads of each group are controlled to start at a set first time interval, one possible first time interval being 1 minute. The first time interval is basically consistent with the set time threshold.

[0040] In practice, the system statistically analyzes and outputs the nodes that need to be upgraded and will restart the service if they meet the restart conditions or reach the set time threshold, and outputs corresponding logs. The upgrade and update status of each node is then obtained from the logs.

[0041] Example 2

[0042] See Figure 5 As shown, this embodiment of the invention provides a distributed storage group online upgrade device, including: a grouping module, which groups the nodes to be upgraded according to a preset grouping rule, wherein the grouping rule ensures that the nodes to be upgraded in different groups do not involve the same storage pool.

[0043] The upgrade control module configures an upgrade thread for each group to control the upgrade process of the nodes to be upgraded in the group. The upgrade control module starts each upgrade thread at a set first time interval, so that each group can upgrade concurrently and at different times, avoiding restarting the service at the same time.

[0044] The upgrade execution module is configured with an upgrade script. The upgrade script executes the upgrade package, verifies the restart conditions, and restarts the service if the restart conditions are met or if the time since the execution of the upgrade package exceeds a set time threshold and the restart conditions are not met. After restarting the service, data reconstruction is performed.

[0045] The logging module is used to statistically analyze and output logs for nodes that are scheduled to restart when they meet the restart conditions and when they reach a set time threshold.

[0046] Example 3

[0047] See Figure 6 As shown, this embodiment of the invention provides a distributed storage group online upgrade terminal, including: a processing unit, a bus unit, and a storage unit, wherein the bus unit connects the storage unit and the processing unit, the storage unit stores a computer program, and the computer program is executed by the processing unit to implement the distributed storage group online upgrade method.

[0048] Example 4

[0049] This invention provides a storage medium for implementing an online upgrade method for distributed storage groups. The storage medium stores a computer program, which, when executed by a processor, implements the online upgrade method for distributed storage groups.

[0050] This invention groups nodes to be upgraded according to preset grouping rules. Nodes in different groups do not share the same storage pool. Each group is configured with an upgrade thread to control the upgrade process of the nodes within that group. Each upgrade thread is started at a set first time interval, allowing for concurrent and staggered upgrades across groups. This invention enables concurrent upgrades across multiple groups, reducing the total online upgrade time. Since nodes in different groups do not share the same storage pool, and upgrades are performed sequentially, nodes within the same storage pool will not be upgraded simultaneously (two or more nodes will not be upgraded at the same time). Furthermore, the staggered upgrades between groups prevent service interruptions caused by a large number of nodes restarting simultaneously.

[0051] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0052] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0053] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0054] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0055] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0056] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should also be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.

Claims

1. A method for online upgrade of distributed storage groups, characterized in that, include: The nodes to be upgraded are grouped according to a preset grouping rule, which ensures that the nodes to be upgraded in different groups do not involve the same storage pool. The grouping rules include: counting the storage pools to which the OSDs of each node to be upgraded belong; if all OSDs of each node to be upgraded belong to one storage pool, then group them by storage pool; if the OSDs of each node to be upgraded belong to different storage pools, then the nodes to be upgraded that have overlapping storage pools are grouped together, and the nodes to be upgraded that do not have overlapping storage pools are grouped into different groups; if for different groups, there is a node to be upgraded that overlaps with nodes to be upgraded in different groups, then the different groups are merged into the same group. Configure an upgrade thread for each group to control the upgrade process of the nodes to be upgraded in the group. Start each upgrade thread at a set first time interval to enable the groups to upgrade concurrently and staggeredly, avoiding a large number of nodes to be upgraded restarting services at the same time. Set verification restart conditions to restrict restarts during the upgrade process, including: after the upgrade script executes the upgrade package, check whether all OSDs of the nodes to be upgraded are in the Up phase and no placement group is in the peering state. If so, the restart conditions are met and the service is restarted. Otherwise, after a second time interval, verify the restart conditions again until the time after the upgrade package is executed reaches the set time threshold or the restart conditions are met, and then terminate the verification restart conditions.

2. The distributed storage group online upgrade method according to claim 1, characterized in that, The upgrade thread calls the upgrade scripts on each node to be upgraded in the group via SSH. The upgrade scripts execute the upgrade package, verify the restart conditions, and restart the service if the restart conditions are met or if the time since the execution of the upgrade package exceeds the set time threshold and the restart conditions are not met. After restarting the service, data reconstruction is performed.

3. The distributed storage group online upgrade method according to claim 2, characterized in that, The upgrade thread iterates through each node to be upgraded in the corresponding group, and calls the upgrade script on the traversed upgrade node via SSH. After the upgrade script on the traversed upgrade node finishes executing and reconstructing the data, it continues to traverse the next node to be upgraded in the group until the group traversal is complete.

4. The distributed storage group online upgrade method according to claim 2, characterized in that, The system outputs statistics on nodes that meet the restart conditions and are scheduled to restart after reaching a set time threshold, along with corresponding logs.

5. A distributed storage group online upgrade device, characterized in that, include: The grouping module groups the nodes to be upgraded according to a preset grouping rule. This grouping rule includes: identifying the storage pool to which each node's OSDs belong; if all OSDs of a node belong to the same storage pool, then the nodes are grouped by storage pool; if the OSDs of a node belong to different storage pools, nodes with overlapping storage pools are grouped together, and nodes without overlapping storage pools are grouped into different groups; if, for different groups, a node overlaps with nodes in different groups, then the different groups are merged into a single group; the grouping rule ensures that nodes to be upgraded in different groups do not involve the same storage pool. The upgrade control module configures an upgrade thread for each group to control the upgrade process of the nodes to be upgraded in the group. The upgrade control module starts each upgrade thread at a set first time interval, so that each group can upgrade concurrently and at different times, avoiding restarting the service at the same time. The upgrade execution module is configured with an upgrade script. The upgrade script executes the upgrade package, verifies the restart conditions, and restarts the service if the restart conditions are met or if the time since the execution of the upgrade package exceeds a set time threshold and the restart conditions are not met. After restarting the service, data reconstruction is performed. During the upgrade process, the verification restart conditions that restrict restarts are set, including: after the upgrade script executes the upgrade package, it determines whether all OSDs on the node to be upgraded are in the Up phase and no placement group is in the peering state. If so, the restart conditions are met and the service is restarted. Otherwise, the restart conditions are verified again after a second time interval until the time since the execution of the upgrade package reaches the set time threshold or the restart conditions are met, at which point the verification restart conditions are terminated.

6. The distributed storage group online upgrade device according to claim 5, characterized in that, It also includes a logging module, which is used to statistically output the nodes to be upgraded that restart the service when they meet the restart conditions and the nodes that restart the service when they reach the set time threshold, and output the corresponding logs.

7. A distributed storage group online upgrade terminal, characterized in that, include: The system includes a processing unit, a bus unit, and a storage unit, wherein the bus unit connects the storage unit and the processing unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, it implements the distributed storage group online upgrade method as described in any one of claims 1-4.

8. A storage medium for implementing an online upgrade method for distributed storage groups, the storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the distributed storage group online upgrade method as described in any one of claims 1-4.

Images