Node isolation method and device, electronic equipment, chip and storage medium
By scoring the failure information and determining the status of the Ceph storage nodes and performing isolation operations, the business delay and cluster read-only problems caused by storage node failure are solved, and cluster stability is improved.
Patent Information
- Application Number
- CN202410990260.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-15
AI Technical Summary
In Ceph distributed storage systems, hardware failure of storage nodes causes slow response of object storage daemons, and deleting OSD operations will trigger group migration, resulting in increased service delays and cluster read-only risks.
By determining the score based on the fault information of the failed node and performing isolation operations on the to-process nodes when specific conditions are met, including stopping the OSD process and shutting down self-start services for monitoring and managing the process, avoiding deleting the OSD operation and preventing PG migration.
Effectively handle node hardware failures, avoid business delays and cluster read-only risks caused by PG migration, and improve the stability of cluster operation and maintenance.
Smart Images

Figure CN118964072A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing, and in particular to a node method, device, electronic device, chip and storage medium. Background Art
[0002] When a hardware failure occurs in a storage node of the distributed storage system Ceph, the Object Storage Daemon (OSD) will respond more slowly. At this time, the OSD of the storage node can be deleted, but it will trigger the migration of the placement group (PG), resulting in increased business latency and risks such as cluster read-only. Summary of the invention
[0003] The present disclosure provides a node isolation method, device, electronic device, chip and storage medium.
[0004] The first aspect of the present disclosure provides a node isolation method, which includes determining a score of a faulty node based on fault information of the faulty node; determining a faulty node whose score is greater than or equal to a score threshold as a node to be processed; determining a state of the node to be processed; and performing an isolation operation on the node to be processed when the state of the node to be processed meets a first condition.
[0005] In some embodiments of the present disclosure, determining the score of the faulty node based on the fault information of the faulty node includes: obtaining address information of the faulty node; obtaining fault information of the faulty node based on the address information of the faulty node; and determining the score of the faulty node based on the fault information of the faulty node.
[0006] In some embodiments of the present disclosure, determining the status of the node to be processed includes: determining that the status of the placement group PG related to the resource pool where the object storage daemon OSD of the node to be processed is located is a first state; determining that the status of the PG related to the OSD of the node to be processed is a second state; and determining the quantity status of the node to be processed.
[0007] In some embodiments of the present disclosure, before determining that the state of the placement group PG related to the resource pool where the object storage daemon OSD of the node to be processed is located is a first state, the method also includes: determining the fault domain where the node to be processed is located; determining the resource pool where the OSD of the node to be processed is located based on the domain name of the fault domain; and determining the PG related to the resource pool.
[0008] In some embodiments of the present disclosure, determining that the state of the PG related to the object storage daemon OSD of the node to be processed is the second state includes: determining the current working state of the cluster, the cluster includes the node to be processed; when the current working state of the cluster is to create a resource pool or delete a resource pool, the second state is a progress state; otherwise, the second state is a non-progress state.
[0009] In some embodiments of the present disclosure, the first condition includes at least one of the following: the first state is an active state; the second state is a non-progressing state; the quantity state is: the number of active OSDs related to the PG is greater than the minimum number of replicas of the resource pool.
[0010] In some embodiments of the present disclosure, performing isolation operations on the node to be processed includes: stopping all OSDs in UP state on the node to be processed; shutting down the boot-up self-starting service of the OSDs in UP state; stopping all monitoring processes MON and / or management processes MGR in UP state on the node to be processed; shutting down the boot-up self-starting service of the monitoring processes MON and / or management processes MGR in UP state.
[0011] The second aspect of the present disclosure proposes a node isolation device, including a first processing unit, used to determine the score of the faulty node according to the fault information of the faulty node; a second processing unit, used to determine the faulty node whose score is greater than or equal to the score threshold as a node to be processed; a third processing unit, used to determine the state of the node to be processed; and a fourth processing unit, used to perform an isolation operation on the node to be processed when the state of the node to be processed meets the first condition.
[0012] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.
[0013] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.
[0014] A fifth aspect embodiment of the present disclosure provides a computer program product, including a computer program, which executes the method described in the first aspect embodiment of the present disclosure when a processor is used to execute the computer program.
[0015] The sixth aspect embodiment of the present disclosure proposes a chip, which includes one or more interfaces and one or more processors; the interface is used to receive signals from a memory of an electronic device and send signals to the processor, the signals include computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.
[0016] In summary, the node isolation method, device, electronic device, chip and storage medium proposed in the present invention can judge the fault information and status of the node, isolate the node when the conditions are met, and handle the hardware failure of the node. At the same time, it can avoid the risks of business delays and cluster read-only caused by PG migration, thereby improving the stability of the cluster operation and maintenance process.
[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0019] Figure 1 A flowchart of a node isolation method provided by an embodiment of the present disclosure;
[0020] Figure 2 A flowchart of a node isolation method provided by an embodiment of the present disclosure;
[0021] Figure 3 A flowchart of a node isolation method provided by an embodiment of the present disclosure;
[0022] Figure 4 A flowchart of a method for isolating an object storage host provided by an embodiment of the present disclosure;
[0023] Figure 5 A schematic diagram of a flow chart of a method for determining an isolated node provided in an embodiment of the present disclosure;
[0024] Figure 6 A schematic diagram of the structure of a node isolation device provided in an embodiment of the present disclosure;
[0025] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;
[0026] Figure 8 A schematic diagram of the chip structure provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0027] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
[0028] When a storage node in a Ceph-based object storage system has a hardware failure, it will cause the backend storage service OSD to respond slowly, thus affecting customer services. Regarding hardware failures, the relevant technology can be used to delete the OSD at the cluster level. The implementation method is to execute ceph osd out on any node in the cluster. <osdid>.
[0029] However, this method will cause placement group (PG) migration when deleting OSD. When the cluster to which the storage node belongs is a meta cluster, PG migration may cause the alarm message "large omap objects" to appear, affecting business latency. When the cluster to which the storage node belongs is a data cluster, PG migration may cause the maximum OSD capacity to increase. When too many tasks are processed simultaneously in the cluster, there may be a risk of cluster read-only.
[0030] Therefore, in order to solve the above technical problems, the present invention proposes a node isolation method, which can avoid PG migration and increase of maximum OSD capacity by stopping the OSD process of the faulty node and not deleting the OSD.
[0031] Figure 1 A schematic diagram of a node isolation method provided by an embodiment of the present disclosure. Figure 1 As shown, the method may include the following steps.
[0032] Step 101: determine the score of the faulty node according to the fault information of the faulty node.
[0033] In some embodiments, determining the score of the faulty node according to the fault information of the faulty node includes:
[0034] Get the address information of the faulty node;
[0035] Obtaining fault information of the faulty node according to address information of the faulty node;
[0036] The score of the faulty node is determined according to the fault information of the faulty node.
[0037] In some embodiments, the address information of the faulty node can be obtained through an out-of-band alarm platform, wherein the faulty node can be a node with alarm information, that is, the node with alarm information can be obtained from the out-of-band alarm platform as a faulty node. By way of example, the address information can be an Internet Protocol (IP) address.
[0038] In some embodiments, the fault information of the faulty node may be further obtained based on the IP address of the faulty node. For example, the fault information may include error information of memory, mainboard, CPU, disk, etc.
[0039] In some embodiments, different scores can be set for different fault information, and the scores of fault information on a node can be superimposed. For example, the score of fault A is 2, the score of fault B is 5, and the score of fault C is 1. When there are faults A and B on the first node, the score of the first node is 7, and there are faults A and C on the second node, the score of the second node is 3.
[0040] Step 102: determine the faulty nodes whose scores are greater than or equal to the score threshold as nodes to be processed.
[0041] In some embodiments, a faulty node with a score greater than or equal to a score threshold may be determined as a node to be processed. For example, in the example of step 101 above, when the score threshold is 6, the first node may be determined as a node to be processed.
[0042] In other words, different scores can be set for different fault information. For example, a higher score can be set for more urgent and serious fault information, and a lower score can be set for fault information with less impact. The higher the score of the fault node, the more or more serious the faults of the node are, which may affect the normal operation of the node and the cluster. At this time, the node can be processed first, that is, the fault node with a score greater than or equal to the score threshold can be determined as the node to be processed, so as to give priority to the processing of the node to be processed and ensure the normal operation of the cluster business.
[0043] In some embodiments, the score threshold may be set according to actual applications, which is not limited by the present disclosure.
[0044] Step 103: determine the status of the node to be processed.
[0045] In some embodiments, the state of the node to be processed may be judged, and judging the state of the node may determine whether the node can perform an isolation operation.
[0046] In some embodiments, the status of the node may include: the status of the placement group PG related to the resource pool where the object storage daemon OSD of the node to be processed is located; the status of the PG related to the OSD of the node to be processed; and the number status of the nodes to be processed.
[0047] Step 104: When the state of the node to be processed meets the first condition, an isolation operation is performed on the node to be processed.
[0048] In some embodiments, the first condition includes at least one of the following:
[0049] The first state is the active state;
[0050] The second state is a non-progressive state;
[0051] The quantity status is: the number of active OSDs related to PG is greater than the minimum number of replicas in the resource pool.
[0052] Among them, the first state can be the state of the placement group PG related to the resource pool where the object storage daemon OSD of the node to be processed is located; the second state can be the state of the PG related to the OSD of the node to be processed.
[0053] In some embodiments, performing isolation operations on the node to be processed includes: stopping all OSDs in UP state on the node to be processed; shutting down the boot-up self-starting service of the OSDs in UP state; stopping all monitoring processes MON and / or management processes MGR in UP state on the node to be processed; shutting down the boot-up self-starting service of the monitoring processes MON and / or management processes MGR in UP state.
[0054] In some embodiments, when the number of active OSDs related to PG is greater than the minimum number of copies of the resource pool, the resource pool can operate normally, wherein the number of active OSDs related to PG at this time can be the number of active OSDs related to PG remaining on the node after the OSD on the node is stopped. For example, the minimum number of copies of the resource pool is 2, and there are three active OSDs related to PG on the node to be processed, of which one OSD is on the node to be processed and two OSDs are on other storage nodes. When the node to be processed is stopped, the number of active OSDs related to PG is 2, which meets the minimum number of copies of the resource pool and the resource pool can still operate normally. At this time, the node to be processed can be isolated.
[0055] In summary, the above-mentioned embodiments of the present application determine the node to be processed according to the score of the faulty node, judge the status of the node to be processed, and isolate the node when the first condition is met, so as to handle the hardware failure. By isolating the node, business delays caused by PG migration, cluster read-only problems and other problems can be avoided, and the stability of cluster operation and maintenance can be improved.
[0056] Figure 2 A schematic diagram of a node isolation method provided by an embodiment of the present disclosure. Figure 2 As shown, the method may include the following steps.
[0057] Step 201, determine that the state of the placement group PG related to the resource pool where the object storage daemon OSD of the node to be processed is located is a first state.
[0058] In some embodiments, before determining that the state of the placement group PG related to the resource pool where the object storage daemon OSD of the node to be processed is located is the first state, the method further includes:
[0059] Determine the fault domain where the node to be processed is located;
[0060] According to the domain name of the fault domain, determine the resource pool where the OSD of the node to be processed is located;
[0061] Determine the placement group PG associated with the resource pool.
[0062] In some embodiments, the fault domain where the node to be processed is located can be determined, and according to the name of the fault domain domain, a resource pool related to the node to be processed, that is, the resource pool where the node to be processed is located can be determined.
[0063] In some embodiments, the placement group PG related to the resource pool may be determined, and the state of the related placement group PG may be determined. The state of the related placement group PG may be a first state. For example, the first state may be an active state or an inactive state.
[0064] For example, there may be multiple placement groups associated with the resource pool. When all the placement groups PG associated with the resource pool are in active state, the first state may be determined to be the active state. When some of the placement groups PG are in inactive state, the first state may be determined to be the inactive state.
[0065] For example, the active state may be active, and the inactive state may be, for example, an unknown state (unknown), a degraded state, and the like.
[0066] In some embodiments, when the status of the placement group PG is inactive, it means that the OSD related to the PG is in a stopped state. At this time, there is no need to additionally stop the OSD service of the node to be processed, that is, the isolation condition is not met.
[0067] Step 202, determine that the state of the PG related to the OSD of the node to be processed is the second state.
[0068] In some embodiments, determining that the state of a PG associated with the object storage daemon OSD of the node to be processed is the second state includes:
[0069] Determine the current working status of the cluster, including the nodes to be processed;
[0070] When the current working state of the cluster is to create a resource pool or delete a resource pool, the second state is a progressing state; otherwise, the second state is a non-progressing state.
[0071] In other words, when the above step 201 determines that the first state is an active state, it indicates that the placement group PG of the node to be processed is also in an active state. At this time, the working content of the cluster where the node to be processed is located can be judged. First, the cluster to which the node to be processed belongs can be determined, and the current working state of the cluster can be determined. When the cluster is creating or deleting a resource pool, it indicates that the second state is in progress. When the cluster is not creating or deleting a resource pool, it indicates that the second state is not in progress.
[0072] For example, when the cluster is creating or deleting a resource pool, isolating the node to be processed may affect the cluster business. Therefore, when the cluster is creating or deleting a resource pool, the node to be processed does not meet the isolation condition.
[0073] Step 203, determining the quantity status of the nodes to be processed.
[0074] In some embodiments, the quantity status of the node to be processed can be determined. The quantity status can be the relationship between the number of active OSDs related to the placement group PG on the node to be processed and the minimum number of copies in the resource pool after stopping the OSD on the node to be processed.
[0075] In some embodiments, when the number of active OSDs related to a PG is less than the minimum number of copies of the resource pool, the resource pool cannot operate normally. At this time, the processing node is not isolated to avoid affecting the normal operation of the business.
[0076] In some embodiments, when the first state is an active state, the second state is a non-progressing state, and the quantity state is that the number of active OSDs related to the PG is greater than the minimum number of copies of the resource pool, the node to be processed can be isolated. When the first condition is met, isolating the node to be processed does not affect the business operation of the cluster and the resource pool.
[0077] In summary, the above-mentioned embodiments of the present application can determine whether the nodes to be processed can be isolated by determining the status of the nodes to be processed, and can ensure that the nodes are isolated and hardware failures are handled without affecting business operations. By isolating the nodes to be processed, business delays caused by PG migration, cluster read-only problems, etc. can be avoided, and the stability of cluster operation and maintenance can be improved.
[0078] Figure 3 A schematic diagram of a node isolation method provided by an embodiment of the present disclosure. Figure 3 As shown, the method may include the following steps.
[0079] Step 301, stop all OSDs in UP state on the node to be processed.
[0080] In some embodiments, when the node to be processed meets the first condition, an isolation operation may be performed on the node to be processed. The specific contents of the isolation operation are as follows.
[0081] Specifically, all OSD processes in UP state on the node to be processed can be determined, where the UP state can indicate that the OSD process is in normal operation. When the OSD process is not in the UP state, it means that the OSD has stopped running, so there is no need to isolate the OSD in the non-UP state.
[0082] For example, the systemctl command may be used to search for all OSD processes in the UP state on the node to be processed.
[0083] For example, a stop (STOP) operation may be performed on all OSD processes in the UP state on the above-mentioned found node to be processed, so that the OSD process is closed to isolate the node to be processed.
[0084] Step 302, shut down the boot-up self-start service of the OSD in the UP state.
[0085] In some embodiments, after stopping the OSD process on the node to be processed in step 301, the boot-up self-start service of the OSD in the UP state on the node to be processed can be further turned off to avoid the impact caused by abnormal restart of the host.
[0086] Step 303: Stop all monitoring processes MON and / or management processes MGR in UP state on the node to be processed.
[0087] In some embodiments, performing the isolation operation on the node to be processed further includes stopping all monitoring processes MON and / or management processes MGR in UP state on the node to be processed.
[0088] For example, the systemctl command may be used to search for all MON processes and / or MGR processes in the UP state on the node to be processed.
[0089] For example, all MON processes and / or MGR processes in the UP state on the nodes to be processed found above may be stopped to isolate the nodes to be processed.
[0090] Step 304: shut down the startup service of the monitoring process MON and / or the management process MGR in the UP state.
[0091] In some embodiments, after stopping the monitoring process MON and / or the management process MGR on the node to be processed through step 303, the boot-up self-starting service of the monitoring process MON and / or the management process MGR in the UP state on the above-mentioned node to be processed can be further shut down to avoid the impact caused by abnormal restart of the host.
[0092] In summary, the above-mentioned embodiments of the present application can implement the processing of hardware failures by performing isolation operations on the processing nodes, avoid business delays and cluster read-only problems caused by PG migration, and improve the stability of group operation.
[0093] For the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously.
[0094] Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.
[0095] The technical solution of the present disclosure is further described in detail below in conjunction with specific application examples.
[0096] An embodiment of the present disclosure provides a method for isolating an object storage host.
[0097] like Figure 4 As shown, the specific content of the method is as follows.
[0098] Step 1: Identify whether the node needs to be isolated.
[0099] The system connects to the out-of-band alarm platform to obtain the address information of the nodes that need to be isolated.
[0100] Step 2: Get parameter information of the node to be isolated.
[0101] According to the node address information (IP), the system further obtains the fault information corresponding to the node that needs to be isolated, including: memory, motherboard, CPU, disk and other error information.
[0102] Step 3: Based on the acquired fault information, evaluate whether the fault information of the node may cause the system to restart frequently. If so, perform isolation operations; if not, stop the service.
[0103] Evaluation criteria may include:
[0104] (1) Set different fault scoring standards for different fault information.
[0105] (2) For superimposed faults, the scoring criteria can be superimposed.
[0106] (3) Isolation scores are set. If the score is exceeded, isolation will be implemented; if the score is not exceeded, service will be suspended.
[0107] Step 4: If Figure 5 As shown, further judgment is made on the node that needs to be isolated in step three to determine whether the node can be isolated.
[0108] 1. Determine whether all PGs in the pool where the OSD of the isolated node is located are active. The specific method is:
[0109] (1) Find the fault domain where the node to be isolated is located, and find all pools related to the node based on the fault domain name.
[0110] (2) List the PGs related to the OSD on the node and the pool found in the previous step.
[0111] (3) Determine whether the PGs are active in turn. If all are active, further judgment can be made. Otherwise, the isolation condition is not met.
[0112] 2. Determine whether the OSD in the isolated node up state can be stopped. The specific process is as follows:
[0113] (1) Identify the state of the isolated node. If the current state of the node is a PG in a non-unkonwn state, further perform the judgment of (2).
[0114] (2) Identify whether there is a PG in the node that is not active or in a degraded state. If not, execute the judgment in (3).
[0115] (3) Evaluate the active PGs and determine the current work content of the cluster. If the cluster is not creating or deleting a pool, further execute the judgment in (4).
[0116] (4) Evaluate the active PGs and whether the number of active OSDs associated with the PG is greater than the minimum number of replicas of the pool after stopping the OSDs that need to be stopped. If so, execute (5).
[0117] (5) Perform a stop operation on OSD.
[0118] 3. If the node meets the isolation conditions, it will be isolated as follows:
[0119] (1) OSD services on isolated nodes
[0120] a. Use systemctl to find all OSD processes in the up state on the node.
[0121] b. Stop the OSD processes on the node one by one.
[0122] c. Disable the OSD startup service to avoid further impact caused by abnormal host restart.
[0123] (2) Isolate the mon / mgr service on the node.
[0124] a. Use systemctl to find all mon / mgr processes in the up state on the node.
[0125] b. Stop the mon / mgr process on the node one by one.
[0126] c. Disable the mon / mgr startup service to avoid further impact caused by abnormal host restart.
[0127] Step 5: Isolate nodes that meet the isolation requirements.
[0128] Step 6: Check whether the isolation is successful.
[0129] (1) Check whether the relevant processes on the node are still running
[0130] Use ps to find the corresponding process name and ensure that all corresponding processes on the host are stopped.
[0131] (2) Check whether the service has stopped at the ceph cluster level
[0132] ceph osd tree down to check that the corresponding osd has stopped;
[0133] Use ceph mon dump to check that the corresponding mon / mgr has stopped.
[0134] (3) Check whether the automatic startup service is disabled
[0135] Run systemctl list-unit-files to check that the osd, mon / mgr processes on the isolated node are all in the disabled state.
[0136] In summary, the above example disclosed in the present invention can make further judgments on the nodes that need to be isolated in step three to determine whether the nodes can be isolated. By stopping the OSD process of the isolated node instead of deleting (out) the OSD process of the isolated node, the risk of affecting business delays caused by PG migration and the risk of cluster read-only due to the increase in maximum OSD capacity can be avoided.
[0137] Figure 6 FIG. 6 is a block diagram of a node isolation device 600 provided in an embodiment of the present disclosure. Figure 6 As shown, the device 600 includes: a first processing unit 610, used to determine the score of the faulty node according to the fault information of the faulty node; a second processing unit 620, used to determine the faulty node whose score is greater than or equal to the score threshold as a node to be processed; a third processing unit 630, used to determine the state of the node to be processed; and a fourth processing unit 640, used to perform an isolation operation on the node to be processed when the state of the node to be processed meets the first condition.
[0138] In some embodiments, the first processing unit 610 may also be used to obtain address information of the faulty node; obtain fault information of the faulty node according to the address information of the faulty node; and determine a score of the faulty node according to the fault information of the faulty node.
[0139] In some embodiments, the third processing unit 630 can also be used to determine that the state of the placement group PG related to the resource pool where the object storage daemon OSD of the node to be processed is located is a first state; determine that the state of the PG related to the OSD of the node to be processed is a second state; and determine the number state of the nodes to be processed.
[0140] In some embodiments, the third processing unit 630 can also be used to determine the fault domain where the node to be processed is located; determine the resource pool where the OSD of the node to be processed is located according to the domain name of the fault domain; and determine the PG related to the resource pool.
[0141] In some embodiments, the third processing unit 630 can also be used to determine the current working state of the cluster, where the cluster includes nodes to be processed; when the current working state of the cluster is to create a resource pool or delete a resource pool, the second state is an ongoing state; otherwise, the second state is a non-ongoing state.
[0142] In some embodiments, the first condition includes at least one of the following: the first state is an active state; the second state is a non-progressing state; the quantity state is: the number of active OSDs related to the PG is greater than the minimum number of copies of the resource pool.
[0143] In some embodiments, the fourth processing unit 640 can also be used to stop all OSDs in UP state on the node to be processed; turn off the boot-up self-start service of the OSD in UP state; stop all monitoring processes MON and / or management processes MGR in UP state on the node to be processed; turn off the boot-up self-start service of the monitoring process MON and / or management process MGR in UP state.
[0144] In summary, the above-mentioned node isolation device can handle the hardware failure of the node by isolating the node when the conditions are met according to the fault information and status of the node to be processed. At the same time, it can avoid the risks of business delays and cluster read-only caused by PG migration, and improve the stability of the cluster operation and maintenance process.
[0145] Figure 7 A block diagram of an electronic device 700 for implementing the above method is provided in an embodiment of the present disclosure.
[0146] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present disclosure, the embodiment of the present disclosure also provides an electronic device, such as Figure 7 As shown, the electronic device 700 includes:
[0147] Communication interface 701, capable of exchanging information with other devices;
[0148] The processor 702 is connected to the communication interface 701 to implement information interaction with other devices, and is used to execute the method provided by one or more of the above technical solutions when running the computer program;
[0149] The computer program is stored in the memory 703 .
[0150] Specifically, the processor 702 may be configured to determine a first family to which the first object belongs based on the communication data of the first object; determine the first family as a target family when the first family satisfies a first preset condition; and push target information to the target family.
[0151] It should be noted that the specific processing process of the processor 702 can be understood by referring to the above method.
[0152] Of course, in actual application, the various components in the electronic device 700 are coupled together through the bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 704 .
[0153] The memory 703 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 700. Examples of such data include: any computer program used to operate on the electronic device 700.
[0154] The method disclosed in the above embodiment of the present application can be applied to the processor 702, or implemented by the processor 702. The processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 702. The above-mentioned first processor 702 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 702 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 703, and the processor 702 reads the information in the memory 703 and completes the steps of the above method in combination with its hardware.
[0155] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0156] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by a processor 720 of an electronic device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0157] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the above embodiments of the present disclosure.
[0158] The embodiments of the present disclosure further provide a computer program product, including a computer program, which implements the method described in the above embodiments of the present disclosure when executed by a processor.
[0159] The embodiment of the present disclosure also provides a chip, such as Figure 8 As shown, the chip includes a processor 810 and an interface 820. The number of the processor 810 may be one or more, the number of the interface 820 may be multiple, and the interface circuit is used to receive a signal from the memory of the electronic device and send a signal to the processor, the signal includes a computer instruction stored in the memory, and when the processor executes the computer instruction, the electronic device executes the method described in the above embodiment of the present disclosure.
[0160] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.
[0161] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.
[0162] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.
[0163] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0164] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.
[0165] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.
[0166] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0167] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0168] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0169] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0170] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0171] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent commodity, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0172] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.< / osdid>
Claims
1. A node isolation method, characterized in that: The method comprises: Determine a score of the faulty node according to the fault information of the faulty node; Determine the faulty nodes whose scores are greater than or equal to the score threshold as nodes to be processed; Determine the state of the node to be processed; When the state of the node to be processed meets the first condition, an isolation operation is performed on the node to be processed.
2. The method according to claim 1, characterized in that Determining the score of the faulty node according to the faulty information of the faulty node includes: Obtaining address information of the faulty node; Acquire fault information of the faulty node according to the address information of the faulty node; The score of the faulty node is determined according to the fault information of the faulty node.
3. The method according to claim 1, characterized in that Determining the state of the node to be processed includes: Determine that the state of the placement group PG related to the resource pool where the object storage daemon OSD of the node to be processed is located is a first state; Determine that the state of the PG related to the OSD of the node to be processed is the second state; Determine the quantity status of the nodes to be processed.
4. The method according to claim 3, characterized in that Before determining that the state of the placement group PG related to the resource pool where the object storage daemon OSD of the node to be processed is located is the first state, the method further includes: Determine the fault domain where the node to be processed is located; Determine, according to the domain name of the fault domain, a resource pool where the OSD of the node to be processed is located; Determine the PG associated with the resource pool.
5. The method according to claim 3, characterized in that: Determining that the state of the PG related to the object storage daemon OSD of the node to be processed is the second state includes: Determining a current working state of a cluster, wherein the cluster includes the node to be processed; When the current working state of the cluster is creating a resource pool or deleting a resource pool, the second state is an ongoing state; otherwise, the second state is a non-ongoing state.
6. The method according to claim 5, characterized in that The first condition includes at least one of the following: The first state is an active state; The second state is a non-progressing state; The quantity status is: the number of active OSDs related to the PG is greater than the minimum number of copies of the resource pool.
7. The method according to claim 3, characterized in that The performing of the isolation operation on the node to be processed includes: Stop all OSDs in UP state on the node to be processed; Disable the startup service of the OSD in the UP state; Stop all monitoring processes MON and / or management processes MGR in UP state on the node to be processed; Shut down the startup self-start service of the monitoring process MON and / or the management process MGR in the UP state.
8. A node isolation device, characterized in that: The device includes: A first processing unit, configured to determine a score of the faulty node according to fault information of the faulty node; A second processing unit, configured to determine a faulty node having a score greater than or equal to a score threshold as a node to be processed; A third processing unit, used to determine the state of the node to be processed; The fourth processing unit is configured to perform an isolation operation on the node to be processed when the state of the node to be processed satisfies the first condition.
9. An electronic device, characterized in that: include: one or more processors; A storage device communicatively connected to the one or more processors, on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.
10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.
