Operation and maintenance method and apparatus for abnormal machine, and electronic device and storage medium

Abstract

Provided in the present disclosure are an operation and maintenance method and apparatus for an abnormal machine, and an electronic device and a storage medium. The method comprises: for a target machine in an abnormal state, performing a check on the target machine before the target machine goes offline, wherein the check before the target machine goes offline is used for ensuring that a cluster to which the target machine belongs operates in a stable manner after the target machine goes offline; and when the target machine meets the requirements for the check before the target machine goes offline, if the target machine further meets automatic offline requirements, controlling the target machine to automatically go offline, wherein the automatic offline requirements are related to the number of machines in the cluster that go offline within a first preset time period.

Description

Operation and maintenance method, device, electronic device and storage medium for abnormal machine

[0001] Cross-reference

[0002] This disclosure claims priority to Chinese patent application number 2023111323848 filed with the Patent Office of China on September 4, 2023, entitled “Operation and maintenance method, device, electronic device and storage medium for abnormal machines”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the field of operation and maintenance technology, and in particular to an operation and maintenance method, device, electronic device, and storage medium for abnormal machines. Background Art

[0004] There are a huge number of machines running on the cloud platform, and these machines experience various anomalies every day, such as central processing unit (CPU) failures, memory failures, network connection failures, etc., which cause the services on the machines to not run stably. Operation and maintenance operations are required for abnormal machines, such as taking the abnormal machine offline or replacing it, repairing the abnormal machine, and bringing the repaired abnormal machine back online. In a related technology, after detecting a machine anomaly, a machine offline or replacement workflow is triggered. This workflow requires manual approval by an approver to obtain an approval result. This manual approval is time-consuming, and the approval result depends on the approver's experience. Incorrect approval results cause some machines to be mistakenly offline, which in turn causes the cluster where the offline machine is located to not run stably, affecting the stability of the services on the machine.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide an operation and maintenance method, device, electronic device, and storage medium for an abnormal machine to solve problems existing in the related art.

[0007] In a first aspect, an embodiment of the present disclosure provides an operation and maintenance method for an abnormal machine, comprising: for a target machine in an abnormal state, performing a pre-offline inspection on the target machine, wherein the pre-offline inspection is used to ensure that the cluster to which the target machine belongs runs stably after the target machine goes offline; if the target machine meets the requirements of the pre-offline inspection, if the target machine also meets the automatic offline requirement, then controlling the target machine to automatically go offline, wherein the automatic offline requirement is related to the number of machines that go offline in the cluster within a first preset time period.

[0008] In a second aspect, an embodiment of the present disclosure provides an operation and maintenance device for an abnormal machine, comprising: an inspection unit, configured to perform a pre-offline inspection on a target machine in an abnormal state, wherein the pre-offline inspection is used to ensure that the cluster to which the target machine belongs runs stably after the target machine goes offline; an offline unit, configured to control the target machine to automatically go offline if the target machine meets the pre-offline inspection requirements and also meets the automatic offline requirement, wherein the automatic offline requirement is related to the number of machines that go offline in the cluster within a first preset time period.

[0009] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above-described methods when executing the computer program.

[0010] In a fourth aspect, the present disclosure provides a computer program product, comprising a computer program, which implements any of the above methods when executed by a processor.

[0011] In a fifth aspect, the present disclosure provides a computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the methods described above.

[0012] In a sixth aspect, the present disclosure provides a computer program, which implements any of the above methods when executed by a processor.

[0013] In a seventh aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any of the methods described above.

[0014] Compared with the prior art, the present disclosure has the following advantages:

[0015] According to the embodiments of the present disclosure, a pre-offline inspection is performed on a target machine in an abnormal state before it goes offline, which can ensure that the cluster to which the target machine belongs can run stably after the target machine goes offline. If the target machine also meets the automatic offline requirements, the target machine can be automatically offline without manual approval. While ensuring the safe offline of the target machine, it also improves the offline rate of the target machine and ensures the stable operation of the service.

[0016] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0018] FIG1 is a flow chart illustrating an operation and maintenance solution for an abnormal machine according to an embodiment of the present disclosure;

[0019] FIG2 is a flow chart illustrating an operation and maintenance method for an abnormal machine according to an embodiment of the present disclosure;

[0020] FIG3 is a structural block diagram illustrating an operation and maintenance device for an abnormal machine according to an embodiment of the present disclosure; and

[0021] FIG4 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure;

[0022] FIG5 is a schematic diagram showing the structure of a processor according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present disclosure. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0024] To facilitate understanding of the technical solutions of the embodiments of the present disclosure, the following describes the related technologies of the embodiments of the present disclosure. The following related technologies are optional solutions that can be combined with the technical solutions of the embodiments of the present disclosure in any way, and all of them fall within the scope of protection of the embodiments of the present disclosure.

[0025] In a related technology prior to the present disclosure, to avoid omissions when collecting exception information, all exception information for machines on cloud platforms was collected through a full collection method. This method collects all exception information for a machine from the beginning of record collection to the present. This full collection method takes a long time to collect exception information, and the collection interval is very long, resulting in low collection efficiency. After a machine exception occurs, it may take several hours for subsequent operation and maintenance processes to be triggered, affecting the quality of service provided. After the exception information is collected, a machine offline or machine replacement workflow is generated for the abnormal machine. To ensure the safety of the machine offline or replacement, this workflow requires manual approval. On the one hand, the approver may not be able to process the approval immediately, resulting in a long approval process. On the other hand, the approver needs to judge whether to approve the approval based on their own experience. Incorrect approval results may cause some machines to be accidentally offline, which in turn causes the cluster where the offlined machines are located to become unstable, affecting service stability. Machine operation and maintenance workflows include machine offline workflows, machine online workflows, and machine replacement workflows. These workflows do not support concurrent execution at the cluster level. If a workflow in a cluster is not completed, subsequent workflows will be blocked, resulting in poor workflow execution efficiency. The machine offline or replacement workflow takes a long time to execute. Only after the machine is offline or replaced can the service on the abnormal machine stop running, and the speed of abnormal resolution is slow. The processing of abnormal machines involves many links (for example, abnormal information collection, triggering machine offline or machine replacement workflows, approving workflows, executing workflows, and bringing the machine back online, etc.), and each link involves multiple different steps. Operations and maintenance personnel need to spend a certain amount of time every day to confirm whether there are any tasks that have failed to execute or are blocked (execution timed out), which wastes manpower costs and time.

[0026] The embodiment of the present disclosure provides an operation and maintenance solution for abnormal machines to fully or partially solve the problems existing in the above-mentioned related technologies. In order to more clearly demonstrate the operation and maintenance solution for abnormal machines provided in the embodiment of the present disclosure, the operation and maintenance solution for abnormal machines is described below in conjunction with Figure 1. Figure 1 is a flow chart showing the operation and maintenance solution for abnormal machines in an embodiment of the present disclosure. As shown in Figure 1, first enter step S101 to perform incremental abnormal information collection on the machines running in the cluster to determine the target machine in the abnormal state. Before the target machine goes offline, enter step S102 to determine whether the target machine meets the approval-free conditions. The target machine that meets the approval-free conditions can go offline without manual approval, and the target machine that meets the approval-free conditions will not affect the normal operation of the cluster to which it belongs after going offline. The target machine that does not meet the approval-free conditions enters step S103 and is manually approved to go offline. Furthermore, after a target machine that meets the approval-free conditions is offline, the system proceeds to step S104 to determine whether the abnormality level of the target machine meeting the approval-free conditions is severe. If it is severe, the system proceeds to step S105 to quickly black out the target machine, effectively stopping it from accepting service requests. The system then proceeds to step S106 to offline the target machine. After the target machine is offline, the system proceeds to steps S107, S108, and S109 to wait for the target machine to be repaired. The repaired target machine is then cloned (i.e., the system is reinstalled) and brought back online.

[0027] An embodiment of the present disclosure provides an operation and maintenance method 200 for an abnormal machine to fully or partially solve the problems existing in the above-mentioned related technologies. The method is applied to electronic devices, which may be electronic devices such as servers, computers, mobile phones, and smart watches. The operation and maintenance method 200 for an abnormal machine disclosed in the present disclosure will be described below with reference to Figure 2. Figure 2 is a flowchart showing an operation and maintenance method for an abnormal machine according to an embodiment of the present disclosure. As shown in Figure 2, the operation and maintenance method 200 for an abnormal machine includes the following steps S201 to S202. Steps S201 to S202 will be described below in conjunction with specific embodiments.

[0028] First, proceed to step S201. In step S201, for a target machine in an abnormal state, a pre-offline check is performed on the target machine, and the pre-offline check is used to ensure that the cluster to which the target machine belongs runs stably after the target machine goes offline.

[0029] In one embodiment, the operation and maintenance method 200 for abnormal machines further includes: collecting abnormal information of the machines running in the cluster within a third preset time period every second preset time period; and determining the target machine based on the abnormal information.

[0030] The third preset time period can be a period of time from the current time (i.e., the most recent period of time). By collecting abnormal information that occurs within the third preset time period every second preset time period, the abnormal information of the most recent period of time can be continuously obtained, and the abnormal information of the machines running in the cluster can be collected through incremental collection.

[0031] The present disclosure collects abnormal information from machines running in a cluster through incremental collection. It can collect abnormal information in multiple clusters at the same time, thereby improving the concurrency of abnormal information collection. By configuring abnormal information collection strategies of multiple dimensions, omissions in the collection of abnormal information can be avoided. First, a basic abnormal information collection strategy can be configured. For example, the second preset time period is configured to be 1 minute, and the third preset time period is configured to be the last 5 minutes. Through this basic abnormal information collection strategy, abnormal information collection can be initiated every 1 minute, and only abnormal information that occurred in the last 5 minutes is collected each time. Based on this, abnormal information in the cluster can be quickly collected, and the possibility of omissions in abnormal information collection is low. Only when abnormal information collection fails for 5 consecutive times, omissions in abnormal information collection may occur. On the basis of the basic abnormal information collection strategy, a backup abnormal information collection strategy can be configured. For example, the second preset time period is configured to be 1 hour, and the third preset time period is configured to be the last 5 hours. Through this backup abnormal information collection strategy, abnormal information collection can be initiated every 1 hour, and abnormal information that occurred in the last 5 hours is collected each time. In addition, full abnormal information collection can be performed every day. Collecting exception information across multiple time dimensions can improve the efficiency of exception information collection and effectively prevent omissions in the collection of exception information. The above description of the time intervals in the exception information collection strategy and the length of time for collecting exception messages are merely exemplary. Those skilled in the art may select the time intervals in the exception information collection strategy and the length of time for collecting exception messages based on actual circumstances, as long as the principles of this embodiment are implemented.

[0032] According to the embodiments of the present disclosure, by collecting machine abnormal information in an incremental manner, the time for collecting abnormal information can be reduced and the collection efficiency can be improved. By configuring abnormal information collection strategies in multiple dimensions and collecting abnormal information in multiple dimensions, omissions can be effectively prevented during abnormal information collection.

[0033] The collected exception information can be used to identify target machines with abnormal conditions, such as those experiencing CPU failure, memory failure, network connection failure, or flash memory (PCIE-Flash) card failure. After identifying the target machine, it is necessary to take it offline for maintenance. Before taking it offline, a pre-download inspection is performed to ensure that the cluster to which the target machine belongs can operate stably after the target machine is offline.

[0034] In one embodiment, the pre-offline check includes a correctness check and a numerical check. For a target machine in an abnormal state, the pre-offline check of the target machine includes: checking whether the offline workflow set for the target machine is correct; and / or checking whether the resource values ​​of the cluster meet the requirements.

[0035] The check before going offline can include a numerical check and a correctness check. The numerical check can be used to check whether the resource values ​​of the cluster meet the requirements. The resource values ​​of the cluster can be resource values ​​such as the storage water level, the number of instances bound to each service, and the number of instances bound to the load balancing device. For example, the numerical check can include checking whether the storage water level of the cluster (the ratio of used hard disk capacity to total hard disk capacity) meets the requirements. If the storage water level of the cluster is higher than the storage water level threshold, then the offline of the target machine may cause the storage of the cluster to be overloaded, and the check will fail. The numerical check can also include checking whether the number of instances bound to each service in the cluster meets the requirements. If the number of instances bound to each service is less than the corresponding service instance number threshold, then the offline of the target machine may cause the number of instances bound to each service in the cluster to be less than the minimum number of instances required for the normal operation of each service, thereby causing some services to not operate normally, and the check will fail. The numerical check can also include checking whether the number of instances bound to the load balancing device in the cluster meets the requirements. If the number of instances bound to the load balancing device is less than the load balancing device instance number threshold, the target machine's offline state may cause the number of instances bound to the load balancing device to be less than the minimum number of instances required for the normal operation of the load balancing device. This may cause the load balancing device to be unable to distribute service requests received from the cluster to each machine, and the check will fail.

[0036] The correctness check can be used to check whether the offline workflow set for the target machine is correct. The offline workflow set for the target machine includes information such as the identifier of the target machine, the identifier of the cluster to which the target machine belongs, and the identifier of the group to which the target machine moves after going offline. The correctness check can include checking whether the target machine in the offline workflow matches the cluster to which the target machine belongs. For example, if the target machine is a machine in cluster A, if the machine offline workflow mistakenly sets the target machine to a machine in cluster B, the target machine and the target cluster do not match, and the check will fail. The correctness check can also include checking whether the group moved by the target machine in the offline workflow meets the requirements. There are multiple groups in a cluster, and the target machine needs to be moved to a group specifically for offline machines after going offline. If the group moved by the target machine is not a group specifically for offline machines, the check will fail.

[0037] In one embodiment, the pre-offline check may further include checking whether the target machine is in another machine operation and maintenance workflow. If the target machine is in another machine operation and maintenance workflow (e.g., a machine online workflow), the check fails.

[0038] In another embodiment, the pre-delisting check may also include checking whether the target cluster has any unknown services deployed. Because unknown services must be deployed on a certain number of instances to run, shutting down the target machine may cause the unknown services to not run properly. Therefore, if the target cluster has any unknown services deployed, the check will fail.

[0039] If the target machine meets the pre-offline check, it means that the target cluster to which the target machine belongs can run stably after the target machine goes offline.

[0040] Next, the process proceeds to step S202. In step S202, if the target machine meets the pre-offline inspection requirements and also meets the automatic offline requirement, the target machine is controlled to be offline. The automatic offline requirement is related to the number of offline machines in the cluster within a first preset time period.

[0041] In one embodiment, before controlling the target machine to go offline, the operation and maintenance method 200 for an abnormal machine further includes: determining the degree of abnormality of the target machine based on the abnormal information of the target machine, and when the degree of abnormality of the target machine is severe, controlling the target machine to stop receiving service requests, wherein the service request is used to request the target machine to provide a service to the service requester.

[0042] Since the machine offline workflow execution time is relatively long, before the target machine completes the offline process, the service running on the target machine is still receiving service requests. If the exception in the target machine is a serious exception, the target machine cannot provide services to the service requester, affecting the service quality. Therefore, it is necessary to determine the degree of abnormality of the target machine based on the collected abnormal information of the target machine. If the abnormal information of the target machine is abnormal information such as system disk read-only or pcie-flash card failure, the degree of abnormality of the target machine is determined to be a serious abnormality. In the case that the degree of abnormality of the target machine is a serious abnormality, the target machine is controlled to stop receiving service requests to ensure the quality of service provided. For example, when receiving a user's service request, the load balancing device can be controlled to prohibit forwarding the service request to the target machine, and the load balancing device can be controlled to send the service request to a machine that does not have an abnormal state and provides the same service. The machine that does not have an abnormal state provides services to the user to ensure the quality of service provided.

[0043] According to the embodiment of the present disclosure, before controlling the target machine to go offline, if the abnormality existing in the target machine is a serious abnormality, the target machine is controlled to stop receiving service requests, so that the negative impact of the abnormality on the service can be quickly eliminated, thereby ensuring the quality of service provision. In the case that the target machine meets the inspection requirements before going offline, it is also necessary to determine whether the target machine meets the automatic offline requirements. If the target machine meets the automatic offline requirements, the target machine is controlled to go offline so that the target machine can be repaired, that is, the offline workflow of the target machine can be exempted from manual approval to achieve the offline of the target machine; if the target machine does not meet the automatic offline requirements, the offline workflow of the target machine needs to be manually approved, and the offline of the target machine can be achieved only after manual approval. The automatic offline requirement is related to the number of machines that go offline in the cluster within the first preset time period. If the number of machines that go offline in the cluster is too large within a time period, the processing performance of the cluster will be affected. At this time, manual approval of the machine offline process is required to avoid the cluster being unable to operate stably due to the machine going offline by mistake.

[0044] In one embodiment, the automatic offline requirement includes at least one of the following: the number of machines automatically offline within the first preset time period is less than a first quantity threshold, and the total number of machines offline within the first preset time period is less than a second quantity threshold.

[0045] If the number of machines that automatically log off within a first preset time period is less than a first threshold and / or the total number of machines that log off within the first preset time period is less than a second threshold, the target machine meets the automatic logoff requirement. The number of machines that automatically log off is the number of machines that log off without manual approval, and the total number of machines that log off is the sum of the number of machines that automatically log off plus the number of machines that log off with manual approval. For example, the first preset time period can be set to 24 hours, the first threshold to 2, and the second threshold to the smaller of 3% and 30 of the total number of cluster machines. Therefore, if the number of machines that automatically log off within 24 hours is less than 2 and / or the total number of machines that log off within 24 hours is less than the smaller of 3% and 30 of the total number of cluster machines, the target machine meets the automatic logoff requirement. For another example, the first preset time period can be set to 5 minutes, the first threshold to 1, and the second threshold to 2. Therefore, if the number of machines that automatically log off within 5 minutes is less than 1 and / or the total number of machines that log off within 5 minutes is less than 2, the target machine meets the automatic logoff requirement. The above description of the upper limit of the number of machines offline within the first preset time period in the automatic offline requirement is only exemplary. Those skilled in the art can select the first preset time period and the upper limit of the number of machines offline according to actual conditions, as long as the principles of this embodiment can be implemented.

[0046] In one embodiment, if the target machine does not meet the automatic offline requirements, the target machine is transferred to manual approval.

[0047] If the target machine does not meet the automatic offline requirements, the target machine will be transferred to manual approval, and the approver will determine whether to take the target machine offline. When transferring the target machine to manual approval, the approver can use the target machine's pre-offline inspection and machine processing information (for example, the results of the above inspections, the number of machines offline in the past 5 minutes, the number of machines offline in the past 24 hours, storage water level information, etc.) as a reference to determine whether to take the target machine offline. This can improve the accuracy of the approval process and reduce the burden on the approver.

[0048] In one embodiment, the operation and maintenance method 200 for an abnormal machine further includes: upon confirming that the maintenance of the target machine is completed, controlling the target machine that has completed the maintenance to go online.

[0049] After the target machine is offline, it can be repaired. After the repair is completed, the target machine can be cloned (ie, the system is reinstalled). After the cloning, the target machine can be put back online to provide corresponding services.

[0050] In one embodiment, after obtaining the target machine in an abnormal state, the machine replacement workflow can also be triggered to replace the target machine. There are various types of machines in the cloud platform. An Admin gateway can be deployed on the cloud platform, and any machine in the cluster can be logged in through the Admin gateway. The cloud platform can also be deployed with machines for the control plane (used to manage cloud platform resources) and machines for the data plane (used to run actual services). The management gateway and the Admin plane machines are more important than the data plane machines. Therefore, when the management gateway and the Admin plane machines are in an abnormal state, the machine replacement workflow can be triggered to replace the management gateway and the Admin plane machines, thereby avoiding the system from stopping when the management gateway and the Admin plane machines go offline. The machine replacement workflow is equivalent to the machine offline workflow plus the machine online workflow. Machine replacement is to complete the machine online at the same time as the machine goes offline.

[0051] In one embodiment, controlling the target machine to go offline and controlling the target machine to go online after maintenance is completed include multiple non-thread-safe sub-steps, and the operation and maintenance method 200 for abnormal machines also includes: before executing the target sub-step among the multiple non-thread-safe sub-steps, setting a mutex lock for the target sub-step; after the target sub-step is executed, releasing the mutex lock.

[0052] Controlling the target machine offline and controlling the target machine back online after maintenance is complete includes multiple sub-steps, which can be executed concurrently to improve execution efficiency. However, some of these sub-steps are non-thread-safe (i.e., concurrent execution will destroy data consistency). Therefore, before executing the target sub-step among the multiple non-thread-safe sub-steps, it is necessary to set a mutex lock for the target sub-step to prevent one process from executing the target sub-step while other processes are also executing the target sub-step, resulting in data consistency being destroyed. After the target sub-step is executed, the mutex lock is released, allowing other processes to execute the target sub-step, thus ensuring data consistency.

[0053] According to the embodiments of the present disclosure, by concurrently executing sub-steps in the machine offline and machine online workflows, the execution efficiency of machine online and offline can be improved, and mutual exclusion locks are set for non-thread-safe sub-steps to ensure data consistency during concurrent execution.

[0054] In one embodiment, the operation and maintenance method 200 for an abnormal machine also includes: setting an alarm strategy based on the type of the abnormality and the cluster, and issuing an alarm based on the alarm strategy, the alarm strategy including an alarm cycle and an alarm method; and issuing an alarm for steps that fail to execute or steps that time out.

[0055] Based on the type of anomaly present on the target machine and the cluster to which it belongs, you can set a corresponding alert policy. This policy includes the recipient of the alert message (i.e., the operations and maintenance personnel for the cluster to which the target machine belongs). You can also configure the alert method based on the severity of the anomaly. For example, for minor anomalies, an alert message can be sent to the operations and maintenance personnel for the cluster via email; for severe anomalies, an alert message can be sent to the operations and maintenance personnel for the cluster via instant messaging or phone. You can also configure the alert interval based on the severity of the anomaly. For example, for minor anomalies, the alert interval can be set to every 300 minutes; for severe anomalies, the alert interval can be set to every 60 minutes. The alert message can include the IP address of the anomaly machine, the cluster to which the anomaly machine belongs, the anomaly type, the time the anomaly was reported, and whether the anomaly machine meets the conditions for automatic offline. The alert message allows operations and maintenance personnel to understand the type and time of the machine anomaly, as well as the subsequent steps for offline operation.

[0056] Handling abnormal machines involves various stages (e.g., collecting abnormal information, triggering machine offline or replacement workflows, approving workflows, executing workflows, etc.), each of which involves many different steps. If these steps fail or time out, alerts are generated to notify operations and maintenance personnel to intervene.

[0057] In one embodiment, the alarming of the failed execution step or the timed-out step includes: sending an alarm message to the operation and maintenance end via instant messaging software, wherein the alarm message includes location information of the failed execution step or the timed-out step.

[0058] After receiving the alarm message through instant messaging software, the operation and maintenance personnel on the operation and maintenance side can jump to the step for investigation and processing according to the location information of the failed step or the timed-out step (for example, the link of the step).

[0059] According to the embodiments of the present disclosure, by introducing full-link monitoring and alarming, operation and maintenance personnel can automatically obtain all steps that have failed or timed out based on instant messaging software. Operation and maintenance personnel no longer need to manually check whether each step has been successfully executed or whether a timeout has occurred, thereby improving the efficiency of monitoring and alarming.

[0060] In one embodiment, a bot is integrated into instant messaging software. By communicating with the bot, operators can obtain key operational information and control the bot to perform simple operations. The bot is a background program responsible for parsing user information, analyzing and matching user intent, and executing related operations.

[0061] Based on the embodiments of the present disclosure, by introducing a real-time chat-driven operation and maintenance model (ChatOps) capability and by implanting robots into chat sessions, an automated linkage between operation and maintenance personnel, robots, and data is formed, enabling operation and maintenance personnel to efficiently perform tasks and communicate and collaborate.

[0062] In one embodiment, operational efficiency is improved by streamlining unnecessary operational steps and adjusting the order in which these steps are executed. Some unnecessary steps can be deleted, for example, the step of checking whether the target machine's account exists. Important steps can also be moved forward, while time-consuming steps can be moved backward. For example, the step of controlling the target machine to stop receiving service requests can be moved forward, while the step of binding / unbinding the load balancing device's virtual IP address can be moved backward, further improving operational efficiency.

[0063] Corresponding to the application scenario and method of the method provided in the embodiment of the present disclosure, the embodiment of the present disclosure also provides an operation and maintenance device 300 for an abnormal machine, which is deployed on an electronic device, which may be an electronic device such as a server, a computer, a mobile phone, or a smart watch. The operation and maintenance device 300 for an abnormal machine disclosed in the present disclosure will be described below with reference to FIG3 . FIG3 is a structural block diagram showing an operation and maintenance device for an abnormal machine according to an embodiment of the present disclosure. As shown in FIG3 , the operation and maintenance device 300 for an abnormal machine may include: an inspection unit 301 and an offline unit 302.

[0064] The inspection unit 301 is configured to perform a pre-offline inspection on a target machine in an abnormal state, wherein the pre-offline inspection is used to ensure that the cluster to which the target machine belongs runs stably after the target machine goes offline.

[0065] The inspection unit 301 may be a component in an electronic device, and the electronic device uses the component to inspect the target machine before it goes offline.

[0066] The offline unit 302 is configured to control the target machine to automatically go offline if the target machine meets the pre-offline inspection requirements and also meets the automatic offline requirement, where the automatic offline requirement is related to the number of machines in the cluster that have gone offline within a first preset time period.

[0067] The offline unit 302 may be a component in an electronic device, and the electronic device controls the target machine to go offline through the component.

[0068] In one embodiment, the operation and maintenance device 300 for an abnormal machine also includes a stop receiving service unit, which is configured to determine the degree of abnormality of the target machine based on the abnormal information of the target machine, and when the abnormality of the target machine is a serious abnormality, control the target machine to stop receiving service requests, wherein the service request is used to request the target machine to provide service to the service requester.

[0069] In one embodiment, the operation and maintenance device 300 for abnormal machines also includes an abnormal information collection unit, which is configured to collect abnormal information occurring in the machines running in the cluster within a third preset time period every second preset time period; based on the abnormal information, the target machine is determined, wherein the second preset time period is used to represent the interval length for collecting the abnormal information, and the third preset time period is used to represent a period of time from the current time.

[0070] In one embodiment, the pre-offline check includes a correctness check and a value check, and the checking unit 301 is further configured to check whether the offline workflow set for the target machine is correct; and check whether the resource values ​​of the cluster meet the requirements.

[0071] In one embodiment, the automatic offline requirement includes at least one of the following: the number of machines automatically offline within the first preset time period is less than a first quantity threshold, and the total number of machines offline within the first preset time period is less than a second quantity threshold.

[0072] In one embodiment, the operation and maintenance device 300 for abnormal machines further includes an online unit. After controlling the target machine to go offline, the online unit is configured to control the target machine to go online after confirming that the maintenance of the target machine is completed.

[0073] In one embodiment, controlling the target machine to go offline and controlling the target machine to go online after maintenance is completed include multiple non-thread-safe sub-steps, and the operation and maintenance device 300 for the abnormal machine also includes a safe execution unit, which is configured to set a mutex lock for the target sub-step before executing the target sub-step in the multiple non-thread-safe steps; and release the mutex lock after the target sub-step is executed.

[0074] In one embodiment, the operation and maintenance device 300 for an abnormal machine also includes an alarm unit, which is configured to set an alarm strategy based on the type of the abnormality and the cluster, and to issue an alarm based on the alarm strategy, wherein the alarm strategy includes an alarm cycle and an alarm method; and to issue an alarm for steps that fail to execute or steps that time out.

[0075] In one embodiment, the alarm unit is further configured to send an alarm message to the operation and maintenance end via instant messaging software, wherein the alarm message includes location information of the step that failed to execute or the step that timed out.

[0076] The functions of each module in each device of the embodiment of the present disclosure can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0077] According to the embodiments of the present disclosure, a target machine in an abnormal state is checked before being taken offline, which can ensure that the cluster to which the target machine belongs can run stably after the target machine goes offline. If the target machine also meets the automatic offline requirements, the target machine can be taken offline without manual approval. While ensuring the safe offline of the target machine, it also improves the offline rate of the target machine and ensures the stable operation of the service.

[0078] According to the embodiments of the present disclosure, by collecting machine abnormal information in an incremental collection manner, the time for collecting abnormal information can be reduced and the collection efficiency can be improved. By configuring abnormal information collection strategies in multiple dimensions and collecting abnormal information in multiple dimensions, omissions in abnormal information collection can be effectively avoided.

[0079] According to the embodiment of the present disclosure, before controlling the target machine to go offline, if the abnormality existing in the target machine is a serious abnormality, controlling the target machine to stop receiving service requests can quickly eliminate the negative impact of the abnormality on the service, thereby ensuring the quality of service provided.

[0080] According to the embodiments of the present disclosure, by concurrently executing sub-steps in the machine offline and machine online workflows, the execution efficiency of machine online and offline can be improved, and mutual exclusion locks are set for non-thread-safe sub-steps to ensure data consistency during concurrent execution.

[0081] According to the embodiments of the present disclosure, by introducing full-link monitoring and alarming, operation and maintenance personnel can automatically obtain all steps that have failed or timed out based on instant messaging software. Operation and maintenance personnel no longer need to manually check whether each step has been successfully executed or whether a timeout has occurred, thereby improving the efficiency of monitoring and alarming.

[0082] According to the embodiments of the present disclosure, by introducing ChatOps capabilities and implanting robots into chat sessions, an automated linkage between operation and maintenance personnel, robots, and data is formed, enabling operation and maintenance personnel to efficiently perform tasks and communicate and collaborate.

[0083] Figure 4 is a block diagram of an electronic device used to implement embodiments of the present disclosure. As shown in Figure 4 , the electronic device includes a memory 401 and a processor 402. Memory 401 stores a computer program executable by processor 402. Processor 402 executes the computer program to implement the methods described in the above embodiments. Memory 401 and processor 402 may be present in one or more locations.

[0084] The electronic device also includes:

[0085] The communication interface 403 is used to communicate with external devices and perform data exchange transmission.

[0086] If the memory 401, processor 402, and communication interface 403 are implemented independently, the memory 401, processor 402, and communication interface 403 can be interconnected via a bus and communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG4 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0087] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0088] The present disclosure also provides a processor. FIG5 shows a schematic diagram of the structure of a processor for implementing the present disclosure. As shown in FIG5, the processor 500 is configured to run a program, wherein the program, when executed by the processor, executes the method in the above embodiment.

[0089] In an embodiment of the present disclosure, the processor 500 may execute an operating program of an operation and maintenance method for an abnormal machine.

[0090] Optionally, the processor 1000 can be configured to perform the following steps: for a target machine in an abnormal state, perform a pre-offline check on the target machine, wherein the pre-offline check is used to ensure that the cluster to which the target machine belongs runs stably after the target machine goes offline; if the target machine meets the pre-offline check requirements, if the target machine also meets the automatic offline requirement, control the target machine to automatically go offline, wherein the automatic offline requirement is related to the number of machines that go offline in the cluster within a first preset time period.

[0091] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present disclosure when the program is executed by a processor.

[0092] Computer-readable storage media may also be referred to as computer storage media. They may include data signals transmitted in baseband or as part of a carrier wave, carrying readable program code. Such transmitted data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable storage media may transmit, propagate, or transfer programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0093] The program code contained in the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the foregoing.

[0094] The embodiments of the present disclosure further provide a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method provided in the embodiments of the present disclosure is implemented.

[0095] The embodiments of the present disclosure further provide a computer program, which, when executed by a processor, implements the method provided in the embodiments of the present disclosure.

[0096] Optionally, the computer program implements program code for the following steps when executed by a processor: for a target machine in an abnormal state, performing a pre-offline check on the target machine, wherein the pre-offline check is used to ensure that the cluster to which the target machine belongs runs stably after the target machine goes offline; and when the target machine meets the pre-offline check requirements, if the target machine also meets the automatic offline requirement, controlling the target machine to automatically go offline, wherein the automatic offline requirement is related to the number of machines that have gone offline in the cluster within a first preset time period.

[0097] An embodiment of the present disclosure further provides a chip, which includes a processor for calling and executing instructions stored in a memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present disclosure.

[0098] An embodiment of the present disclosure also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided by the disclosed embodiment.

[0099] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0100] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus random access memory (DR RAM).

[0101] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present disclosure are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0102] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0103] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0104] Any process or method described in the flowchart or otherwise described herein can be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present disclosure includes additional implementations in which the functions may be performed in a different order than shown or discussed, including in a substantially simultaneous manner or in a reverse order depending on the functions involved.

[0105] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with such instruction execution systems, apparatuses or devices.

[0106] It should be understood that various parts of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0107] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0108] The above description is merely an exemplary embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope of the present disclosure, and such modifications or substitutions should be included within the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims. Industrial Applicability

[0109] The solution provided by the embodiments of the present disclosure can be applied to the operation and maintenance process of abnormal machines. For a target machine in an abnormal state, a pre-offline inspection is performed on the target machine. The pre-offline inspection is used to ensure that the cluster to which the target machine belongs operates stably after the target machine goes offline. When the target machine meets the requirements of the pre-offline inspection, if the target machine also meets the automatic offline requirement, the target machine is controlled to automatically go offline. The automatic offline requirement is related to the number of machines that have gone offline in the cluster within a first preset time period, thereby solving the technical problem of a low target machine offline rate.

Claims

1. An operation and maintenance method for an abnormal machine, comprising: For a target machine in an abnormal state, a pre-offline check is performed on the target machine, wherein the pre-offline check is used to ensure that the cluster to which the target machine belongs runs stably after the target machine goes offline; When the target machine meets the pre-offline inspection requirements, if the target machine also meets the automatic offline requirement, the target machine is controlled to automatically go offline, and the automatic offline requirement is related to the number of machines that go offline in the cluster within a first preset time period.

2. The method according to claim 1, wherein: The method further comprises: Every second preset time period, collect exception information occurring in a third preset time period of the machines running in the cluster; and determine the target machine based on the exception information, wherein the second preset time period is used to represent the interval duration for collecting the exception information, and the third preset time period is used to represent a period of time from the current time.

3. The method according to claim 2, wherein: Before controlling the target machine to automatically go offline, the method further includes: The abnormality degree of the target machine is determined according to the abnormality information of the target machine. If the abnormality degree of the target machine is severe, the target machine is controlled to stop receiving service requests, wherein the service request is used to request the target machine to provide a service to a service requester.

4. The method according to claim 1 or 2, wherein: The pre-offline check includes a correctness check and a numerical check. The pre-offline check on the target machine in an abnormal state includes: Check whether the offline workflow set for the target machine is correct; and / or, Check whether the resource values ​​of the cluster meet the requirements.

5. The method according to claim 1 or 2, wherein: The automatic offline requirement includes at least one of the following: the number of machines that automatically go offline within the first preset time period is less than a first quantity threshold, and the total number of machines that go offline within the first preset time period is less than a second quantity threshold.

6. The method according to claim 1 or 2, wherein: After controlling the target machine to automatically go offline, the method further includes: When it is confirmed that the target machine has been repaired, the target machine that has been repaired is controlled to go online.

7. The method according to claim 6, wherein: The controlling the target machine to automatically go offline and the controlling the target machine to go online after the maintenance is completed include multiple non-thread-safe sub-steps, and the method further includes: Before executing a target sub-step among the multiple non-thread-safe sub-steps, setting a mutex lock for the target sub-step; After the target sub-step is executed, the mutex lock is released.

8. The method according to claim 1 or 2, wherein: The method further comprises: According to the type of the abnormality and the cluster, an alarm strategy is set, and an alarm is issued according to the alarm strategy, wherein the alarm strategy includes an alarm period and an alarm mode; Alerts are issued for steps that fail to execute or steps that time out.

9. The method according to claim 8, wherein: The step of issuing an alarm for a failed execution step or a timed-out execution step includes: The alarm information is sent to the operation and maintenance end through the instant messaging software, and the alarm information includes the location information of the step where the execution failed or the step where the execution timed out.

10. The method according to claim 1, wherein: The method further comprises: The pre-offline check also includes checking whether the target machine is in other machine operation and maintenance workflows; If the target machine is in the other machine operation and maintenance workflow, it is determined that the target machine does not meet the pre-offline inspection requirement.

11. The method according to claim 1, wherein: The method further comprises: The pre-offline check also includes checking whether the cluster deploys unknown services; If the cluster has deployed an unknown service, it is determined that the target machine does not meet the requirement of the pre-offline check.

12. The method according to claim 2, wherein: The method further comprises: The exception information occurring on the machines running in the cluster is collected by incremental collection.

13. The method according to claim 2, wherein: The method further comprises: The abnormal information occurring in the machines running in the cluster is collected through an abnormal information collection strategy of multiple dimensions, wherein the abnormal information collection strategy is used to configure the second preset time period and the third preset time period.

14. The method according to claim 4, wherein: The method further comprises: The offline workflow includes at least one of the following: an identifier of the target machine, an identifier of the cluster to which the target machine belongs, and an identifier of the group to which the target machine moves after going offline. The resource values ​​of the cluster include at least one of the following: a storage water level, a number of instances bound to each service, and a number of instances bound to a load balancing device.

15. An operation and maintenance device for an abnormal machine, comprising: An inspection unit is configured to perform a pre-offline inspection on a target machine in an abnormal state, wherein the pre-offline inspection is used to ensure that the cluster to which the target machine belongs runs stably after the target machine goes offline; The offline unit is configured to control the target machine to automatically go offline if the target machine meets the pre-offline inspection requirements and also meets the automatic offline requirement, wherein the automatic offline requirement is related to the number of machines that go offline in the cluster within a first preset time period.

16. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

17. A computer program product, wherein: The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 14.

18. A computer program product, wherein: The invention comprises a non-volatile computer-readable storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 14 when executed by a processor.

19. A computer program, wherein When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

20. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images