Self-healing implementation method and device and electronic equipment

Abstract

The invention relates to a self-healing implementation method and device and electronic equipment, and the method comprises the steps: obtaining an alarm event of a unitized service system, extracting a first event attribute of the alarm event, and obtaining an event strategy set, each event strategy in the event strategy set being configured with a corresponding second event attribute, and under the condition that the first event attribute is matched with the second event attribute, determining a self-healing process corresponding to the target event strategy, and performing self-healing processing on the target business service according to the self-healing process. The method effectively makes up the defect that a traditional unitized architecture lacks a self-healing mechanism, remarkably reduces the dependence of fault processing on manual operation, shortens the fault recovery time, reduces the operation and maintenance cost and pressure, adapts to a complex scene of a privatized environment based on the characteristics of the unitized architecture, greatly improves the system stability and service continuity, and improves the user experience. And a powerful support is provided for reliable operation of enterprise-level applications in a privatized deployment mode.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a self-healing method, apparatus, and electronic device. Background Technology

[0002] In private enterprise collaborative office systems, traditional standard cluster (Kubernetes clusters, cloud-hosted clusters, etc.) architectures, which are based on self-built and self-maintained systems and deployed on physical or virtual machines, offer basic elastic scalability. However, they are complex, have high maintenance costs, and lack automated self-healing capabilities, making it difficult to meet the requirements for high availability and rapid response. As the customer base expands, the maintenance pressure on these deployment models increases dramatically, the MTTR (Mean Time To Repair) fails to meet service targets, and system stability and business continuity face severe challenges.

[0003] While some projects have attempted to address the issue by introducing modular architectures, traditional modular solutions lack self-healing mechanisms and are ill-suited for complex private environments. Therefore, a modular architecture with self-healing capabilities is urgently needed to improve system stability and operational efficiency, supporting the reliable operation of enterprise applications in private environments. Summary of the Invention

[0004] To address the aforementioned technical challenges of needing a self-healing modular architecture to improve system stability and operational efficiency, and to support the reliable operation of enterprise applications in private environments, this application provides a self-healing implementation method, apparatus, electronic device, and storage medium. The specific technical solution is as follows: In a first aspect of this application, a self-healing implementation method is provided, applied to a unitized business system. The unitized business system includes multiple business units, each containing multiple business services, and all business units contain the same business services. The method includes: Acquire alarm events from the unitized business system, wherein the alarm events are used to issue alarms for the target business services of the target business units in the unitized business system; Extract the first event attribute of the alarm event and obtain the event policy set, wherein each event policy in the event policy set is configured with a corresponding second event attribute; If the first event attribute matches the second event attribute, determine the self-healing process corresponding to the target event strategy; The target business service is self-healed according to the self-healing process described above.

[0005] In an optional implementation, extracting the first event attribute of the alarm event includes: The alarm source of the alarm event is determined, and the alarm event is standardized according to the alarm source to obtain a standard structured event; Extract the first event attribute of the standard structured event.

[0006] In one optional implementation, there are multiple alarm sources, and different alarm sources have different priorities; there are also multiple alarm events. The process of determining the alarm source of the alarm event and standardizing the alarm event based on the alarm source to obtain a standard structured event includes: For any of the aforementioned alarm events, determine the alarm source of the alarm event and determine the priority of the alarm source; The alarm events are sorted according to the priority to obtain an alarm event sequence; The alarm events in the alarm event sequence are traversed sequentially, and the alarm events are standardized according to their alarm sources to obtain standard structured events.

[0007] In an optional implementation, before obtaining the event strategy set, the method further includes: Extract the target event attribute from the first event attribute and obtain the configured admission policy, wherein the admission policy includes the admission event attribute; If the target event attribute matches the admission event attribute, the step of obtaining the event strategy set is executed.

[0008] In an optional implementation, each event strategy in the event strategy set is associated with at least two event strategy matching control parameters, which include parameters for determining the strategy selection order and parameters for determining attribute comparison rules. The step of determining the self-healing process corresponding to the target event strategy when the first event attribute matches the second event attribute includes: Event strategies are obtained from the event strategy set according to the parameters associated with the event strategy used to determine the strategy selection order; Based on the parameters associated with the event strategy used to determine the attribute comparison rules, the second event attribute of the event strategy is matched with the first event attribute; If the second event attribute of the event strategy matches the first event attribute, the event strategy is determined as the target event strategy. Determine the self-healing process corresponding to the target event strategy.

[0009] In an optional implementation, the parameter used to determine the strategy selection order is the strategy priority, and different event strategies in the event strategy set have different strategy priorities; The step of obtaining event strategies from the event strategy set according to the parameters associated with the event strategy for determining the strategy selection order includes: Event policies are retrieved from the event policy set according to the policy priority associated with the event policy.

[0010] In an optional implementation, the parameter used to determine the attribute comparison rule is a matching method, and the matching methods are different for different event strategies in the event strategy set; The step of matching the second event attribute of the event strategy with the first event attribute based on the parameters associated with the event strategy for determining the attribute comparison rules includes: Determine the matching method of the event strategy, and according to the type of the matching method, perform a comparison operation between the second event attribute and the first event attribute of the event strategy.

[0011] In one optional implementation, the unitized business system has a corresponding configuration file; The method further includes: pre-assigning a corresponding virtual network identifier for each service in the configuration file.

[0012] In an optional implementation, the method further includes: For any service within any business unit, during the deployment phase of the service, obtain the physical network identifier of the service. Based on the virtual network identifier and physical network identifier of the service, establish corresponding routing rules; The routing rules are synchronously distributed to the remaining business services in the business unit.

[0013] In an optional implementation, the service accesses the remaining service services in the service unit in the following manner: Generate access requests from the business service to other business services, and obtain the domain names corresponding to the other business services in the business unit; The remaining virtual IPs of the other business services corresponding to the domain name are resolved, and the remaining routing rules of the other business services are determined based on the remaining virtual IPs; According to the remaining routing rules, the access requests of the business service to other business services are forwarded to the other business services.

[0014] In a second aspect of this application, a self-healing implementation apparatus is also provided, applied to a unitized business system. The unitized business system includes multiple business units, each business unit contains multiple business services, and the business services contained in each business unit are the same. The apparatus includes: The event acquisition module is used to acquire alarm events of the unitized business system. The alarm events are used to issue alarms for the target business services of the target business units in the unitized business system. The attribute extraction module is used to extract the first event attribute of the alarm event; The strategy set acquisition module is used to acquire an event strategy set, wherein each event strategy in the event strategy set is configured with a corresponding second event attribute. The process determination module is used to determine the self-healing process corresponding to the target event strategy when the first event attribute matches the second event attribute. The self-healing module is used to perform self-healing processing on the target business service according to the self-healing process.

[0015] In an optional implementation, the attribute extraction module specifically includes: The alarm source determination submodule is used to determine the alarm source of the alarm event, and to standardize the alarm event according to the alarm source to obtain a standard structured event; The event attribute extraction submodule is used to extract the first event attribute of the standard structured event.

[0016] In one optional implementation, there are multiple alarm sources, and different alarm sources have different priorities; there are also multiple alarm events. The alarm source determination submodule is specifically used for: For any of the aforementioned alarm events, determine the alarm source of the alarm event and determine the priority of the alarm source; The alarm events are sorted according to the priority to obtain an alarm event sequence; The alarm events in the alarm event sequence are traversed sequentially, and the alarm events are standardized according to their alarm sources to obtain standard structured events.

[0017] In an optional implementation, the apparatus further includes: an admission verification module, configured to extract a target event attribute from the first event attribute and obtain a configured admission policy, the admission policy including admission event attributes; and, if the target event attribute matches the admission event attribute, to execute the step of obtaining the event policy set.

[0018] In an optional implementation, each event strategy in the event strategy set is associated with at least two event strategy matching control parameters, which include parameters for determining the strategy selection order and parameters for determining attribute comparison rules; the process determination module specifically includes: The event strategy acquisition submodule is used to acquire event strategies from the event strategy set according to the parameters associated with the event strategy for determining the strategy selection order. The event attribute matching submodule is used to match the second event attribute of the event strategy with the first event attribute based on the parameters associated with the event strategy for determining the attribute comparison rules. The event strategy determination submodule is used to determine the event strategy as the target event strategy when the second event attribute of the event strategy matches the first event attribute. The process determination submodule is used to determine the self-healing process corresponding to the target event strategy.

[0019] In an optional implementation, the parameter used to determine the strategy selection order is the strategy priority, and different event strategies in the event strategy set have different strategy priorities; the event strategy acquisition submodule is specifically used for: Event policies are retrieved from the event policy set according to the policy priority associated with the event policy.

[0020] In an optional implementation, the parameter used to determine the attribute comparison rule is a matching method, and the matching methods are different for different event strategies in the event strategy set; the event attribute matching submodule is specifically used for: Determine the matching method of the event strategy, and according to the type of the matching method, perform a comparison operation between the second event attribute and the first event attribute of the event strategy.

[0021] In one optional implementation, the unitized business system has a corresponding configuration file; The device further includes a virtual network identifier pre-allocation module, used to pre-allocate a corresponding virtual network identifier for each of the service services in the configuration file.

[0022] In an optional implementation, the apparatus further includes: a routing rule synchronization module, configured to, for any service in any business unit, during the deployment phase of the service, obtain the physical network identifier of the service; establish corresponding routing rules based on the virtual network identifier and physical network identifier of the service; and synchronously distribute the routing rules to the remaining service in the business unit.

[0023] In an optional implementation, the apparatus further includes a service access module, configured to allow the business service to access other business services within the business unit in the following manner: generating an access request from the business service for the other business services and obtaining the domain names corresponding to the other business services within the business unit; resolving the remaining virtual IPs of the other business services corresponding to the domain names, determining the remaining routing rules for the other business services based on the remaining virtual IPs; and forwarding the access request from the business service for the other business services to the other business services according to the remaining routing rules.

[0024] In a third aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the self-healing implementation methods described in the first aspect above.

[0025] In a fourth aspect of the embodiments of this application, a storage medium is also provided, wherein the storage medium stores instructions that, when run on a computer, cause the computer to execute any of the self-healing implementation methods described in the first aspect above.

[0026] In a fifth aspect of the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the self-healing implementation methods described above.

[0027] The technical solution provided in this application embodiment obtains alarm events of a unitized business system, extracts the first event attribute of the alarm event, and obtains an event policy set. Each event policy in the event policy set is configured with a corresponding second event attribute. When the first event attribute matches the second event attribute, the self-healing process corresponding to the target event policy is determined, and the target business service is self-healed according to the self-healing process.

[0028] By automatically capturing alarm events of target business services in target business units within a unitized business system, extracting event attributes, and accurately matching them with a preset event strategy set, the system triggers corresponding self-healing processes to automate the handling of faulty services. This effectively compensates for the shortcomings of traditional unitized architectures in lacking self-healing mechanisms, significantly reduces reliance on manual operations for fault handling, shortens fault recovery time, and lowers operational costs and pressure. At the same time, leveraging the characteristics of unitized architecture, it adapts to complex scenarios in private environments, greatly improving system stability and business continuity, and providing strong support for the reliable operation of enterprise-level applications in private deployment modes. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0032] Figure 1 This is a schematic diagram of the architecture of a unitized business system shown in an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation process of a self-healing implementation method shown in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the implementation process of another self-healing implementation method shown in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the implementation process of another self-healing implementation method shown in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the admission event attributes included in an admission policy as shown in an embodiment of this application; Figure 6 This is a schematic diagram of the service access path for a service unit 1 as shown in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a self-healing device shown in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0035] like Figure 1 The diagram shown is an architectural representation of a modular business system according to an embodiment of this application. This modular business system includes multiple business units, each containing multiple business services. These multiple business services form a complete business system, and all business units contain the same business services. For example, business unit 1, business unit 2, and business unit 3 all contain all the services of a collaborative office system.

[0036] The unitized business system includes a built-in load balancer for routing and forwarding, and supports weight allocation. To achieve built-in fault isolation and high availability capabilities, this application deploys the business in multiple independent business units. Each business unit contains complete business services, and each unit can guarantee the normal operation of the business. The business units share middleware resources (such as databases and storage). Business traffic is strictly limited to flow within the business unit, maximizing fault isolation and improving system robustness.

[0037] Self-healing refers to a technical solution that, in the event of a system failure, anomaly, or performance degradation, automatically detects the problem, locates the root cause, and performs recovery operations without human intervention (or with minimal intervention), ultimately returning the system to a stable operating state. Its core objective is to improve the system's high availability (HA) and fault tolerance, and reduce operational costs. It is particularly suitable for scenarios with high stability requirements, such as distributed systems, cloud computing, network equipment, and industrial control.

[0038] Based on the above, Figure 1 The modular system shown, such as Figure 2 The diagram shown is a schematic representation of the implementation process of a self-healing method provided in this application. This method is applied to electronic devices and may specifically include the following steps: S201, retrieve alarm events from the unitized business system.

[0039] In this embodiment, alarm events of the unitized business system are acquired. These alarm events are used to alert the target business service of the target business unit in the unitized business system, indicating that the target business service of the target business unit in the unitized business system has failed. This step is the starting point of the self-healing process, and its core is to collect abnormal alarms of the target business service (e.g., business service 1) within the target business unit (e.g., business unit 1) in the unitized business system.

[0040] It should be noted that these alarm events come from a variety of sources, including metric alarms from monitoring tools such as VictoriaMetrics and Prometheus, log alarms from open-source log monitoring and alarm tools such as Loki and ELK Stack, and abnormal notifications triggered by scheduled tasks of the operation and maintenance platform. This application embodiment does not limit these alarm sources.

[0041] The core function of alarm events is to accurately identify abnormal objects, that is, to clarify "which business service in which business unit has a problem" (e.g., business service 1 of business unit 1 is unavailable due to program failure), providing a clear target for subsequent processing. At the same time, the acquired alarm events will first undergo attribute cleaning and parameter validation, and be transformed into structured data in a unified format, laying the foundation for attribute extraction and strategy matching.

[0042] In this embodiment, alarm events are detected and output through a multi-dimensional detection mechanism and by setting alarm thresholds. For example, the multi-dimensional detection mechanism is used to build indicator monitoring (such as CPU, memory, interface response time, database connection count, etc.) and combine log monitoring and link tracing for detection. For example, by setting alarm thresholds, dynamic threshold adjustment is supported, and the normal fluctuation range is predicted based on historical data to avoid false alarms. At the same time, hierarchical alarms are set for critical faults (such as unit service downtime, database unavailability, etc.) (such as SMS, email, operation and maintenance platform pop-ups in different services).

[0043] S202, extract the first event attribute of the alarm event and obtain the event policy set. Each event policy in the event policy set is configured with a corresponding second event attribute.

[0044] In this embodiment, the first event attribute of the alarm event is extracted. The alarm event has been standardized beforehand. Here, the key features of the standardized alarm event are extracted as the "first event attribute", which includes, for example, event name, event source (e.g., "VictoriaMetrics"), event content (e.g., "service response timeout"), business unit (e.g., business unit 1), event status, creation time, etc. These event attributes are the core basis for subsequent matching.

[0045] Furthermore, in this embodiment, a preset set of event policies is retrieved from the system. Each event policy is configured with a "second event attribute" (e.g., preset "event source = VictoriaMetrics, event type = SLO alarm, alarm service = business service 1"), and is associated with corresponding processing rules (e.g., bound self-healing processes). The event policy set supports configuration and management through a visual interface, adapting to the needs of different business scenarios.

[0046] S203, if the first event attribute matches the second event attribute, determine the self-healing process corresponding to the target event strategy.

[0047] In this embodiment, the first event attribute of an alarm event is compared with the second event attribute of an event policy in the event policy set. The matching logic is based on preset rules (e.g., whether the dimensions such as event source, event type, alarm service, and affiliated business unit are consistent). For example, when an alarm event of "Business Service 1 of Business Unit 1 is abnormal" has its first event attribute (event source = VictoriaMetrics, alarm service = business service 1, affiliated business unit = business unit 1) completely matched with the second event attribute of an event policy, the event policy is determined to be the "target event policy," and its bound self-healing process is triggered. If they do not match, the alarm event may enter the event suppression process and self-healing will not be initiated. This step ensures that the self-healing action corresponds to the fault scenario through precise matching, avoiding misoperation.

[0048] S204, Perform self-healing processing on the target business service according to the self-healing process.

[0049] In this embodiment, the target business service is self-healed according to a self-healing process. Specifically, the self-healing engine invokes a predefined automated process bound to the target event policy to handle the abnormal target business service. The specific actions include: Traffic control: Remove traffic from abnormal units (e.g., temporarily switch traffic out of business unit 1 through IPVS rules) to prevent the spread of faults.

[0050] Fault diagnosis and repair: Perform operations such as anomaly checks, automatic service restarts, and configuration resets (e.g., restarting the business service 1 container).

[0051] Service recovery verification: Determine whether the service has returned to normal through health checks and business inspections.

[0052] Traffic rollback: If the service is restored, traffic will be automatically redirected back to the business unit (e.g., updating IPVS rules).

[0053] Furthermore, once the self-healing action is completed, the event status will be automatically updated (e.g., "recovered") to ensure a closed-loop fault handling process. The entire process supports end-to-end traceability, facilitating post-event analysis and strategy optimization.

[0054] In addition, recovery strategies can be set in the self-healing process. For example, service-level self-healing can be set: service instances within a unit detect their status through probes (such as K8s liveness probes, ready probes, etc.). When an instance is abnormal, a restart is automatically triggered. If the restart fails, the core control layer automatically schedules a standby instance to take over the service, ensuring uninterrupted service. For example, unit-level self-healing can be set: when the entire unit fails (such as a server cluster crash), the core control layer quickly switches the business traffic of the unit to the standby unit based on preset rules, and triggers the unit reconstruction process (automatically pulling the image and restoring data shards). User operations are not affected during the recovery process. For example, data-level self-healing can be set: data within a unit can adopt a "local storage + off-site backup" mode, supporting scheduled snapshots and real-time incremental backups. In the event of data corruption or loss, a data recovery process is automatically triggered to restore data from the most recent backup point, and MTTR is controlled within the required threshold period.

[0055] Based on the above description of the technical solution provided in the embodiments of this application, alarm events of a unitized business system are obtained, the first event attribute of the alarm event is extracted, and an event policy set is obtained. Each event policy in the event policy set is configured with a corresponding second event attribute. When the first event attribute matches the second event attribute, the self-healing process corresponding to the target event policy is determined, and the target business service is self-healed according to the self-healing process.

[0056] By automatically capturing alarm events of target business services in target business units within a unitized business system, extracting event attributes, and accurately matching them with a preset event strategy set, the system triggers corresponding self-healing processes to automate the handling of faulty services. This effectively compensates for the shortcomings of traditional unitized architectures in lacking self-healing mechanisms, significantly reduces reliance on manual operations for fault handling, shortens fault recovery time, and lowers operational costs and pressure. At the same time, leveraging the characteristics of unitized architecture, it adapts to complex scenarios in private environments, greatly improving system stability and business continuity, and providing strong support for the reliable operation of enterprise-level applications in private deployment modes.

[0057] like Figure 3 The diagram shown illustrates the implementation flow of another self-healing method provided in this application. This method is applied to electronic devices and may specifically include the following steps: S301, retrieve alarm events from the unitized business system.

[0058] In this embodiment of the application, this step is similar to step S201 above, and will not be described in detail here.

[0059] S302, determine the alarm source of the alarm event, and perform standardized processing on the alarm event according to the alarm source to obtain a standard structured event.

[0060] In this embodiment, it supports access to various alarm sources, including metric alarms from monitoring tools such as VictoriaMetrics and Prometheus, log alarms from open-source log monitoring and alarm tools such as Loki, and abnormal notifications triggered by scheduled tasks of the operation and maintenance platform. The non-standard alarm events output by different alarm sources are uniformly converted into standard structured events, and field mapping is completed (for example, mapping the "trigger name" of Zabbix to the "event name", and the "log source" of ELK to the "event source"), to ensure the universality of subsequent attribute extraction and policy matching.

[0061] Based on this, for the alarm events acquired from the unitized business system, the alarm source of the alarm event is determined, and the alarm event is standardized according to the alarm source to obtain a standard structured event. The standardization process involves finding the field mapping rule corresponding to the alarm source and mapping the fields in the alarm event according to that rule (for example, mapping Zabbix's "trigger name" to "event name", and ELK's "log source" to "event source") to obtain a standard structured event.

[0062] Furthermore, there are multiple alarm events and multiple alarm sources, with different alarm sources having different priorities (e.g., core business alarm sources have higher priority than non-core ones), adapting to the hierarchical processing requirements of multi-source alarms in complex operation and maintenance scenarios. It also supports user configuration of a whitelist of allowed alarm sources.

[0063] Therefore, step S302 can specifically be as follows: for any alarm event, determine the alarm source of the alarm event and determine the priority of the alarm source; sort multiple alarm events according to the priority to obtain an alarm event sequence; traverse the alarm events in the alarm event sequence in order, and standardize the traversed alarm events according to the alarm source of the traversed alarm events to obtain standard structured events.

[0064] S303, extract the first event attribute of the standard structured event and obtain the event strategy set. Each event strategy in the event strategy set is configured with a corresponding second event attribute.

[0065] In this embodiment, the first event attribute of the alarm event is extracted. The alarm event has been standardized beforehand to obtain a standard structured event. Therefore, the key features of the standard structured event are extracted as the "first event attribute," which specifically includes the event name, event source (e.g., "VictoriaMetrics"), event content (e.g., "service response timeout"), the business unit to which it belongs (e.g., business unit 1), event status, creation time, etc. These event attributes are the core basis for subsequent matching.

[0066] Furthermore, in this embodiment, a preset set of event policies is retrieved from the system. Each event policy is configured with a "second event attribute" (e.g., preset "event source = VictoriaMetrics, event type = SLO alarm, alarm service = business service 1"), and is associated with corresponding processing rules (e.g., bound self-healing processes). The event policy set supports configuration and management through a visual interface, adapting to the needs of different business scenarios.

[0067] Before obtaining the event policy set, the target event attribute can be extracted from the first event attribute, and the configured admission policy can be obtained. The admission policy contains admission event attributes. If the target event attribute matches the admission event attribute, the step of obtaining the event policy set is executed.

[0068] For example, target event attributes such as event source (prometheus), event type (slo), and alarm service (business service 1) are extracted from the first event attributes, and the configured admission policy is obtained. The admission policy includes admission event attributes (e.g., event source = prometheus, event type = SLO alarm, alarm service = business service 1). When the target event attributes match the admission event attributes, the admission is matched with the event policy, thereby obtaining the event policy set.

[0069] S304, if the first event attribute matches the second event attribute, determine the self-healing process corresponding to the target event strategy.

[0070] In this embodiment of the application, this step is similar to step S203 above, and will not be described in detail here.

[0071] S305, follow the self-healing process to perform self-healing processing on the target business service.

[0072] In this embodiment of the application, this step is similar to step S204 above, and will not be described in detail here.

[0073] By automatically capturing alarm events of target business services in target business units within a unitized business system, and standardizing alarm events based on their alarm sources to obtain standard structured events, the system extracts the event attributes of these standard structured events and precisely matches them with a preset event strategy set. This triggers corresponding self-healing processes to automate the handling of faulty services. This effectively compensates for the lack of self-healing mechanisms in traditional unitized architectures, significantly reduces reliance on manual operations for fault handling, shortens fault recovery time, and lowers operational costs and pressure. At the same time, leveraging the characteristics of unitized architecture, it adapts to complex scenarios in private environments, greatly improving system stability and business continuity, and providing strong support for the reliable operation of enterprise-level applications in private deployment modes.

[0074] like Figure 4 The diagram shown illustrates the implementation flow of another self-healing method provided in this application. This method is applied to electronic devices and may specifically include the following steps: S401, retrieve alarm events from the unitized business system.

[0075] In this embodiment of the application, this step is similar to step S201 above, and will not be described in detail here.

[0076] S402, extract the first event attribute of the alarm event and obtain the event policy set. Each event policy in the event policy set is configured with a corresponding second event attribute.

[0077] In this embodiment of the application, this step is similar to step S202 above, and will not be described in detail here.

[0078] S403, retrieve the event strategy from the event strategy set according to the parameters associated with the event strategy used to determine the strategy selection order.

[0079] S404, based on the parameters associated with the event strategy used to determine the attribute comparison rules, match the second event attribute of the event strategy with the first event attribute.

[0080] In the embodiments of this application, the event strategy set includes multiple different event strategies, and each event strategy in the event strategy set is associated with at least two event strategy matching control parameters. The event strategy matching control parameters include parameters for determining the strategy selection order and parameters for determining the attribute comparison rules.

[0081] Therefore, according to the parameters associated with the event strategy used to determine the strategy selection order, the event strategy is obtained from the event strategy set, and the second event attribute of the event strategy is matched with the first event attribute according to the parameters associated with the event strategy used to determine the attribute comparison rules.

[0082] Among them, the parameter used to determine the order of strategy selection is the strategy priority. Different event strategies in the event strategy set have different strategy priorities. Therefore, event strategies can be obtained according to the strategy priority and matched with alarm events, thus adapting to different business needs.

[0083] Furthermore, the matching method can be different for the parameter used to determine the attribute comparison rules, and different event strategies in the event strategy set can also use different matching methods. For example, the matching method can be divided into exact matching, fuzzy matching, range matching (such as alarm threshold range), etc. Some high-priority event strategies can be set to exact matching to prevent false triggering of the self-healing process, while some low-priority event strategies can be set to fuzzy matching.

[0084] Therefore, according to the strategy priority, the event strategy is obtained from the event strategy set, the matching method of the event strategy is determined, and according to the type of matching method (e.g., exact matching, fuzzy matching, range matching), the comparison operation between the second event attribute and the first event attribute of the corresponding event strategy is performed.

[0085] S405, if the second event attribute of the event strategy matches the first event attribute, the event strategy is determined as the target event strategy, and the self-healing process corresponding to the target event strategy is determined.

[0086] In this embodiment, when the second event attribute of the event policy matches the first event attribute of the alarm event, the event policy is determined as the target event policy, thereby determining the self-healing process corresponding to the target event policy. The self-healing process consists of at least one action, including but not limited to: removing abnormal unit traffic, anomaly checking and diagnosis, anomaly analysis, service flow switching and degradation, automatic restart, health check, status update and alarm callback, and determining whether the service has recovered through inspection.

[0087] S406, Perform self-healing processing on the target business service according to the self-healing process.

[0088] In this embodiment of the application, this step is similar to step S204 above, and will not be described in detail here.

[0089] By automatically capturing alarm events of target business services in target business units within a unitized business system, extracting event attributes, and accurately matching them with preset event policy sets according to policy priorities, the system triggers corresponding self-healing processes to automate the handling of faulty services. This effectively compensates for the shortcomings of traditional unitized architectures in lacking self-healing mechanisms, significantly reduces reliance on manual operations for fault handling, shortens fault recovery time, and lowers operational costs and pressure. At the same time, leveraging the characteristics of unitized architecture, it adapts to complex scenarios in private environments, greatly improving system stability and business continuity, and providing strong support for the reliable operation of enterprise-level applications in private deployment modes.

[0090] The self-healing implementation method provided in this application embodiment is described below with reference to specific examples: When the business service 1 of business unit 1 is abnormal due to program failure, the business of unit 1 becomes unavailable.

[0091] The self-healing process is roughly as follows: 1. Business service 1 of business unit 1 is out of service due to a program failure, and the business of unit 1 is unavailable.

[0092] 2. A failure occurs in service 1 of the business unit 1 monitored by VictoriaMetrics, which generates an SLO alarm event through Altermanager.

[0093] 3. Alarm events are processed into standard structured event instances, specifically by extracting event attributes, including: event name, event source, event content, unit to which it belongs, event status, creation time, etc.

[0094] 4. If the event attribute of the alarm event belongs to, for example Figure 5 The types shown will be admitted and matched with the event policy.

[0095] 5. If the event attributes of the alarm event match the event attributes of the event policy stored in the platform, and the event source, event type, and alarm service are consistent, then the bound self-healing process will be entered; otherwise, the self-healing process will not be entered.

[0096] 6. Successfully matched event strategies will be bound to a self-healing process, which will be automatically executed by the self-healing engine. The self-healing engine processes the target through a predefined automated process.

[0097] 7. After the self-healing action is completed, the system will automatically update the event status to ensure a closed-loop fault handling process. The entire process supports full-link traceability, facilitating post-event analysis and strategy optimization.

[0098] Furthermore, in this embodiment, business traffic is strictly restricted to flow within the business unit to isolate the scope of failures to the greatest extent and improve system robustness. To this end, business service deployment adopts a containerized approach, such as using Docker containerization. Traffic scheduling is based on failover components and traffic distribution components to achieve an efficient and stable service access and forwarding mechanism, for example, based on Keepalived and IPVS.

[0099] Each business service pre-assigns a unique virtual IP (VIP) and virtual port in its configuration file, and generates traffic distribution rules (e.g., IPVS rules) through a failover component (e.g., Keepalived). Specifically, during the deployment phase, the system automatically generates corresponding traffic distribution rules (e.g., IPVS routing rules) based on the actual deployment node information of the service, and synchronously distributes them to the remaining business services within the business unit. At this time, the actual backend address of the traffic distribution rules (e.g., IPVS rules) will only be the business service within the business unit.

[0100] Simultaneously, a unified ` / etc / hosts` service name resolution configuration is generated for each business service. Service names can be resolved to VIPs, and the VIPs are then forwarded to the actual service backend, ensuring cross-business service access is achieved through service names. This enables a stable, isolated, and efficient communication mechanism between services. The internal service access path of Business Unit 1 is roughly as follows: Figure 6 As shown, node1, node2, and node3 belong to three business services within business unit 1, and IPVS rule traffic is limited to circulation within the business unit.

[0101] Based on this, the unitized business system has corresponding configuration files, in which a corresponding virtual network identifier (e.g., virtual IP and virtual port) is pre-assigned for each business service. For any business service in any business unit, during the deployment phase of the business service, the physical network identifier (e.g., physical IP and physical port) of the business service is obtained; based on the virtual network identifier and physical network identifier of the business service, corresponding routing rules are established; and the routing rules are synchronously distributed to the remaining business services in the business unit.

[0102] Thus, for a business service within a business unit, access to other business services within the business unit can be achieved through the following methods: generating an access request from the business service to other business services and obtaining the domain names corresponding to the other business services within the business unit; resolving the other virtual IPs of the other business services corresponding to the domain names, determining the other routing rules for the other business services based on the other virtual IPs; and forwarding the access request from the business service to other business services to the other business services based on the other routing rules.

[0103] This effectively achieves built-in fault isolation and high availability, significantly improving system stability. Furthermore, a complete self-healing process has been designed and implemented, enabling automatic fault detection and repair, significantly improving system stability and shortening fault recovery time (MTTR).

[0104] Corresponding to the above method embodiments, this application also provides a self-healing implementation device applied to a unitized business system. The unitized business system includes multiple business units, each containing multiple business services, and all business units contain the same business services, such as... Figure 7 As shown, the device may include: an event acquisition module 710, an attribute extraction module 720, a strategy set acquisition module 730, a process determination module 740, and a self-healing processing module 750.

[0105] The event acquisition module 710 is used to acquire alarm events of the unitized business system, and the alarm events are used to alarm the target business service of the target business unit in the unitized business system. The attribute extraction module 720 is used to extract the first event attribute of the alarm event; The strategy set acquisition module 730 is used to acquire an event strategy set, wherein each event strategy in the event strategy set is configured with a corresponding second event attribute. The process determination module 740 is used to determine the self-healing process corresponding to the target event strategy when the first event attribute matches the second event attribute. The self-healing module 750 is used to perform self-healing processing on the target business service according to the self-healing process.

[0106] In an optional implementation, the attribute extraction module specifically includes: The alarm source determination submodule is used to determine the alarm source of the alarm event, and to standardize the alarm event according to the alarm source to obtain a standard structured event; The event attribute extraction submodule is used to extract the first event attribute of the standard structured event.

[0107] In one optional implementation, there are multiple alarm sources, and different alarm sources have different priorities; there are also multiple alarm events. The alarm source determination submodule is specifically used for: For any of the aforementioned alarm events, determine the alarm source of the alarm event and determine the priority of the alarm source; The alarm events are sorted according to the priority to obtain an alarm event sequence; The alarm events in the alarm event sequence are traversed sequentially, and the alarm events are standardized according to their alarm sources to obtain standard structured events.

[0108] In an optional implementation, the apparatus further includes: an admission verification module, configured to extract a target event attribute from the first event attribute and obtain a configured admission policy, the admission policy including admission event attributes; and, if the target event attribute matches the admission event attribute, to execute the step of obtaining the event policy set.

[0109] In an optional implementation, each event strategy in the event strategy set is associated with at least two event strategy matching control parameters, which include parameters for determining the strategy selection order and parameters for determining attribute comparison rules; the process determination module specifically includes: The event strategy acquisition submodule is used to acquire event strategies from the event strategy set according to the parameters associated with the event strategy for determining the strategy selection order. The event attribute matching submodule is used to match the second event attribute of the event strategy with the first event attribute based on the parameters associated with the event strategy for determining the attribute comparison rules. The event strategy determination submodule is used to determine the event strategy as the target event strategy when the second event attribute of the event strategy matches the first event attribute. The process determination submodule is used to determine the self-healing process corresponding to the target event strategy.

[0110] In an optional implementation, the parameter used to determine the strategy selection order is the strategy priority, and different event strategies in the event strategy set have different strategy priorities; the event strategy acquisition submodule is specifically used for: Event policies are retrieved from the event policy set according to the policy priority associated with the event policy.

[0111] In an optional implementation, the parameter used to determine the attribute comparison rule is a matching method, and the matching methods are different for different event strategies in the event strategy set; the event attribute matching submodule is specifically used for: Determine the matching method of the event strategy, and according to the type of the matching method, perform a comparison operation between the second event attribute and the first event attribute of the event strategy.

[0112] In one optional implementation, the unitized business system has a corresponding configuration file; The device further includes a virtual network identifier pre-allocation module, used to pre-allocate a corresponding virtual network identifier for each of the service services in the configuration file.

[0113] In an optional implementation, the apparatus further includes: a routing rule synchronization module, configured to, for any service in any business unit, during the deployment phase of the service, obtain the physical network identifier of the service; establish corresponding routing rules based on the virtual network identifier and physical network identifier of the service; and synchronously distribute the routing rules to the remaining service in the business unit.

[0114] In an optional implementation, the apparatus further includes a service access module, configured to allow the business service to access other business services within the business unit in the following manner: generating an access request from the business service for the other business services and obtaining the domain names corresponding to the other business services within the business unit; resolving the remaining virtual IPs of the other business services corresponding to the domain names, determining the remaining routing rules for the other business services based on the remaining virtual IPs; and forwarding the access request from the business service for the other business services to the other business services according to the remaining routing rules.

[0115] This application also provides an electronic device, such as... Figure 8 As shown, it includes a processor 81, a communication interface 82, a memory 83, and a communication bus 84, wherein the processor 81, the communication interface 82, and the memory 83 communicate with each other through the communication bus 84. Memory 83 is used to store computer programs; When processor 81 executes the program stored in memory 83, it performs the following steps: The system acquires alarm events from a unitized business system, whereby the alarm events are used to alert the target business service of a target business unit within the unitized business system; it extracts a first event attribute from the alarm events and acquires an event policy set, where each event policy in the event policy set is configured with a corresponding second event attribute; when the first event attribute matches the second event attribute, it determines the self-healing process corresponding to the target event policy; and it performs self-healing processing on the target business service according to the self-healing process.

[0116] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0117] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0118] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0119] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0120] In another embodiment provided in this application, a storage medium is also provided, which stores instructions that, when run on a computer, cause the computer to execute any of the self-healing implementation methods described in the above embodiments.

[0121] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the self-healing implementation methods described in the above embodiments.

[0122] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0123] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0124] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0125] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A self-healing implementation method, characterized by, The method is applied to a unit service system, and comprises the following steps: An alarm event of the unit service system is acquired; First event attributes of the alarm event are extracted, and a set of event strategies is acquired, each event strategy in the set of event strategies being configured with corresponding second event attributes; In a case where the first event attributes match the second event attributes, a self-healing process corresponding to a target event strategy is determined; According to the self-healing process, self-healing processing is performed on a target service.

2. The method of claim 1, wherein, The first event attributes of the alarm event are extracted, comprising the following steps: An alarm source of the alarm event is determined, and the alarm event is standardized according to the alarm source to obtain a standard structured event; The first event attributes of the standard structured event are extracted.

3. The method of claim 2, wherein, The number of alarm sources is multiple, and the priority of different alarm sources is different, and the number of alarm events is multiple; The alarm source of the alarm event is determined, and the alarm event is standardized according to the alarm source to obtain a standard structured event, comprising the following steps: For any alarm event, the alarm source of the alarm event is determined, and the priority of the alarm source is determined; According to the priority, the multiple alarm events are sorted to obtain an alarm event sequence; The alarm events in the alarm event sequence are traversed in sequence, and the alarm events traversed are standardized according to the alarm source of the alarm events traversed to obtain a standard structured event.

4. The method of claim 1, wherein, Before the set of event strategies is acquired, the following steps are further included: Target event attributes are extracted from the first event attributes, and a configured admission strategy is acquired, the admission strategy containing admission event attributes; In a case where the target event attributes match the admission event attributes, the step of acquiring the set of event strategies is performed.

5. The method of claim 1, wherein, Each event strategy in the set of event strategies is associated with at least two event strategy matching control parameters, the event strategy matching control parameters including a parameter for determining a strategy selection order and a parameter for determining an attribute comparison rule; In a case where the first event attributes match the second event attributes, a self-healing process corresponding to a target event strategy is determined, comprising the following steps: According to the parameter for determining a strategy selection order associated with an event strategy, an event strategy is acquired from the set of event strategies; According to the parameter for determining an attribute comparison rule associated with the event strategy, the second event attributes of the event strategy are matched with the first event attributes; In a case where the second event attributes of the event strategy match the first event attributes, the event strategy is determined as the target event strategy; A self-healing process corresponding to the target event strategy is determined.

6. The method of claim 5, wherein, The parameter for determining a strategy selection order is a strategy priority, and the strategy priority of different event strategies in the set of event strategies is different; According to the parameter for determining a strategy selection order associated with an event strategy, an event strategy is acquired from the set of event strategies, comprising the following steps: According to the strategy priority associated with an event strategy, an event strategy is acquired from the set of event strategies.

7. The method of claim 5, wherein, The parameter for determining an attribute comparison rule is a matching mode, and the matching mode of different event strategies in the set of event strategies is different; The parameter for determining the attribute comparison rule associated with the event strategy is matched with the second event attribute of the event strategy and the first event attribute, including: The matching mode of the event strategy is determined, and the comparison operation of the second event attribute of the event strategy and the first event attribute is performed according to the type of the matching mode.

8. The method of claim 1, wherein, The unit business system has a corresponding configuration file; The method further comprises: pre-allocating a corresponding virtual network identifier for each business service in the configuration file.

9. The method of claim 8, wherein, The method further comprises: For any business service in any business unit, in the deployment stage of the business service, the physical network identifier of the business service is obtained; According to the virtual network identifier and the physical network identifier of the business service, a corresponding routing rule is established; The routing rule is synchronized and issued to the remaining business services in the business unit.

10. The method of claim 9, wherein, The business service accesses the remaining business services in the business unit by the following way: Generate an access request of the business service to the remaining business services, and obtain the domain name corresponding to the remaining business services in the business unit; Parse the remaining virtual IP of the remaining business services corresponding to the domain name, and determine the remaining routing rule of the remaining business services according to the remaining virtual IP; According to the remaining routing rule, the access request of the business service to the remaining business services is forwarded to the remaining business services.

11. A self-healing implementation device, characterized by The device is applied to a unit business system, and the device comprises: An event acquisition module configured to acquire an alarm event of the unit business system; An attribute extraction module configured to extract a first event attribute of the alarm event; A strategy set acquisition module configured to acquire an event strategy set, each event strategy in the event strategy set being configured with a corresponding second event attribute; A flow determination module configured to determine a self-healing flow corresponding to a target event strategy in a case where the first event attribute is matched with the second event attribute; A self-healing processing module configured to perform self-healing processing on a target business service according to the self-healing flow.

12. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are in communication with each other through the communication bus; The memory is configured to store a computer program; The processor is configured to execute the program stored on the memory to implement the method in any one of claims 1-10. The device is applied to a unit business system, and the device comprises: An event acquisition module configured to acquire an alarm event of the unit business system; An attribute extraction module configured to extract a first event attribute of the alarm event; A strategy set acquisition module configured to acquire an event strategy set, each event strategy in the event strategy set being configured with a corresponding second event attribute; A flow determination module configured to determine a self-healing flow corresponding to a target event strategy in a case where the first event attribute is matched with the second event attribute; A self-healing processing module configured to perform self-healing processing on a target business service according to the self-healing flow. The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are in communication with each other through the communication bus; The memory is configured to store a computer program; The processor is configured to execute the program stored on the memory to implement the method in any one of claims 1-10.

Images