Monitoring and automatic fault processing method and system based on service bus

Abstract

The invention discloses a monitoring and automatic fault processing method and system based on a service bus, and the method comprises the following steps: a management module configures infrastructure information needing to be monitored, and selects a corresponding collection module for registration based on an equipment instance; the acquisition module acquires hardware states, configuration parameters and alarm information from each device according to a preset frequency and protocol, and transmits data through a bus; the service bus pushes the transmitted data to all connected modules in real time; in response to the data transmitted in the service bus, the message module pushes the alarm information to a specified user or group through a preset medium mode according to user and event types; and in response to the abnormal data transmitted in the service bus, the action module executes processing according to a predefined action rule. According to the processing method provided by the invention, the event processing consistency of the whole monitoring system can be ensured, custom operation can be carried out through the action module, and the high availability requirement of the infrastructure is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrastructure operation and maintenance, and in particular relates to a monitoring and automated fault handling method and system based on a service bus. Background Art

[0002] The existing monitoring method adopts a process-based approach. The collaborative ability of each module is limited. The collection process, storage process, and processing process are mutually called and highly coupled. Each module is easy to affect each other, and it is difficult to modify and maintain a single module. The processing efficiency of monitoring information is low. The message and log storage process controlled by the process is easy to cause the storage and release of messages to be out of sync, and maintenance is difficult. In addition, traditional monitoring cannot perform automated fault handling operations on the monitored objects.

[0003] For example, the Chinese invention patent application with publication number CN111193616A discloses an automatic operation and maintenance method, device, system, storage medium and automatic operation and maintenance server. The method includes: the automatic operation and maintenance server is connected to the business server based on the network, wherein the network includes an internal local area network and an external public network; the automatic fault detection module in the automatic operation and maintenance server performs abnormal detection on the business server based on a preset interval detection time; when the business server is detected to be abnormal, the automatic repair program is triggered in the automatic operation and maintenance server to automatically repair the business server abnormality; the automatic alarm program is triggered in the automatic operation and maintenance server to notify the person in charge of the fault through a preset alarm method.

[0004] The above scheme provides a process-based event processing management function to realize the monitoring of infrastructure equipment, event alarm reminder and other functions, but it cannot provide any automated fault handling operations for the managed objects. In order to solve the problem of low processing efficiency of the process-based method, the Chinese invention patent application with publication number CN117909167A discloses a monitoring method, device, electronic device and storage medium for a service bus, the method comprising: collecting monitoring logs; storing monitoring logs; processing monitoring logs to generate a monitoring indicator system of three dimensions of service bus, service and caller; displaying the monitoring indicator system in the dimensions of service bus, service or caller respectively. This application forms a monitoring indicator system of three dimensions of service bus, service and caller based on the call monitoring data, and presents the monitoring indicator system in a visual form, which is convenient for operation and maintenance personnel to obtain abnormal information and then respond to abnormalities quickly, and can avoid the process of writing code for operation and maintenance personnel, which is conducive to improving the efficiency of operation and maintenance personnel in handling abnormalities.

[0005] The above solution solves the problem of low efficiency of process-based methods, but there are still defects in the consistency of event processing in the overall monitoring system, and the data consistency requirements of monitoring transaction processing cannot be guaranteed. In addition, the above solution only solves the monitoring problem of infrastructure, and cannot automatically handle failures that occur during the monitoring process. Summary of the invention

[0006] The present invention provides a monitoring and automated fault handling method based on a service bus, aiming to solve the problems that the prior art cannot guarantee the data consistency requirement in monitoring transaction processing and cannot handle faults found in the monitoring process.

[0007] In order to solve the above technical problems, the monitoring and automatic fault handling method proposed in the present invention comprises the following steps:

[0008] The management module configures the infrastructure information that needs to be monitored and selects the corresponding acquisition module for registration based on the device instance;

[0009] The acquisition module obtains hardware status, configuration parameters and alarm information from each device according to the preset frequency and protocol, and transmits the data through the bus;

[0010] The service bus pushes the transmitted data to all connected modules in real time;

[0011] In response to the data transmitted in the service bus, the message module pushes the alarm information to the specified user or group through the preset media according to the user and event type;

[0012] In response to abnormal data transmitted within the service bus, the action module performs processing according to predefined action rules.

[0013] Preferably, the method further comprises the following steps:

[0014] In response to the data transmitted in the service bus, the storage module logs and stores all collected data, event processing records, and configuration updates.

[0015] Preferably, the preset media methods include text messages, emails, system prompts and customized push notifications.

[0016] Preferably, the processing method of the action module includes: notifying through the bus triggering message module, automatically executing repair commands, adjusting event levels or durations, directly ending events, and executing script commands to restart specified services on the client, close specified ports, network devices ban specified IPs, and storage devices switch controllers.

[0017] Preferably, the data transmitted in the service bus includes hardware status, configuration parameters, alarm information, notification messages and time information.

[0018] Preferably, the management module stores configuration templates of the acquisition module for configuration of acquisition modules of devices of the same brand or with similar parameters, and the configuration templates include agent client templates, storage SNMP templates, storage SMI templates, network device SNMP and syslog templates.

[0019] Preferably, the information of the user or group is maintained by a user and group management module, and the user and group management module sets permissions and alarm notification media for different users.

[0020] Preferably, one host of the management module can use multiple acquisition modules, and one acquisition module can provide services for multiple hosts.

[0021] Another aspect of the present invention further provides a monitoring and automated fault handling system based on a service bus, wherein the system is used to implement the above-mentioned monitoring and automated fault handling method, and comprises:

[0022] A collection module, used to collect status data and alarm data of the infrastructure, transmit the status data and alarm data through the service bus, and receive configuration or collection instructions through the service bus;

[0023] The service bus is used to push the transmitted data to all connected modules in real time. The data transmitted in the service bus includes hardware status, configuration parameters, alarm information, notification messages and time information;

[0024] The host management module is configured with the infrastructure information to be monitored, and selects the corresponding acquisition module for registration based on the device instance; it also receives the data transmitted on the service bus, monitors the infrastructure according to the status data, and sends action instructions through the service bus according to the monitoring results;

[0025] The message module pushes the alarm information to the designated users or groups through the preset media according to the user and event type. The preset media includes one or more of SMS, email, system prompt and customized push;

[0026] The action module is configured with predefined action rules and performs corresponding operations based on the alarm data received from the service bus in combination with the action rules.

[0027] Preferably, the system further comprises a storage module and a user and group management module;

[0028] The storage module is used to provide log storage, configuration storage and database interaction storage, and logs and stores all collected data, event processing records and configuration updates;

[0029] The user and group management module is used to set the permissions and alarm notification media of different users, so that users at different levels obtain corresponding monitoring permissions and alarm messages.

[0030] Compared with the prior art, the present invention has the following technical effects:

[0031] 1. The processing method proposed in the present invention constructs a bus structure, and the information flow transmission of the monitoring system adopts a global push mode. For example, the generation of an alarm data flow will be pushed to all modules connected to the bus. After receiving the data flow, different modules will perform corresponding operations, thereby ensuring the consistency of the overall event processing of the monitoring system.

[0032] 2. The processing method proposed in the present invention can not only monitor the status of the infrastructure by setting the action module, but also customize the operation through the action module, arrange the action sequence flow, action start rules, etc. It can be used as a fault emergency measure. After the fault occurs, the fault handling operation is automatically performed according to the defined arrangement action. In the production environment, it is very practical for systems with high requirements for real-time, timeliness and high availability.

[0033] 3. The processing method proposed in the present invention also provides template configuration. For devices of the same brand or with similar parameters, a template clone can be used and put into production after a small amount of modification. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic flow chart of the processing method of the present invention;

[0035] Figure 2 It is a schematic diagram of internal data of the service bus according to an embodiment of the present invention;

[0036] Figure 3 It is a schematic diagram of a single event processing flow of the processing method of the present invention;

[0037] Figure 4 It is a structural schematic diagram of the processing system of the present invention. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in combination with specific embodiments of the present application and with reference to the accompanying drawings.

[0039] Embodiment 1

[0040] like Figure 1 As shown, this embodiment is a monitoring and automated fault handling method based on a service bus, comprising the following steps:

[0041] The management module configures the infrastructure information that needs to be monitored and selects the corresponding acquisition module for registration based on the device instance;

[0042] The acquisition module obtains hardware status, configuration parameters and alarm information from each device according to the preset frequency and protocol, and transmits the data through the bus;

[0043] The service bus pushes the transmitted data to all connected modules in real time;

[0044] In response to the data transmitted in the service bus, the message module pushes the alarm information to the specified user or group through the preset media according to the user and event type;

[0045] In response to abnormal data transmitted within the service bus, the action module performs processing according to predefined action rules.

[0046] In other embodiments of the present invention, the method further comprises the following steps:

[0047] In response to the data transmitted in the service bus, the storage module logs and stores all collected data, event processing records, and configuration updates.

[0048] The management module configures the infrastructure information that needs to be monitored. Based on the device instance, the corresponding acquisition module is selected for registration. The implementation method is as follows:

[0049] The management module scans and identifies various infrastructure devices (such as servers, storage devices, network devices, etc.) through the system interface, and obtains their basic information, including the unique identifier of the device (such as IP address, device ID, etc.); enters the identified device information into the system, including device type, brand model, location, main configuration parameters, etc.;

[0050] Determine the required monitoring items based on the device type and monitoring requirements, such as CPU load, memory usage, network throughput, storage capacity, etc. For common device types and brands, the system provides predefined collection templates (such as agent clients, SNMP templates, syslog templates, etc.). The management module selects the collection template that best suits the device based on the specific parameters of the device. For devices of the same brand or similar configuration, you can select existing templates and clone and fine-tune them;

[0051] After the collection template is determined, the management module automatically selects the corresponding collection module according to the template type and associates the device instance with the corresponding collection module. For example, network devices usually use the SNMP collection module, while storage devices may use the SMI collection module; the management module registers the device instance to the corresponding collection module to ensure that the collection module can obtain the status data of the device and configure the collection parameters according to the required monitoring items;

[0052] For specific devices or device groups, the management module supports custom monitoring strategies, such as collection frequency, alarm threshold, retry interval, etc. After completing the registration of the device instance, the collection module starts to automatically collect data at the configured frequency and transmits it to the management system through the bus.

[0053] In other embodiments of the present invention, after the device enters the monitoring state, the system continuously monitors the connection status of the device. Once the connection is interrupted or the collection fails, the system will issue an alarm to prompt the administrator to check.

[0054] In this embodiment, the specific implementation process of the message module responding to the data transmitted in the bus and pushing the alarm information to the designated user or group is as follows:

[0055] Initialize the configuration of the message module. The management module pre-configures the contact information (such as mobile phone numbers, email addresses, etc.), role permissions, and alarm receiving preferences (such as only receiving serious alarms, receiving all alarms, etc.) of system users and groups, and synchronizes this information to the message module. The notification media include SMS, email, system notification, push service (such as App push), etc. Each user or group can customize the receiving media within the set permissions;

[0056] The message module monitors the data stream from the bus in real time. When new alarm information arrives, it immediately receives and analyzes the data. It analyzes the received alarm information, including alarm level, event description, timestamp, source device, etc. The analyzed data is used for subsequent priority sorting and push rule judgment.

[0057] The message module determines the priority of the alarm based on the severity level (such as high, medium, and low) of the alarm information and the event type (such as network failure, storage alarm, etc.); according to the type and level of the alarm, it matches the configured receiving users and groups. Different types of events match different user permission groups, such as serious alarms are pushed to administrators and technical support, and low-level alarms are pushed to ordinary users or designated operation and maintenance personnel;

[0058] According to the user's preferences, select the appropriate media method (such as serious alarms are pushed through SMS + email dual channels, and ordinary alarms are pushed through system notifications); format the parsed alarm information into easy-to-read text or rich text content (such as HTML email), and generate push information content, including event title, detailed description, occurrence time and recommended operations, etc.

[0059] In this embodiment, the specific process of the action module executing the processing according to the predefined action rules is as follows:

[0060] During the system configuration phase, administrators can define action rules, including trigger conditions (such as CPU usage exceeding 80%, disk space less than 10%, device offline, etc.), applicable devices or device groups, action priorities, and execution conditions; configure corresponding processing actions for different trigger conditions, such as sending message notifications, automatically executing scripts, raising event levels, adjusting event duration, etc.; configure fault tolerance mechanisms and retry conditions for key actions to ensure execution reliability, such as making multiple attempts when an action fails;

[0061] The action module monitors the data flow in the service bus in real time, automatically obtains and analyzes abnormal data, and ensures immediate response; when abnormal data is received, the action module compares the conditions in the rule base one by one to confirm whether the abnormal data meets a certain trigger condition. For example, when receiving data that the CPU usage exceeds the standard, the bus will immediately trigger the action rule associated with it;

[0062] If multiple rules are triggered at the same time, the action module will sort the actions according to priority, and the high-priority actions will be executed first; for the matching rules, the action module will call the processing operations in sequence according to the predefined action types. Actions include: sending notifications (if the alarm level is high, the message module will be triggered to send alarm information via SMS, email or system notification), automatically executing scripts (for exceptions that support automated processing, the action module calls predefined scripts or commands to repair the fault, such as restarting services, clearing caches, closing abnormal processes, etc.), adjusting event levels or durations (if the event duration exceeds the preset threshold, the action module will dynamically adjust the event level or extend the event response time), and ending events (if the abnormal state returns to normal, the action module will trigger the end action, record the event end time and update the event status on the bus).

[0063] In this embodiment, the action module monitors the execution status of the action during execution to ensure that the action is completed. For example, the system checks whether the automatic execution script ends successfully, or confirms whether the notification has been delivered to the specified user. If the execution fails or the response times out, the system will try to re-execute the action according to the fault tolerance configuration, or select a different processing path according to the upper limit of the number of retries, such as raising the event level or marking the task as requiring manual intervention.

[0064] The preset media methods include text messages, emails, system prompts and customized push notifications.

[0065] The action module establishes a simple-to-implement fault handling operation. The processing method of the action module includes: notifying through the bus triggering message module, automatically executing repair commands, adjusting event levels or durations, directly ending events, and executing script commands to restart specified services on the client, close specified ports, block specified IPs on network devices, and switch controllers on storage devices.

[0066] By setting the action module, this embodiment can not only monitor the status of the infrastructure, but also customize the operation through the action module, arrange the action sequence flow, action startup rules, etc. It can be used as a fault emergency measure. After the fault occurs, the fault handling operation is automatically performed according to the defined arrangement action (for example: executing a script command to restart a service on the client, closing a port, blocking the IP of the network device, switching the controller of the storage device, etc.). In the production environment, it is very practical for systems with high requirements for real-time, timeliness and high availability.

[0067] like Figure 2 As shown, the data transmitted by the service bus in this embodiment includes hardware status, configuration parameters, alarm information, notification messages and time information. The functions of the bus are: providing an interface to provide connection and data exchange channels for other modules; the bus internally processes and transmits information data including but not limited to hardware indicators, configuration parameters, original alarm information, notification messages, etc.; the bus sends the data processing results to each docking module in real time. This information transmission is not one-way, but is globally pushed and released.

[0068] By establishing a bus structure, the monitoring system's information flow transmission adopts a global push method. For example, the generation of an alarm data stream will be pushed to all modules connected to the bus. After receiving the data stream, different modules will perform corresponding operations, ensuring the consistency of the overall event processing of the monitoring system. Specifically, for the processing of any event or data stream, if a module fails to operate successfully due to an exception, the bus will receive the failure information sent by the module and push it to all connected modules, so that the processing of any event or data stream is marked as all successful or all failed in the entire monitoring system, thereby ensuring data consistency. Figure 3 As shown in FIG. 1 , a schematic diagram of a single event processing flow in this embodiment is shown, in which the event sent by the event sender is synchronized to all modules connected to the service bus (such as Figure 3 After receiving the data stream, different modules will perform corresponding operations to ensure the consistency of event processing by the monitoring system as a whole.

[0069] For the bus structure of this embodiment, there can be multiple acquisition modules for collecting status information data of different types of hardware, which can correspond to acquisition methods of different protocols. The management module stores the configuration template of the acquisition module, which is used for the configuration of the acquisition module of the same brand or similar parameters of the equipment. The configuration template includes the agent client template, storage SNMP template, storage SMI template, network equipment SNMP and syslog template. Through template configuration, for the equipment of the same brand or similar parameters, a template clone can be used, and a small modification can be made to put it into production.

[0070] The user or group information is maintained by the user and group management module, which sets the permissions of different users and the alarm notification medium. Specifically, it includes configuring user information, configuring group information, user level type, alarm media notification method information, user permissions on the host,

[0071] One host of the management module can use multiple acquisition modules, and one acquisition module can provide services for multiple hosts.

[0072] Embodiment 2

[0073] like Figure 4 As shown, this embodiment is a monitoring and automatic fault handling system based on a service bus, and the system is used to implement the monitoring and automatic fault handling method as described in the first embodiment, including:

[0074] A collection module, used to collect status data and alarm data of the infrastructure, transmit the status data and alarm data through the service bus, and receive configuration or collection instructions through the service bus;

[0075] The service bus is used to push the transmitted data to all connected modules in real time. The data transmitted in the service bus includes hardware status, configuration parameters, alarm information, notification messages and time information;

[0076] The host management module is configured with the infrastructure information to be monitored, and selects the corresponding acquisition module for registration based on the device instance; it also receives the data transmitted on the service bus, monitors the infrastructure according to the status data, and sends action instructions through the service bus according to the monitoring results;

[0077] The message module pushes the alarm information to the designated users or groups through the preset media according to the user and event type. The preset media includes one or more of SMS, email, system prompt and customized push;

[0078] The action module is configured with predefined action rules and performs corresponding operations based on the alarm data received from the service bus in combination with the action rules.

[0079] The system also includes a storage module and a user and group management module;

[0080] The storage module is used to provide log storage, configuration storage and database interaction storage, and logs and stores all collected data, event processing records and configuration updates;

[0081] The user and group management module is used to set the permissions and alarm notification media of different users, so that users at different levels obtain corresponding monitoring permissions and alarm messages.

[0082] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, which all fall within the protection scope of the present invention.

Claims

1. A monitoring and automated fault handling method based on a service bus, characterized in that: The following steps are involved: The management module configures the infrastructure information that needs to be monitored and selects the corresponding acquisition module for registration based on the device instance; The acquisition module obtains hardware status, configuration parameters and alarm information from each device according to the preset frequency and protocol, and transmits the data through the bus; The service bus pushes the transmitted data to all connected modules in real time; In response to the data transmitted in the service bus, the message module pushes the alarm information to the specified user or group through the preset media according to the user and event type; In response to abnormal data transmitted within the service bus, the action module performs processing according to predefined action rules.

2. The monitoring and automated fault handling method based on the service bus according to claim 1, characterized in that: The method further comprises the following steps: In response to the data transmitted in the service bus, the storage module logs and stores all collected data, event processing records, and configuration updates.

3. The monitoring and automated fault handling method based on a service bus according to claim 1, characterized in that: The preset media methods include text messages, emails, system prompts and customized push notifications.

4. The monitoring and automated fault handling method based on a service bus according to claim 1, characterized in that: The processing method of the action module includes: notifying through the bus triggering message module, automatically executing repair commands, adjusting event levels or durations, directly ending events, and executing script commands to restart specified services on the client, close specified ports, block specified IP addresses on network devices, and switch controllers on storage devices.

5. The monitoring and automated fault handling method based on service bus according to claim 1, characterized in that: The data transmitted in the service bus includes hardware status, configuration parameters, alarm information, notification messages and time information.

6. The monitoring and automated fault handling method based on service bus according to claim 1, characterized in that: The management module stores the configuration templates of the collection module, which are used for the collection module configuration of devices of the same brand or with similar parameters. The configuration templates include agent client templates, storage SNMP templates, storage SMI templates, network equipment SNMP and syslog templates.

7. The monitoring and automated fault handling method based on service bus according to claim 1, characterized in that: The information of the user or group is maintained by a user and group management module, and the user and group management module sets the authority and alarm notification medium of different users.

8. The monitoring and automated fault handling method based on service bus according to claim 1, characterized in that: One host of the management module can use multiple acquisition modules, and one acquisition module can provide services for multiple hosts.

9. A monitoring and automated fault handling system based on a service bus, characterized in that: The system is used to implement the monitoring and automated fault handling method according to any one of claims 1 to 8, comprising: A collection module, used to collect status data and alarm data of the infrastructure, transmit the status data and alarm data through the service bus, and receive configuration or collection instructions through the service bus; The service bus is used to push the transmitted data to all connected modules in real time. The data transmitted in the service bus includes hardware status, configuration parameters, alarm information, notification messages and time information; The host management module is configured with the infrastructure information to be monitored, and selects the corresponding acquisition module for registration based on the device instance; it also receives the data transmitted on the service bus, monitors the infrastructure according to the status data, and sends action instructions through the service bus according to the monitoring results; The message module pushes the alarm information to the designated users or groups through the preset media according to the user and event type. The preset media includes one or more of SMS, email, system prompt and customized push; The action module is configured with predefined action rules and performs corresponding operations based on the alarm data received from the service bus in combination with the action rules.

10. The monitoring and automated fault handling system based on service bus according to claim 9, characterized in that: The system also includes a storage module and a user and group management module; The storage module is used to provide log storage, configuration storage and database interaction storage, and logs and stores all collected data, event processing records and configuration updates; The user and group management module is used to set the permissions and alarm notification media of different users, so that users at different levels obtain corresponding monitoring permissions and alarm messages.

Images