A distributed service monitoring and automatic operation and maintenance processing system

Abstract

The present invention provides a distributed service monitoring and automatic operation and maintenance processing system, including a core service module, a business service module, a real-time data transmission module and a client, wherein the core service module is used to perform business service subscription and business service data distribution for the client, and to perform status monitoring, traffic load balancing and fault operation and maintenance on the nodes where each module and the client are located; the business service module is used to register business services with the core service module and provide business services; the real-time data transmission module is used to perform data preprocessing on the massive business service data transmitted in real time, and distribute the preprocessed data to the client; the client is used to register its own information with the core service module, subscribe to business services, and receive distributed business service data. The system can dynamically adjust the distribution nodes, reduce the pressure of network transmission, and realize the real-time transmission of massive data; and process the faulty nodes in sequence in the form of queues according to certain priorities according to the node types.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed automatic operation and maintenance processing, and in particular to a distributed service monitoring and automatic operation and maintenance processing system. Background Art

[0002] For complex business systems, various service applications are often managed in a unified manner through the form of service directories, a mechanism for accessing service resources is established, and user resources are managed at the same time. In the process of implementing the present invention, the inventor found that this framework is difficult to meet the business scenarios where multiple client nodes apply for massive data from the background and require real-time transmission. The real-time transmission of massive data also brings difficulties to service operation and maintenance. The prior art "A Brief Discussion on the Design and Implementation of Information-based Automatic Operation and Maintenance Application Platform" (Information Communication, Issue 8, 2017) briefly analyzes the relevant technologies for the development of automatic operation and maintenance application platforms and the functions of platform modules, and explains the basic design of the platform. However, the technical feature of this paper lies in the backup and restoration of the database. At present, there is no automatic operation and maintenance processing architecture for real-time transmission of massive data. Therefore, it is urgent to establish a distributed service monitoring and operation and maintenance processing framework that is adapted to the real-time transmission of massive data, which can uniformly monitor the service status of each node, adaptively allocate data distribution nodes according to network traffic status monitoring information, and automatically locate nodes with faults. The main and standby service switching is adopted to ensure that the system can normally push real-time data to the client, thereby reducing the operation and maintenance pressure for system operation and maintenance personnel. Summary of the invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a distributed service monitoring and automatic operation and maintenance processing system in view of the deficiencies in the prior art.

[0004] In order to solve the above technical problems, the present invention discloses a distributed service monitoring and automatic operation and maintenance processing system, including a core service module, a business service module, a real-time data transmission module and a client.

[0005] The core service module is used to perform business service subscription and business service data distribution for clients, and to perform status monitoring, traffic load balancing and fault operation and maintenance on business service modules, real-time data transmission modules and clients;

[0006] The business service module is used to register business services with the core service module and provide business services, and push its own traffic status and subscription information to the core service module in real time;

[0007] The real-time data transmission module is used to pre-process the massive business service data transmitted in real time and distribute the pre-processed data to the client; push its own traffic status and subscription information to the core service module in real time;

[0008] The client is used to register its own information with the core service module, subscribe to business services, receive distributed business service data; and push its own traffic status to the core service module in real time.

[0009] Furthermore, the core service module includes a subscription distribution unit, which is used for business service registration of the business service module, real-time data transmission module registration and client registration, and performs business service subscription and business service data distribution for the client;

[0010] The business service data distribution includes general business service data distribution with high reliability requirements and business service data distribution with high real-time requirements. The general business service data distribution with high reliability requirements includes: after the subscription distribution unit receives the subscription request from the client, it obtains the business service data from the business service module and forwards it to the client; the business service data distribution with high real-time requirements includes: after the subscription distribution unit receives the subscription request from the client, it calls the real-time data transmission module to pre-process the business service data, and the real-time data transmission module distributes the pre-processed business service data to the client.

[0011] Furthermore, the real-time data transmission module includes a data processing and distribution unit and a subordinate data distribution unit.

[0012] The data processing and distribution unit is used to pre-process the business service data with high real-time requirements, wherein the pre-processing includes data clipping; and distribute the pre-processed business service data to the lower-level data distribution unit;

[0013] The lower-level data distribution unit is used to distribute the pre-processed business service data to the client.

[0014] Furthermore, the core service module also includes a monitoring unit, which is used to monitor the status of the business service module, the real-time data transmission module and the client, including: each node where the business service module is located, each node where the real-time data transmission module is located and each node where the client is located establishes a heartbeat mechanism with the monitoring unit, and sends a heartbeat packet to the monitoring unit every heartbeat packet sending cycle, wherein the heartbeat packet includes the IP port of the current node, network traffic information, a preset heartbeat packet sending cycle and service subscription information; the monitoring unit performs heartbeat packet detection, and if the monitoring unit does not receive a node heartbeat packet within two heartbeat packet sending cycles, it determines that the node is disconnected, and it is necessary to restart the node or enable a standby node.

[0015] Furthermore, the heartbeat packet transmission period is greater than or equal to the data transmission period of the real-time data transmission module, that is, the data transmission step length, so as to realize the network service status monitoring of the real-time transmission module of massive data. The heartbeat transmission period of each node can be configured according to the demand, and the shortest time interval of the heartbeat packet (the minimum transmission period of the heartbeat packet T min ) should be equal to the data transmission cycle of the real-time data transmission module. The reason for this design is that in some business system application scenarios, the process of real-time data transmission is not continuous, but massive real-time data is updated in the system at a fixed frequency. This is reflected in the change of network traffic as the instantaneous upload traffic or instantaneous download traffic has a peak value according to a certain period. This peak traffic information is useful information that the background core service node needs to monitor. The content of network traffic data includes the upload traffic and download traffic per second at the current moment. Taking the case where the heartbeat packet sending period T is equal to the data transmission period of the real-time data transmission module as an example, the network traffic data in the heartbeat packet should be the maximum value of the traffic data per second within the heartbeat packet sending time interval. In actual business scenarios, the sending time interval of the heartbeat packet can be adjusted according to the real-time requirements of the service. The core service module obtains the network status information of each node by parsing the heartbeat packet as a quantitative basis for the subsequent automatic operation and maintenance processing.

[0016] Furthermore, when registering with a client, the subscription distribution unit needs to register information including the IP of the node where the client is located, the received transmission port and the role authority of the client; when subscribing to business services for the client, the corresponding business service interface of the business service module is first allocated and called according to the role authority and business needs of the client; when distributing business service data with high real-time requirements, the data processing distribution unit is first allocated according to the role authority of the client to perform data preprocessing, and the subordinate data distribution unit is allocated to distribute the preprocessed business service data to the client.

[0017] Furthermore, the network traffic information of the heartbeat packet of each node where the real-time data transmission module is located includes upload traffic data and download traffic data;

[0018] The core service module also includes a traffic load balancing unit, which is used to balance the traffic load of each node where the real-time data transmission module is located according to the upload traffic data and download traffic data of each node where the real-time data transmission module is located, as well as the network transmission pressure, including: the traffic load balancing adopts the principle of least connection to distribute the client's massive data distribution request to the subordinate distribution node with the least number of connections at present. The reason for selecting this method is that the core service module queries the node where the corresponding data processing and distribution unit is located according to the client request, and further sends it to the node where the subordinate data distribution unit is located after cutting the massive data according to the user's authority. For the node where the subordinate data distribution unit is located, the data that needs to be forwarded to each path is consistent. When a new client initiates a business service data subscription request with high real-time requirements to the core service module, the traffic load balancing unit first searches for the corresponding data processing and distribution unit according to the role authority registered by the client, and then allocates the current request to the node where the subordinate data distribution unit with the least number of connections is located. If there is no subordinate data distribution unit with the least number of connections or the traffic load of the currently available subordinate data distribution unit has reached the upper limit, a new subordinate data distribution unit node is allocated to the client based on the node registration information of the real-time data transmission module; when multiple clients initiate business service data subscription requests with high real-time requirements to the core service module, the traffic load balancing unit considers whether it is necessary to reallocate the node where the data processing and distribution unit is located and the node where the subordinate data distribution unit is located based on the node traffic information returned by the data distribution nodes at all levels in the heartbeat packets and the node registration information of the real-time data transmission module, according to the load of each node.

[0019] Furthermore, the core service module also includes an operation and maintenance unit, which is used to locate and maintain faulty nodes; after the monitoring unit detects that the heartbeat service of a node is stopped, the operation and maintenance unit locates the faulty node according to the traffic changes of other nodes, and performs operation and maintenance processing in the form of queues according to priority for the nodes where the data processing and distribution unit is located and the nodes where the client is located. The operation and maintenance processing includes the use of master-slave service for important service nodes, and completing the switching and restart of the faulty service nodes to ensure real-time data transmission and normal operation of the server.

[0020] Furthermore, after the monitoring unit detects that the heartbeat service of a certain node is stopped, the operation and maintenance unit locates the faulty node according to the traffic changes of other nodes, including:

[0021] The impact of client failure and offline on traffic monitoring at all levels: the heartbeat packets sent by the client to the core service module are stopped; the corresponding change in the traffic sent per second at the node where the lower-level data distribution unit is located; the change in the traffic sent per second at the node where the corresponding data processing and distribution unit is located; the impact of the failure and offline of the node where the lower-level data distribution unit is located on traffic monitoring at all levels: the heartbeat packets of all clients connected to it are sent normally, and the traffic data records in the client heartbeat packets are reduced compared with before; the heartbeat packets of the node where the lower-level data distribution unit that has failed and gone offline are located are stopped, and the heartbeat packets of other lower-level data distribution nodes are normal; the impact of the failure and offline of the node where the data processing and distribution unit is located on traffic monitoring at all levels: the heartbeat packets of the lower-level data distribution units and clients connected to it are sent normally, and the traffic records in the heartbeat packets are reduced compared with before; the heartbeat packets of the node where the data processing and distribution unit that has failed and gone offline are stopped, and the heartbeat packets of other data processing and distribution nodes are normal; the impact of the failure and offline of the node where the business service unit is located on traffic monitoring at all levels: the heartbeat packets of the node where the business service unit that has failed and gone offline are stopped, and the heartbeat packets of other nodes are normal.

[0022] The monitoring unit detects that a node still has not received new heartbeat data after two heartbeat packet sending cycles. The operation and maintenance unit makes a preliminary judgment on the fault situation based on the node type registered in the background, and then queries whether the traffic changes in the heartbeat packets of other related nodes are consistent with the impact of the client failure offline on traffic monitoring at all levels and the impact of the intermediate distribution node failure offline on traffic monitoring at all levels. Expectations are made to locate the faulty node.

[0023] Furthermore, the core service module adopts active-standby deployment, including a primary core service node and a backup core service node. The core service module adopts the primary core service node in normal operation, and the primary core service node sends heartbeat packet data containing its own status information to the backup core service node; when the backup core service node detects that the heartbeat data of the primary core service node stops sending, the business function of the primary core service node is temporarily switched to the backup core service node; after the primary core service node is restored, the business function is switched back to the primary core service node.

[0024] Beneficial effects:

[0025] This application aims at business scenarios that require both functional modules for real-time transmission of massive data and general data services that focus on data transmission reliability, and constructs a distributed service monitoring and automatic operation and maintenance processing framework. In addition to the subscription and distribution functions, the core service module is also responsible for traffic status operation and maintenance monitoring of the business service module, real-time data transmission module and client. It can uniformly monitor the service status of each node, adaptively allocate data distribution nodes according to network traffic status monitoring information, and automatically locate failed nodes. It adopts the form of primary and secondary service switching to ensure that the system can push real-time data to the client normally, reduce the operation and maintenance pressure for system operation and maintenance personnel, and realize unified management and automatic operation and maintenance processing of multiple functional services. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0027] Figure 1 A structural diagram of a distributed service monitoring and automatic operation and maintenance processing system provided in an embodiment of the present application.

[0028] Figure 2 A schematic diagram of network service monitoring based on heartbeat detection by a monitoring unit of a distributed service monitoring and automatic operation and maintenance processing system provided in an embodiment of the present application.

[0029] Figure 3 A workflow diagram of an operation and maintenance unit of a distributed service monitoring and automatic operation and maintenance processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0031] The distributed service monitoring and automatic operation and maintenance processing system provided in this application can be applied to application scenarios of real-time transmission of massive data.

[0032] The present application embodiment discloses a distributed service monitoring and automatic operation and maintenance processing system, such as Figure 1 As shown, it includes core service module, business service module, real-time data transmission module and client.

[0033] The core service module is used to perform business service subscription and business service data distribution for clients, and to perform status monitoring, traffic load balancing and fault operation and maintenance on business service modules, real-time data transmission modules and clients;

[0034] The business service module is used to register business services with the core service module and provide business services, and push its own traffic status and subscription information to the core service module in real time;

[0035] The real-time data transmission module is used to pre-process the massive business service data transmitted in real time and distribute the pre-processed data to the client; push its own traffic status and subscription information to the core service module in real time;

[0036] The client is used to register its own information with the core service module, subscribe to business services, receive distributed business service data; and push its own traffic status to the core service module in real time.

[0037] In this embodiment, the core service module includes a subscription distribution unit, which is used for business service registration of the business service module, real-time data transmission module registration and client registration, and performs business service subscription and business service data distribution for the client;

[0038] The business service data distribution includes general business service data distribution with high reliability requirements and business service data distribution with high real-time requirements. The general business service data distribution with high reliability requirements includes: after the subscription distribution unit receives the subscription request from the client, it obtains the business service data from the business service module and forwards it to the client; the business service data distribution with high real-time requirements includes: after the subscription distribution unit receives the subscription request from the client, it calls the real-time data transmission module to pre-process the business service data, and the real-time data transmission module distributes the pre-processed business service data to the client.

[0039] The real-time data transmission module includes a data processing and distribution unit and a lower-level data distribution unit.

[0040] The data processing and distribution unit is used to pre-process the business service data with high real-time requirements, wherein the pre-processing includes data clipping; and distribute the pre-processed business service data to the lower-level data distribution unit;

[0041] The lower-level data distribution unit is used to distribute the pre-processed business service data to the client.

[0042] When registering with a client, the subscription distribution unit needs to register information including the IP of the node where the client is located, the received transmission port and the role authority of the client; when subscribing to business services for the client, the corresponding business service interface of the business service module is first allocated and called according to the role authority and business needs of the client; when distributing business service data with high real-time requirements, the data processing distribution unit is first allocated according to the role authority of the client to perform data preprocessing, and the subordinate data distribution unit is allocated to distribute the preprocessed business service data to the client.

[0043] The subscription distribution unit can manage all registered business service modules (servers) and clients in a unified manner. The business service module must first register the business service in the core service module for the client to call the service. The client registers its own IP, receiving transmission port, client role authority and other information with the core service module, and subscribes to the corresponding business service from the core service module on demand. The business service here includes two forms. If the business service has no requirements for the amount of data transmitted and the real-time performance of data transmission, but has high requirements for reliability, after receiving the subscription request from the client, the core service module calls the corresponding business service module to obtain the corresponding data and push it to the client. If the business service has high requirements for the real-time performance of data transmission, such as real-time transmission service for massive data, the core service module will allocate the corresponding real-time data transmission module to perform data preprocessing and distribution on demand according to the role authority information registered by the client, notify the corresponding distribution node to send the preprocessed data to the client, and notify the client that sent the subscription request to receive real-time data from the corresponding lower-level data distribution unit node.

[0044] The client's subscription request is uniformly applied to the core service module. The core service module assigns the corresponding node where the business service module is located to provide services based on the role permissions and other information registered by the client. If it is a general business service with high reliability requirements, the core service module obtains the corresponding business service data from the node where the business service module is located and forwards it to the client that submitted the subscription application. For data transmission services with high real-time requirements, the core service module first calls the node where the data processing and distribution unit is located to perform data trimming and preliminary flow control based on the subscriber's permission information, and then assigns the node where the lower-level data distribution unit is located to forward the trimmed real-time data to the client. The real-time data is no longer forwarded through the core service module.

[0045] In this embodiment, the core service module also includes a monitoring unit, which is used to monitor the status of the business service module, the real-time data transmission module and the client, including: each node where the business service module is located, each node where the real-time data transmission module is located and each node where the client is located establishes a heartbeat mechanism with the monitoring unit, and sends a heartbeat packet to the monitoring unit every heartbeat packet sending cycle, wherein the heartbeat packet includes the IP port of the current node, network traffic information, a preset heartbeat packet sending cycle and service subscription information; the monitoring unit performs heartbeat packet detection, and if the monitoring unit does not receive a node heartbeat packet within two heartbeat packet sending cycles, it is determined that the node is disconnected, and the node needs to be restarted or a standby node needs to be enabled. The core service module itself establishes a set of master-slave service switching mechanisms, including the main core service node and the backup core service node. The core service module uses the main core service node in normal operation. The main core service node sends a heartbeat packet containing its own status information to the backup core service node. In the case of a failure of the main core service node and offline, the backup core service node temporarily assumes the business functions of the original main core service node. When it is detected that the status of the original main core service node has been restored (such as through device service restart, etc.), it switches back to the original main core service node.

[0046] The heartbeat packet sending period is greater than or equal to the data transmission period of the real-time data transmission module, and each node is configured as needed during registration.

[0047] The network traffic information of the heartbeat packets of each node where the real-time data transmission module is located includes upload traffic data and download traffic data;

[0048] For the distribution nodes at all levels that distribute massive real-time data, it is necessary to conduct statistical monitoring on the network traffic data of the distribution nodes at all levels. Here, the network traffic data of the distribution node is sent to the core service module in the form of a heartbeat packet according to the preset time interval for sending the heartbeat packet (heartbeat packet sending period T). The time interval for sending the heartbeat packet can be configured. The shortest time interval for sending the heartbeat packet should be equal to the data transmission period of the real-time data transmission module. The reason for this design is that in some business system application scenarios, the process of real-time data transmission is not continuous, but massive real-time data is updated in the system at a fixed frequency. This is manifested in the change of network traffic as the instantaneous upload traffic or instantaneous download traffic appears at a certain period. Peak traffic information is useful information that the background core service node needs to monitor. The content of the network traffic data includes the upload traffic and download traffic per second at the current moment. Taking the case where the sending time interval of the heartbeat packet is equal to the data transmission period of the real-time data transmission module as an example, the network traffic data in the heartbeat packet should be the maximum value of the traffic data per second within the heartbeat packet sending time interval. In the actual business scenario, the sending time interval of the heartbeat packet can be adjusted according to the real-time requirements of the service. The core service module obtains the network status information of each node by parsing the heartbeat packet, which serves as the quantitative basis for subsequent automatic operation and maintenance processing.

[0049] Considering the two situations of continuous transmission of real-time data and sending of massive data in a certain step size, the following is designed especially for the latter application scenario: Figure 2 The network traffic monitoring mechanism shown in the figure. Each service node needs to synchronize the minimum heartbeat period with the real-time data transmission step (data transmission period) in the configuration file according to the current application characteristics. Within the minimum heartbeat period, the upload and download traffic of the current node are counted every second and the maximum value is counted. Then, the traffic peak value of each minimum heartbeat period within the heartbeat packet sending period is averaged as the extraction of effective traffic information of massive real-time data. Then, the heartbeat packet sending period T is set according to the actual business needs, and the current node IP port, heartbeat packet sending period, traffic monitoring data, service subscription information and other effective data are packaged and sent to the core service module.

[0050] In this embodiment, the core service module also includes a traffic load balancing unit, which is used to balance the traffic load of each node where the real-time data transmission module is located according to the upload traffic data and download traffic data of each node where the real-time data transmission module is located, as well as the network transmission pressure. According to the heartbeat packet information returned by each node where the real-time data transmission module is located and each node where the client is located, the core service module can evaluate the network traffic load of each node where the real-time data transmission is located, and determine whether it is necessary to reallocate the data distribution node to the client that initiates the subscription request according to the traffic transmission and reception peak value of the current node, so as to achieve a network load balancing effect. The invention comprises: the traffic load balancing adopts the least connection principle. When a new client initiates a business service data subscription request with high real-time requirements to the core service module, the traffic load balancing unit first searches for the corresponding data processing and distribution unit according to the role authority registered by the client, and then allocates the current request to the node where the subordinate data distribution unit with the least number of connections is located. If there is no subordinate data distribution unit with the least number of connections or the traffic load of the currently available subordinate data distribution unit has reached the upper limit, a new subordinate data distribution unit node is allocated to the client according to the node registration information of the real-time data transmission module; when multiple clients initiate business service data subscription requests with high real-time requirements to the core service module, the traffic load balancing unit considers whether it is necessary to reallocate the node where the data processing and distribution unit is located and the node where the subordinate data distribution unit is located according to the node traffic information returned by the data distribution nodes at all levels in the heartbeat packets and the load of each node.

[0051] In this embodiment, the core service module also includes an operation and maintenance unit, which is used to locate and maintain faulty nodes; after the monitoring unit detects that the heartbeat service of a certain node is stopped (no heartbeat packets are received from other nodes within two heartbeat packet sending cycles), the operation and maintenance unit locates the position of the faulty node according to the traffic changes of other nodes, and performs operation and maintenance processing in the form of queues according to priority for the nodes where the data processing and distribution unit is located and the nodes where the client is located. The operation and maintenance processing includes the use of master-slave service for important service nodes to complete the switching and restart of the faulty service node.

[0052] After the monitoring unit detects that the heartbeat service of a node has stopped, the operation and maintenance unit locates the faulty node based on the traffic changes of other nodes, including:

[0053] The impact of client failure and offline on traffic monitoring at all levels: the heartbeat packets sent by the client to the core service module are stopped; the corresponding change in the traffic sent per second at the node where the lower-level data distribution unit is located; the change in the traffic sent per second at the node where the corresponding data processing and distribution unit is located; the impact of the failure and offline of the node where the lower-level data distribution unit is located on traffic monitoring at all levels: the heartbeat packets of all clients connected to it are sent normally, and the traffic data records in the client heartbeat packets are reduced compared with before; the heartbeat packets of the node where the lower-level data distribution unit that has failed and gone offline are located are stopped, and the heartbeat packets of other lower-level data distribution nodes are normal; the impact of the failure and offline of the node where the data processing and distribution unit is located on traffic monitoring at all levels: the heartbeat packets of the lower-level data distribution units and clients connected to it are sent normally, and the traffic records in the heartbeat packets are reduced compared with before; the heartbeat packets of the node where the data processing and distribution unit that has failed and gone offline are stopped, and the heartbeat packets of other data processing and distribution nodes are normal; the impact of the failure and offline of the node where the business service unit is located on traffic monitoring at all levels: the heartbeat packets of the node where the business service unit that has failed and gone offline are stopped, and the heartbeat packets of other nodes are normal. The monitoring unit detects that a node still has not received new heartbeat data after two heartbeat packet sending cycles. The operation and maintenance unit makes a preliminary judgment on the fault situation based on the node type registered in the background, and then queries whether the traffic changes in the heartbeat packets of other related nodes are consistent with the impact of the client failure offline on traffic monitoring at all levels and the impact of the intermediate distribution node failure offline on traffic monitoring at all levels. Expectations are made to locate the faulty node.

[0054] It is known that the failure of client nodes and data distribution nodes will produce different phenomena in heartbeat detection and the traffic monitoring data contained in the heartbeat packet. The most important failure phenomenon is that the heartbeat packet sent by the faulty node to the core service module stops sending, and the traffic data contained in the heartbeat packet may be affected by many factors, especially when multiple nodes fail to go offline. If the traffic data is taken into account, it is difficult to quickly locate the faulty node. Therefore, the main factor for locating the faulty node is set to the node where the background heartbeat packet stops sending.

[0055] like Figure 3As shown, after the core service module detects a node failure, it first determines whether the node is the node where the data processing and distribution unit is located, the node where the lower-level data distribution unit is located, or the node where the client is located, and adds it to the corresponding operation and maintenance processing queue according to the node type. The faulty nodes within the same type of queue are processed in sequence according to the monitoring arrival time. After all the faulty nodes in the queue with a higher priority are maintained, the faulty nodes in the queue with a lower priority are processed. For the core service module, the form of primary and backup service switching is adopted. The main core service node is equipped with a set of backup core service nodes. When the system is in normal operation, the main core service node sends heartbeat packet data to the backup core service node. When the backup core service node detects that the heartbeat data of the main core service node stops sending, the original main core service node is switched to the backup core service node to ensure the operation status of the system.

[0056] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a distributed service monitoring and automatic operation and maintenance processing system provided by the present invention and some or all of the steps in each embodiment can be executed. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0057] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention are essentially or partly contributed to the prior art can be embodied in the form of a computer program, i.e., a software product, which can be stored in a storage medium and includes several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, a MUU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0058] The present invention provides a distributed service monitoring and automatic operation and maintenance processing system. There are many methods and ways to implement the technical solution. The above is only a specific implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A distributed service monitoring and automatic operation and maintenance processing system, characterized in that: Including core service module, business service module, real-time data transmission module and client, The core service module is used to perform business service subscription and business service data distribution for clients, and to perform status monitoring, traffic load balancing and fault operation and maintenance on business service modules, real-time data transmission modules and clients; The business service module is used to register business services with the core service module and provide business services, and push its own traffic status and subscription information to the core service module in real time; The real-time data transmission module is used to pre-process the massive business service data transmitted in real time and distribute the pre-processed data to the client; Push its own traffic status and subscription information to the core service module in real time; The client is used to register its own information with the core service module, subscribe to business services, receive distributed business service data, and push its own traffic status to the core service module in real time; The core service module includes a subscription distribution unit, which is used for business service registration of the business service module, real-time data transmission module registration and client registration, and performs business service subscription and business service data distribution for the client; The business service data distribution includes business service data distribution with high real-time requirements, and the business service data distribution with high real-time requirements includes: after the subscription distribution unit receives the subscription request of the client, calling the real-time data transmission module to pre-process the business service data, and the real-time data transmission module distributes the pre-processed business service data to the client; When registering with a client, the subscription distribution unit needs to register information including the IP of the node where the client is located, the received transmission port and the role authority of the client; when subscribing to business services for the client, the corresponding business service interface of the business service module is first allocated and called according to the role authority and business needs of the client; when distributing business service data with high real-time requirements, the real-time data transmission module is first allocated according to the role authority of the client to perform data preprocessing, and the preprocessed business service data is distributed to the client.

2. A distributed service monitoring and automatic operation and maintenance processing system according to claim 1, characterized in that: The business service data distribution also includes general business service data distribution with high reliability requirements, and the general business service data distribution with high reliability requirements includes: after the subscription distribution unit receives the subscription request from the client, it obtains the business service data from the business service module and forwards it to the client.

3. A distributed service monitoring and automatic operation and maintenance processing system according to claim 2, characterized in that: The real-time data transmission module includes a data processing and distribution unit and a lower-level data distribution unit. The data processing and distribution unit is used to pre-process the business service data with high real-time requirements, wherein the pre-processing includes data clipping; and distribute the pre-processed business service data to the lower-level data distribution unit; The lower-level data distribution unit is used to distribute the pre-processed business service data to the client.

4. A distributed service monitoring and automatic operation and maintenance processing system according to claim 3, characterized in that: The core service module also includes a monitoring unit, which is used to monitor the status of the business service module, the real-time data transmission module and the client, including: each node where the business service module is located, each node where the real-time data transmission module is located and each node where the client is located establishes a heartbeat mechanism with the monitoring unit, and sends a heartbeat packet to the monitoring unit every heartbeat packet sending cycle, wherein the heartbeat packet includes the IP port of the current node, network traffic information, a preset heartbeat packet sending cycle and service subscription information; the monitoring unit performs heartbeat packet detection, and if the monitoring unit does not receive a node heartbeat packet within the heartbeat packet sending cycle T, it is determined that the node is disconnected, and the node needs to be restarted or a standby node needs to be enabled.

5. A distributed service monitoring and automatic operation and maintenance processing system according to claim 4, characterized in that: The heartbeat packet sending period is greater than or equal to the data transmission period of the real-time data transmission module, and the heartbeat packet sending period of each node is configured according to demand.

6. A distributed service monitoring and automatic operation and maintenance processing system according to claim 5, characterized in that: When distributing business service data with high real-time requirements, the subscription distribution unit first assigns a data processing distribution unit to preprocess the data according to the role authority of the client, and assigns a subordinate data distribution unit to distribute the preprocessed business service data to the client.

7. A distributed service monitoring and automatic operation and maintenance processing system according to claim 6, characterized in that: The network traffic information of the heartbeat packets of each node where the real-time data transmission module is located includes upload traffic data and download traffic data; The core service module also includes a traffic load balancing unit, which is used to balance the traffic load of each node where the real-time data transmission module is located according to the upload traffic data and download traffic data of each node where the real-time data transmission module is located, and the network transmission pressure, including: the traffic load balancing adopts the principle of least connection. When a new client initiates a business service data subscription request with high real-time requirements to the core service module, the traffic load balancing unit first searches for the corresponding data processing and distribution unit according to the role authority registered by the client, and then allocates the current request to the node where the subordinate data distribution unit with the least number of connections is located. If there is no subordinate data distribution unit with the least number of connections or the traffic load of the currently available subordinate data distribution unit has reached the upper limit, a new subordinate data distribution unit node is allocated to the client according to the node registration information of the real-time data transmission module; when more than two clients initiate business service data subscription requests with high real-time requirements to the core service module, the traffic load balancing unit considers whether it is necessary to reallocate the node where the data processing and distribution unit is located and the node where the subordinate data distribution unit is located according to the node traffic information returned by the data distribution nodes at all levels in the heartbeat packet and the node registration information of the real-time data transmission module according to the load of each node.

8. A distributed service monitoring and automatic operation and maintenance processing system according to claim 7, characterized in that: The core service module also includes an operation and maintenance unit, which is used to locate and maintain faulty nodes. After the monitoring unit detects that the heartbeat service of a node has stopped, the operation and maintenance unit locates the faulty node according to the traffic changes of other nodes, and performs operation and maintenance processing in the form of queues according to priority for the nodes where the data processing and distribution unit is located and the nodes where the client is located. The operation and maintenance processing includes the use of master-slave service for important service nodes and the switching and restarting of the faulty service nodes.

9. A distributed service monitoring and automatic operation and maintenance processing system according to claim 8, characterized in that: After the monitoring unit detects that the heartbeat service of a node has stopped, the operation and maintenance unit locates the faulty node based on the traffic changes of other nodes, including: Impact of client failure and offline on traffic monitoring at all levels: heartbeat packets sent by the client to the core service module are stopped; the corresponding change in the traffic sent per second at the node where the lower-level data distribution unit is located; the change in the traffic sent per second at the node where the corresponding data processing and distribution unit is located; impact of the failure and offline of the node where the lower-level data distribution unit is located on traffic monitoring at all levels: heartbeat packets of all clients connected to it are sent normally, and the traffic data records in the client heartbeat packets are reduced compared with before; heartbeat packets of the node where the lower-level data distribution unit that has failed and offline is located are stopped, and the heartbeat packets of other lower-level data distribution nodes are normal; impact of the failure and offline of the node where the data processing and distribution unit is located on traffic monitoring at all levels: heartbeat packets of the lower-level data distribution units and clients connected to it are sent normally, and the traffic records in the heartbeat packets are reduced compared with before; heartbeat packets of the node where the data processing and distribution unit that has failed and offline is located are stopped, and the heartbeat packets of other data processing and distribution unit nodes are normal; impact of the failure and offline of the node where the business service unit is located on traffic monitoring at all levels: heartbeat packets of the node where the business service unit that has failed and offline is located are stopped, and the heartbeat packets of other nodes are normal; The monitoring unit detects that a node still has not received new heartbeat data after two heartbeat packet sending cycles. The operation and maintenance unit makes a preliminary judgment on the fault situation based on the node type registered in the background, and then queries whether the traffic changes in the heartbeat packets of other related nodes are consistent with the impact of the client failure offline on traffic monitoring at all levels and the impact of the intermediate distribution node failure offline on traffic monitoring at all levels. Expectations are made to locate the faulty node.

10. A distributed service monitoring and automatic operation and maintenance processing system according to claim 9, characterized in that: The core service module adopts active-standby deployment, including active core service nodes and standby core service nodes. The core service module adopts active core service nodes in normal operation, and the active core service nodes send heartbeat packet data containing their own status information to the standby core service nodes; When the backup core service node detects that the heartbeat data of the main core service node stops sending, the business function of the main core service node is temporarily switched to the backup core service node; After the main core service node is restored, the business functions will be switched back to the main core service node.

Images