Private network state monitoring system and method based on kafka architecture

Abstract

The invention discloses a private network state monitoring system based on kafka architecture, and belongs to the technical field of local area network supervision, the system comprises a cloud server, an intermediate proxy server and a local area network private server, the cloud server is used for responding to a network state monitoring request initiated by a user side and echoing a diagnosis result; at least kafka message middleware is configured on the server; the intermediate proxy server is arranged between the cloud server and the plurality of local area network private servers, the intermediate proxy server is arranged in a double-network-port mode, a kafka service is configured on the intermediate proxy server, and the kafka service interacts with the kafka message middleware; a kafka-in-hand service module and a network diagnosis module are configured on the local area network private server, the kafka-in-hand service module interacts with the kafka message middleware, and the network monitoring module is used for monitoring the current network state; the invention further discloses a method adopting the system. The cloud server and the private network server are effectively combined through the kafka architecture, and real-time monitoring and diagnosis of the private server of the remote local area network are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of local area network supervision, and in particular relates to a private network status monitoring system and method based on a Kafka architecture. Background Art

[0002] With the deepening of enterprise informatization construction, more and more enterprises and organizations choose to build private networks to ensure data security and network stability. Private networks are usually used to protect the privacy of internal enterprise communications and effectively resist external network attacks. Due to the business needs of enterprises or organizations, private networks may be distributed in multiple geographical locations, which makes network management and monitoring more complicated. Existing network monitoring solutions include SNMP (Simple Network Management Protocol) and ICMP (Internet Control Message Protocol), but these technologies often have problems such as complex configuration, slow response, and poor real-time performance when dealing with multi-regional network monitoring. Especially when it is necessary to perform diagnostic tasks across networks, traditional methods are difficult to provide real-time diagnostic results due to the influence of network isolation and latency. In addition, existing solutions are difficult to integrate cloud computing capabilities to simplify and unify cross-regional network monitoring tasks.

[0003] Moreover, enterprise private networks are usually isolated from the external Internet, and traditional monitoring tools are difficult to directly access and perform network diagnostic tasks, resulting in delays and inaccuracies in monitoring results. In the case of multiple data centers, the configuration differences of different network environments make maintenance and management more difficult. The real-time performance of network monitoring is poor. When performing cross-regional network monitoring, traditional network monitoring methods are often affected by network delays and find it difficult to provide real-time network status information, affecting the ability to respond quickly to network failures. Due to the lack of a unified platform, existing monitoring tools are usually unable to effectively integrate cloud service capabilities, making it difficult to achieve centralized monitoring and management of multiple regions and multiple networks on a unified cloud platform, resulting in low monitoring efficiency.

[0004] Therefore, based on the existing technology, this application designs a private network status monitoring system and method based on the Kafka architecture. Summary of the invention

[0005] The purpose of the present invention is to provide a private network status monitoring system and method based on the Kafka architecture to solve the above-mentioned shortcomings in the prior art. Specifically, the present invention aims to effectively combine the cloud server with the private network server through the Kafka architecture as a bridge, so as to realize real-time monitoring and diagnosis of the network status of the remote server. The system can span different regions and network environments, providing enterprises with a unified and real-time network monitoring solution, thereby improving the efficiency and response speed of remote local area network management.

[0006] In order to solve the above problems, the present application provides a method for private network status monitoring based on Kafka architecture, comprising the following steps:

[0007] A cloud server, which is used to respond to a network status monitoring request initiated by a user terminal and to echo a diagnosis result, and is configured with at least a Kafka message middleware, and the Kafka message middleware creates two topics, PING-COMMAND and PING-RESULT;

[0008] A LAN private server, the LAN private server includes a plurality of servers, which are respectively set up in different places. At least a Kafka-in-handle service module and a network diagnosis module are configured on each LAN private server. The Kafka-in-handle service module interacts with the Kafka message middleware. The network diagnosis module can diagnose the network status of the current LAN according to the request.

[0009] An intermediate proxy server includes a group of intermediate proxy servers, which are arranged between the cloud server and multiple LAN private servers. The intermediate proxy server is set up with dual network ports and is configured with a Kafka service. The intermediate proxy server communicates with the cloud server and multiple LAN private servers respectively through the dual network ports, and transmits and processes data with the LAN private servers and the cloud server through the Kafka service.

[0010] As the preferred solution of this application:

[0011] The intermediate proxy server is also configured with an SSH service module to ensure secure remote access.

[0012] As the preferred solution of this application:

[0013] The network status monitoring request includes at least the following parameters of the target host: at least one of the device type, the device IP, the computer room to which the device belongs, the number of requests, and the request duration.

[0014] The present application also provides a method for monitoring the status of a private network based on the Kafka architecture. The method adopts any of the above-mentioned private network status monitoring systems based on the Kafka architecture, and specifically comprises the following steps:

[0015] Step 1) The cloud server constructs a PING request message according to the user request, and pushes the PING request message to the intermediate proxy server through the PING-COMMAND topic created by the Kafka message middleware;

[0016] Step 2) The intermediate proxy server respectively monitors the data sent by the LAN private server and the cloud server through its dual network ports, and uses the Kafka service to receive the PING request message pushed by the cloud server through the subscribed PING-COMMAND topic, parses the PING request message and forwards the PING request message to the corresponding LAN private server according to the parsed target host information;

[0017] Step 3) The kafka-in-handle service module on the LAN private server receives the PING request message through the subscribed PING-COMMAND topic, parses the PING request message and initiates PING diagnosis according to the parsed parameters, the network diagnosis module diagnoses and records the diagnosis results and encapsulates the diagnosis results into a PING diagnosis result message, and at the same time, pushes the PING diagnosis result message to the cloud server through the intermediate proxy server;

[0018] Step 4) The cloud server obtains the PING diagnostic result message pushed by the intermediate proxy server in real time through the Kafka message middleware and parses it, and the parsed diagnostic result is echoed on the client terminal.

[0019] As the preferred solution of this application:

[0020] In step 4), the cloud server uses a queue to receive and cache the PING diagnostic result message to achieve decoupling of the reception and processing of the PING diagnostic result message. At the same time, multi-threading or asynchronous IO is used to perform the reception and processing operations of the PING diagnostic result message in parallel.

[0021] As the preferred solution of this application:

[0022] In step 4), the cloud server aggregates the data after parsing the PING diagnostic result message, combines multiple diagnostic results from the same host into a whole, and displays the latest diagnostic result on the client terminal through the provided REST API interface.

[0023] As the preferred solution of this application:

[0024] In step 3), the specific steps of the LAN private server pushing the PING diagnosis result message to the cloud server through the intermediate proxy server include:

[0025] Step 31) The LAN private server pushes the PING diagnostic result message to the intermediate proxy server through the PING-RESULT topic subscribed by the kafka-in-handle service module;

[0026] Step 32) The intermediate proxy server receives the PING diagnosis result message through the subscribed PING-RESULT topic and pushes it to the cloud server;

[0027] As the preferred solution of this application:

[0028] The user request is sent through a WEB or APP terminal, and the user request includes at least one of the following parameters of the target host: device type, device IP, the computer room to which the device belongs, the number of requests, and the request duration.

[0029] As the preferred solution of this application:

[0030] The diagnosis result includes at least one of network traffic, network device status, delay and packet loss rate.

[0031] Compared with the prior art, the advantages of this application solution are:

[0032] The present application configures an intermediate proxy server at the front end of the LAN private server and the cloud server. The intermediate proxy server has a dual network port design, which can monitor the data sent by the LAN private server and the cloud server respectively. On the one hand, it realizes the networking of the cloud server and the LAN private servers in different places. On the other hand, it realizes the remote and secure access to the LAN private servers in different places through the cloud server, thereby improving the security of the system. In addition, the Kafka message middleware, Kafka-in-handle service module and Kafka service in the Kafka architecture are configured on the cloud server, the LAN private server and the intermediate proxy server respectively. By adding The kafka architecture can realize efficient and reliable transmission of data streams (the data stream includes PING requests and PING diagnostic results). At the same time, the high throughput, low latency and fault tolerance of the kafka architecture provide strong support for large-scale, distributed network status monitoring. It can be seen that in this solution of the present application, by setting up an intermediate proxy server and a kafka architecture between the cloud server and each LAN private server, not only a complete, closed-loop LAN monitoring system is formed, but also the network status of each LAN can be monitored in real time, reliably and accurately, thereby improving the efficiency and response speed of remote LAN supervision, and providing a practical basis and theoretical basis for the dynamic supervision of remote LANs. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is an architecture diagram of the private network status monitoring system based on the Kafka architecture provided by the present invention.

[0034] Figure 2 A flow chart of a method for private network status monitoring based on Kafka architecture provided by the present invention.

[0035] Figure 3 The present invention provides a complete message flow from a user sending a request to the final diagnosis result being displayed under the Kafka architecture. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below in conjunction with specific implementations and with reference to the accompanying drawings. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope of the present invention and its application.

[0037] like Figure 1The above is a private network status monitoring system architecture diagram based on the Kafka architecture provided in this embodiment, the system includes a cloud server, an intermediate proxy server and a LAN private server, wherein the cloud server is used to respond to the network status monitoring request initiated by the user end and to echo the diagnosis result, and at least a Kafka message middleware is configured thereon. The Kafka message middleware is the core of the Kafka architecture system, which can configure the master node of the Kafka cluster and create PING-COMMAND and PING-RESULT topics for message publishing and subscription, so as to ensure that the Kafka message middleware can process messages from various LANs in different regions. In this embodiment, the kafka message middleware can act as a producer or a consumer. When acting as a producer, it can construct a PING request message according to a user request. When acting as a consumer, it can be responsible for processing and analyzing data streams uploaded by LAN private servers in different regions. The LAN private servers include multiple ones, which are set up in different places. At least a kafka-in-handle service module and a network diagnosis module are configured on each LAN private server. The kafka-in-handle service module subscribes to the two topics of PING-COMMAND and PING-RESULT in the cloud server to realize the connection with the kafka fka message middleware; in this embodiment, the kafka-in-handle service module is both a consumer and a producer. When acting as a producer, it can generate a PING diagnostic result message based on the diagnostic result of the network diagnostic module, and when acting as a consumer, it can read messages from the subscribed kafka topic; the network diagnostic module can diagnose the network status of the current local area network according to the request. It can be understood that the network diagnostic module can be an existing conventional diagnostic tool to monitor at least one parameter in the current network, including network traffic, network device status, delay and packet loss rate; the intermediate proxy server includes a group and is set on the cloud server A group of intermediate proxy servers are used between the server and multiple LAN private servers, which not only realizes the secure connection between the LAN private servers in different places and the cloud server, but also facilitates maintenance. The intermediate proxy server is configured with a Kafka service, which interacts with the Kafka message middleware. The Kafka service is mainly a Kafka server and a client. In this embodiment, the function of the intermediate proxy server is equivalent to a jump server in a traditional network system. On the one hand, it realizes the networking of the cloud server and the LAN private servers in different places, and on the other hand, it realizes remote and secure access to the LAN private servers in different places through the cloud server, thereby improving the security of remote access to the system.In this embodiment, the Kafka service is configured as a consumer, which is responsible for subscribing and consuming PING request messages from the cloud server on the one hand, and for subscribing and consuming diagnostic data from private servers in various local area networks and sending them to the Kafka message middleware of the cloud server on the other hand. ;

[0038] In order to further improve the security of the operation of the intermediate proxy server, this embodiment also configures an SSH service module on the intermediate proxy server, which allows the cloud server to securely access each LAN private server to further improve the security of networking and data stream transmission.

[0039] When the system is implemented, the user sends a network status monitoring request to the cloud server through a web or app client. The request includes at least the following parameters of the target host: device type, device IP, the computer room to which the device belongs, number of requests and at least one of request duration; the cloud server generates a corresponding PING request message according to the network status monitoring request sent by the user and pushes it to the intermediate proxy server through the PING-COMMAND topic created by the kafka message middleware. The above parameters are encapsulated in the PING request message. The intermediate proxy server receives the PING request message pushed from the PING-COMMAND topic through the kafka service and parses it. According to the parameters parsed from the PING request message, the target host to be monitored is known, and then the PING request message is sent to the intermediate proxy server through the PING-COMMAND topic. The ND topic is pushed to the corresponding LAN private server. The corresponding LAN private server subscribes to the pushed PING request message through the kafka-in-handle service module and starts the network diagnosis module to perform network diagnosis according to the message content; the diagnosis result is pushed in the form of a message by the corresponding LAN private server through the PING-COMMAND topic subscribed by the kafka-in-handle service module to the kafka service of the intermediate proxy server. The intermediate proxy server then pushes the diagnosis result to the cloud server through the PING-COMMAND topic subscribed by the kafka service. The cloud server receives the diagnosis result through the kafka message middleware and parses, analyzes, and stores it, and then displays it on the corresponding web or app client, so that users can understand the real-time network status or query the historical network status.

[0040] It can be seen that the system of the present application not only forms a complete, closed-loop LAN monitoring system by setting up an intermediate proxy server and a Kafka architecture between the cloud server and each LAN private server, but also can monitor the network status of each LAN in real time, reliably and accurately, thereby improving the efficiency and response speed of off-site LAN supervision, and providing a practical basis and theoretical basis for the dynamic supervision of off-site LANs. In addition, the modular design makes it easy to add new LAN private servers or monitoring functions in the future, providing a basis for subsequent expansion; at the same time, this embodiment effectively utilizes the advantages of the Kafka architecture such as high throughput, low latency, high reliability and flexible expansion, ensures the real-time and reliability of data transmission, and improves the efficiency and response speed of off-site LAN management.

[0041] As a preferred embodiment of the present application, the present application also provides a method for monitoring the status of a private network based on the Kafka architecture, such as Figure 2-3 As shown, the monitoring method adopts the private network status monitoring system based on the Kafka architecture described in the above embodiment, and specifically includes the following steps:

[0042] Step 1) The cloud server constructs a PING request message according to the user request, and pushes the PING request message to the intermediate proxy server through the PING-COMMAND topic created by the Kafka message middleware; In this embodiment, the Kafka message middleware is the core of the Kafka architecture system, which can configure the master node of the Kafka cluster and create PING-COMMAND and PING-RESULT topics for message publishing and subscription to ensure that the Kafka message middleware can process message flows or data flows from various local area networks in different regions. In this embodiment, the Kafka message middleware can be used as a production It can also be used as a consumer. When it is a producer, it can construct a PING request message according to the user's request. When it is a consumer, it can be used to process and analyze the data stream uploaded by the LAN private server from different regions. In this step, the user sends a network status monitoring request to the cloud server through the web or app client. The request includes at least the following parameters of the target host: device type, device IP, the computer room to which the device belongs, the number of requests and the request duration. It can be understood that the constructed PING request message should also include at least one of the above parameters, so as to facilitate the acquisition of the target host to be monitored according to the parameters parsed from the PING request message.

[0043] Step 2) The intermediate proxy server respectively monitors the data sent by the LAN private server and the cloud server through its dual network ports, and uses the kafka service to receive the PING request message pushed by the cloud server through the subscribed PING-COMMAND topic, parses the PING request message and forwards the PING request message to the corresponding LAN private server according to the parsed target host information; in this step, only one group of intermediate proxy servers is set, and it is located between the cloud server and multiple LAN private servers. The use of a group of intermediate proxy servers not only realizes the secure connection between the LAN private servers in different locations and the cloud server, but also facilitates maintenance; the functions of the intermediate proxy server, etc. Similar to the jump server in the traditional network system, it realizes the networking between the cloud server and the private servers in the LAN in different places, and realizes the remote and secure access to the private servers in the LAN in different places through the cloud server, which improves the security of remote access. The Kafka service subscribes to the two topics of PING-COMMAND and PING-RESULT to interact with the Kafka message middleware. In this step, the Kafka service is configured as a consumer. It is responsible for subscribing to and consuming the PING request messages from the Kafka message middleware, and for subscribing to and consuming the diagnostic data from the private servers in the LAN in different places and sending them to the cloud server.

[0044] Step 3) The kafka-in-handle service module on the LAN private server receives the PING request message pushed from the intermediate proxy server through the subscribed PING-COMMAND topic, parses the PING request message and initiates PING diagnosis according to the parsed parameters, the network diagnosis module diagnoses and records the diagnosis results and encapsulates the diagnosis results into a PING diagnosis result message, and at the same time, pushes the PING diagnosis result message to the cloud server through the intermediate proxy server; in this embodiment, the LAN private server includes multiple, which are set up remotely and located in corresponding computer rooms, and the kafka-in-handle service module subscribes to the PING-COMMAND topic in the cloud server. AND and PING-RESULT are two topics to interact with the Kafka message middleware; in this embodiment, the Kafka-in-handle service module is both a consumer and a producer. When acting as a producer, it can generate a PING diagnostic result message based on the diagnostic result of the network diagnostic module; when acting as a consumer, it can read messages from the subscribed Kafka topics; the network diagnostic module can diagnose the network status of the current local area network according to the request. It can be understood that the network diagnostic module can be an existing conventional diagnostic tool, so as to be able to monitor at least one parameter in the current network including network traffic, network device status, delay and packet loss rate.

[0045] Step 4) The cloud server obtains the PING diagnostic result message pushed from the intermediate proxy server in real time through the PING-RESULT topic of the Kafka message middleware and parses it. The parsed diagnostic result is echoed on the client terminal.

[0046] In this step 4), the cloud server uses a queue to receive and cache PING diagnostic result messages to achieve decoupling of the reception and processing of PING diagnostic result messages. Specifically, in the consumer program, the queue Queue is used as a buffer of the processing flow to cache the received PING diagnostic result message data. The Kafka middleware, as a consumer, can continuously receive messages without being blocked by the processing logic, and uses multi-threading or asynchronous IO to process the received messages in parallel. One thread or asynchronous IO task is responsible for receiving messages and putting them into the queue Queue, and another thread or asynchronous IO is responsible for taking messages out of the queue Queue for processing, thereby achieving rapid data processing and enhancing the stability of data processing. In this embodiment, the cloud server aggregates the data after parsing the PING diagnostic result message, merges multiple diagnostic results from the same host into a whole, and echoes the latest diagnostic results on the client terminal through the provided REST API interface, so that users can view network status information in real time and obtain the latest diagnostic results, thereby enhancing user experience and system availability. It can be understood that the use of data aggregation can effectively reduce data redundancy and make the network status clearer.

[0047] In step 3) of this embodiment, the specific steps of the LAN private server pushing the PING diagnosis result message to the cloud server through the intermediate proxy server include:

[0048] Step 31) The LAN private server pushes the PING diagnostic result message to the intermediate proxy server through the PING-RESULT topic subscribed by the kafka-in-handle service module;

[0049] Step 32) The intermediate proxy server receives the PING diagnosis result message through the subscribed PING-RESULT topic and pushes it to the cloud server.

[0050] From the above analysis, it can be seen that this embodiment effectively combines the cloud server with the private network server through the Kafka architecture and the intermediate proxy server, forming a closed-loop monitoring process, and realizing real-time monitoring and diagnosis of the network status of the remote server. The system can span different regions and network environments to provide enterprises with a unified, real-time network monitoring solution. In addition, the Kafka architecture has the advantages of high throughput, low latency, high reliability, and flexible expansion. It is suitable for large-scale network monitoring, and can ensure the real-time and reliability of data transmission, improving the efficiency and response speed of remote local area network management.

[0051] The above is only an embodiment of the present invention, and the common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several improvements can be made without departing from the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of the claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A private network status monitoring system based on Kafka architecture, characterized in that: Include: A cloud server, which is used to respond to a network status monitoring request initiated by a user terminal and to echo a diagnosis result, and is configured with at least a Kafka message middleware, and the Kafka message middleware creates two topics, PING-COMMAND and PING-RESULT; A LAN private server, the LAN private server includes a plurality of servers, which are respectively set up in different places. At least a Kafka-in-handle service module and a network diagnosis module are configured on each LAN private server. The Kafka-in-handle service module interacts with the Kafka message middleware. The network diagnosis module can diagnose the network status of the current LAN according to the request. An intermediate proxy server includes a group of intermediate proxy servers, which are arranged between the cloud server and multiple LAN private servers. The intermediate proxy server is set up with dual network ports and is configured with a Kafka service. The intermediate proxy server communicates with the cloud server and multiple LAN private servers respectively through the dual network ports, and transmits and processes data with the LAN private servers and the cloud server through the Kafka service.

2. The private network status monitoring system based on Kafka architecture according to claim 1 is characterized in that: The intermediate proxy server is also configured with an SSH service module to ensure secure remote access.

3. The private network status monitoring system based on Kafka architecture according to claim 1 is characterized in that: The network status monitoring request includes at least the following parameters of the target host: at least one of the device type, the device IP, the computer room to which the device belongs, the number of requests, and the request duration.

4. A method for monitoring the status of a private network based on a Kafka architecture, characterized in that: The method adopts the private network status monitoring system based on the Kafka architecture as described in any one of claims 1 to 3 above, and specifically comprises the following steps: Step 1) The cloud server constructs a PING request message according to the user request, and pushes the PING request message to the intermediate proxy server through the PING-COMMAND topic created by the Kafka message middleware; Step 2) The intermediate proxy server respectively monitors the data sent by the LAN private server and the cloud server through its dual network ports, and receives the PING request message pushed by the cloud server by using the PING-COMMAND topic subscribed by the Kafka service, parses the PING request message and forwards the PING request message to the corresponding LAN private server according to the parsed target host information; Step 3) The kafka-in-handle service module on the LAN private server receives the PING request message through the subscribed PING-COMMAND topic, parses the PING request message and initiates PING diagnosis according to the parsed parameters, the network diagnosis module diagnoses and records the diagnosis results and encapsulates the diagnosis results into a PING diagnosis result message, and at the same time, pushes the PING diagnosis result message to the cloud server through the intermediate proxy server; Step 4) The cloud server obtains the PING diagnostic result message pushed by the intermediate proxy server in real time through the Kafka message middleware and parses it, and the parsed diagnostic result is echoed on the client terminal.

5. The method for monitoring the status of a private network based on the Kafka architecture according to claim 4 is characterized in that: In step 4), the cloud server uses a queue to receive and cache the PING diagnostic result message to achieve decoupling of the reception and processing of the PING diagnostic result message. At the same time, multi-threading or asynchronous IO is used to perform the reception and processing operations of the PING diagnostic result message in parallel.

6. The method for monitoring the status of a private network based on the Kafka architecture according to claim 4 is characterized in that: In step 4), the cloud server aggregates the data after parsing the PING diagnostic result message, combines multiple diagnostic results from the same host into a whole, and displays the latest diagnostic result on the client terminal through the provided REST API interface.

7. The method for monitoring the status of a private network based on the Kafka architecture according to claim 4 is characterized in that: In step 3), the specific steps of the LAN private server pushing the PING diagnosis result message to the cloud server through the intermediate proxy server include: Step 31) The LAN private server pushes the PING diagnostic result message to the intermediate proxy server through the PING-RESULT topic subscribed by the kafka-in-handle service module; Step 32) The intermediate proxy server receives the PING diagnosis result message through the subscribed PING-RESULT topic and pushes it to the cloud server.

8. The method for monitoring the status of a private network based on the Kafka architecture according to claim 4 is characterized in that: The user request is sent through a WEB or APP terminal, and the user request includes at least one of the following parameters of the target host: device type, device IP, the computer room to which the device belongs, the number of requests, and the request duration.

9. The method for monitoring the status of a private network based on the Kafka architecture according to claim 4 is characterized in that: The diagnosis result includes at least one of network traffic, network device status, delay and packet loss rate.

Images