A network IP monitoring method and system based on message transmission

Abstract

The application discloses a network IP monitoring method and system based on message transmission, wherein the system comprises a monitoring backstage, a monitoring middle stage and a monitoring foreground; the monitoring backstage comprises a data acquisition unit, a data storage, a message transmission intermediate and a message service processing module; the monitoring middle stage is provided with a monitoring service module; the data acquisition unit is in communication connection with the data storage and the message transmission intermediate respectively; the message transmission intermediate is in communication connection with the message service processing module; the message service processing module is in communication connection with the monitoring service module through a gateway component; and the monitoring service module is in communication connection with the monitoring foreground. The application can greatly improve the monitoring efficiency of IP websites and improve the limitations of the prior art.

Description

Technical Field

[0001] This invention relates to the field of network technology, specifically to a network IP monitoring method and system based on message transmission. Background Technology

[0002] The existing national IPv6 development monitoring platform (china-ipv6.cn) has overly simplistic inspection content and insufficient data depth. Furthermore, the website limits the number of daily inspections and relies on traditional HTTP request methods for IPv6 monitoring. Under the current technological environment, HTTP transmission in weak network conditions may encounter abnormalities and security issues. Summary of the Invention

[0003] Therefore, the technical problem to be solved by the present invention is to provide a network IP monitoring method and system based on message transmission, which can improve the monitoring efficiency of IP websites to a greater extent and overcome the limitations of the existing technology.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0005] A network IP monitoring system based on message transmission includes a monitoring backend, a monitoring middleware, and a monitoring frontend. The monitoring backend includes a data acquisition unit, a data storage unit, a message transmission middleware, and a message service processing module. The monitoring middleware is equipped with a monitoring service module. The data acquisition unit is communicatively connected to the data storage unit and the message transmission middleware. The message transmission middleware is communicatively connected to the message service processing module. The message service processing module is communicatively connected to the monitoring service module through a gateway component. The monitoring service module is communicatively connected to the monitoring frontend. The message transmission middleware is a NATS message service-based middleware, a Kafuka message middleware, or a RocketMQ message middleware. The message service processing module reads data from the message service in a listening mode, consumes messages according to the corresponding stream and subject, and ensures the accuracy of consumption through an ask mode.

[0006] The aforementioned network IP monitoring system based on message transmission includes a data acquisition unit comprising a probe component, a crawler component, and an open API interface component.

[0007] The aforementioned network IP monitoring system based on message transmission includes a monitoring mode configuration module and a monitoring alarm module in its monitoring platform.

[0008] The aforementioned network IP monitoring system based on message transmission includes a service subscription module in its monitoring front-end.

[0009] In the aforementioned network IP monitoring system based on message transmission, when the message transmission middleware is a NATS message service-based middleware, the data is fragmented and partitioned using NATS stream and subject when storing data in the data storage.

[0010] In the aforementioned network IP monitoring system based on message transmission, the configuration component in the message transmission middleware assigns a unique clientID and sets the clientID in the message header. The uniqueness of message data on the client is restricted by the corresponding key and unique value, as well as the corresponding message repetition time window.

[0011] The aforementioned network IP monitoring system based on message transmission also includes a reporting service module, a data interoperability service module, and a data profiling service module in the monitoring platform. When performing data interoperability, the data interoperability service module divides the scattered monitoring data by analogy and builds the data interoperability mode according to the principle of progressive layering.

[0012] A method for network IP monitoring using the aforementioned message transmission-based network IP monitoring system includes the following steps:

[0013] S1) Based on the monitoring service request, consume the message transmission queue tasks in the middleware and filter and distinguish the data through the gateway component;

[0014] S2) Load the data filtered in step S1) into the monitoring platform;

[0015] S3) The monitoring platform obtains the domain from step S2) by calling the Linux DNS service.

[0016] The IP address corresponding to the name;

[0017] S4) The monitoring platform uses the curl command to obtain information about the curl command in the data loaded in step S2);

[0018] S5) The monitoring platform performs a ping test on the IP address 5 obtained in step S3) by calling the Linux ping service;

[0019] S6) The monitoring platform calls the Linux telnat service to check the IP address obtained in step S3).

[0020] Verify whether the address was successfully telnat.

[0021] S7) The monitoring platform obtains the result of whether the authoritative DNS can resolve the IP address by querying;

[0022] S8) The monitoring platform obtains monitoring results based on the information obtained in steps S1) to S7) and feeds back the monitoring results to the monitoring front end.

[0023] In the above method, in step S4), the curl information is obtained by using multiple curl commands to verify whether the information obtained by the curl command was successful.

[0024] The above method retrieves data from web pages before filtering in step S1) using a web crawler component.

[0025] First, crawl the homepage elements and filter out useless URLs using regular expressions, then concatenate the correct URLs.

[0026] The technical solution of the present invention achieves the following beneficial technical effects:

[0027] 1. This invention is highly flexible and can enable data flow and coordination between services.

[0028] 2. The message-transmission-based network IP monitoring system in this invention has strong availability, and can be used through different...

[0029] All service outputs can be analyzed, summarized, and used to create closed-loop data analysis profiles. 3. This invention offers flexible segmentation, allowing for division by different regions / organizations and industry dimensions.

[0030] It can be further subdivided into a specific category of data, and the data has a high degree of analyzability and accuracy.

[0031] 4. A highly available message middleware service that is scalable and provides reliable data storage and a self-healing service model to ensure that monitored data is not lost. Attached Figure Description

[0032] Figure 1 This is a schematic diagram illustrating the working principle of the network IP monitoring system based on message transmission in this invention.

[0033] Figure 2 This is a flowchart for network IP monitoring based on message transmission. Detailed Implementation

[0034] The present invention will be further explained below with reference to examples.

[0035] like Figure 1As shown, the network IP monitoring system based on message transmission in this invention includes a monitoring backend, a monitoring middleware, and a monitoring frontend. The monitoring backend includes a data acquisition unit, a data storage unit, a message transmission middleware, and a message service processing module. The monitoring middleware is equipped with a monitoring service module. The data acquisition unit is communicatively connected to the data storage unit and the message transmission middleware. The message transmission middleware is communicatively connected to the message service processing module. The message service processing module is communicatively connected to the monitoring service module through a gateway component. The monitoring service module is communicatively connected to the monitoring frontend. The message transmission middleware is a NATS message service-based middleware, a Kafuka message middleware, or a RocketMQ message middleware, used for service push and data transmission by the data acquisition unit. The message service processing module reads data from the message service through a listening mode, consumes messages according to the corresponding stream and subject, and ensures the accuracy of consumption through an ask mode.

[0036] In this embodiment, the data acquisition unit includes a probe component, a crawler component, and an open API interface component. These components provide data support for the services of the monitoring platform. The monitoring platform is equipped with a monitoring method configuration module and a monitoring alarm module. The monitoring front end is equipped with a service subscription module and a visualization display module.

[0037] In the data storage, when the message transmission middleware is a NATS message service-based middleware, the data is fragmented and partitioned using NATS stream and subject when storing data.

[0038] The configuration component in the message transmission middleware assigns a unique clientID and sets the clientID in the message header. It also restricts the uniqueness of message data on the client by using the corresponding key and unique value, as well as the corresponding message repetition time window.

[0039] The monitoring platform also includes a reporting service module, a data interoperability service module, and a data profiling service module. When performing data interoperability, the data interoperability service module divides the scattered monitoring data by analogy and builds the data interoperability mode according to the principle of progressive layering.

[0040] In this invention, the monitoring front-end is an independent IP monitoring client; the monitoring mode configuration can be flexibly configured to achieve basic website IP monitoring, in-depth website IP monitoring, and continuous IP monitoring; when Kafuka message middleware or RocketMQ message middleware is used as the message transmission middleware, it can be pre-configured to have a high self-healing mode similar to NATS message service-based middleware, as well as a seamless upward or downward scaling mode, ensuring the high availability of the message transmission-based network IP monitoring system, while also making the message transmission-based network IP monitoring system have high monitoring efficiency for IP websites, thus improving the limitations of the prior art.

[0041] The aforementioned message-based network IP monitoring system can provide basic monitoring services, in-depth monitoring services, continuous monitoring services, monitoring and alarm services, and link monitoring services. The basic monitoring service monitors network IP data through probe components, allowing for automated monitoring configuration and customized probe frequencies. It also supports manual monitoring for real-time single-word IP data monitoring with unlimited monitoring attempts. The in-depth monitoring service utilizes multiple probe components for data monitoring and employs a web crawler component for full-link data capture, ensuring data integrity and fairness. Due to its longer detection cycle and wider reach, the in-depth monitoring service tracks different data return contents to guarantee data accuracy. The continuous monitoring service uses a random probe mode to monitor IP data. According to monitoring, by configuring monitoring strategies to push data on a minute-by-minute basis, and using sockets to build long connections, data services can be proactively pushed. Furthermore, continuous monitoring services and monitoring alarm services can be linked together to achieve continuous monitoring of data services, enabling timely handling of abnormal data and correction of detection links. The monitoring alarm module can issue alarms based on abnormal data detected during the monitoring service, and alarm information can be sent by setting thresholds. The link monitoring service is implemented by a crawler component that crawls the entire link of a website step by step.

[0042] In this invention, the network IP monitoring system based on message transmission provides reporting services through a reporting service module, including regional reports, industry reports, and overall reports. When providing reporting services, monitoring reports within the authorized scope can be obtained based on subscription attributes, or custom reports can be selected for specific regions / organizations within the authorized scope. Monitoring data from different industries can be compared within a region, enabling flexible selection, comparison, and more efficient data analysis, thereby achieving high availability of report content.

[0043] The available data interoperability services include industry data interoperability, operator data interoperability, and regional data interoperability. Specifically, the data interoperability service involves: breaking down fragmented monitoring data by analogy and building a data interoperability model according to a progressively layered principle, ensuring high data availability; comparing and interoperating within the scope of permissions; enabling interoperability and shared use of industry and regional data through analogy models; and integrating and interoperating operator node data, thereby achieving overall data connectivity and high availability, laying the foundation for subsequent data profiling.

[0044] The available data profiling services include real-time computing, a search engine, and a web profiling API. Specifically, the data profiling service involves: using real-time computing of monitoring data; constructing a feasibility analysis profile of IPv6 monitoring data for the corresponding region through summarization and comparative analysis of the monitoring data; integrating and summarizing the data through qualitative research; defining usable data categories using segmentation hypotheses; ensuring the completeness of data for each segment through quantitative data collection; and finally forming the corresponding data profile through cluster analysis. A search engine is also provided for rapid data retrieval and analysis, along with a web-based API interface for profiling generation and analysis.

[0045] like Figure 2 As shown, the method for network IP monitoring using the message transmission-based network IP monitoring system includes the following steps:

[0046] S1) Based on the monitoring service request, the message transmission queue task in the middleware is consumed and the data is filtered and distinguished by the gateway component; when crawling the web page before the data in this step is filtered by the crawler component, the home page elements are crawled first and the useless URLs are filtered by regular expressions, and then the correct URLs are concatenated.

[0047] S2) Load the data filtered in step S1) into the monitoring platform;

[0048] S3) The monitoring platform obtains the IP address corresponding to the domain name in the data loaded in step S2) by calling the Linux DNS service; when the IP address is an IPv4 address, the obtained IP address corresponding to the domain name is recorded as an A record; when the IP address is an IPv6 address, the obtained IP address corresponding to the domain name is recorded as an AAA record.

[0049] S4) The monitoring platform uses the curl command to obtain curl information from the data loaded in step S2); In this step, the curl information is obtained through multiple curl commands to verify whether the information is successfully obtained through the curl command.

[0050] S5) The monitoring platform performs a ping test on the IP address obtained in step S3) by calling the Linux ping service;

[0051] S6) The monitoring platform verifies whether the IP address obtained in step S3) was successfully telnat by calling the Linux telnat service;

[0052] S7) The monitoring platform obtains the result of whether the authoritative DNS can resolve the IP address by querying;

[0053] S8) The monitoring platform obtains monitoring results based on the information obtained in steps S1) to S7) and feeds back the monitoring results to the monitoring front end.

[0054] When retrieving information using the curl command, the command to retrieve website IPv4 protocol-related metrics using curl-4 is as follows:

[0055] curl -4 -o / dev / null-sw --- Server IP: [%{remote_ip}"]\n" --- Homepage response status code: %{http_code}"\n" --- DNS resolution time: %{time_namelookup} seconds"\n" --- TCP connection establishment time: %{time_connect} seconds"\n" --- Server response first packet latency: %{time_starttransfer} seconds"\n" --- Page response time: %{time_total} seconds"\n" --- Page load speed: %{speed_download} bytes / second"""http: / / www.xxx.cn".

[0056] The command to obtain website IPv6 protocol related metrics using curl-6 is:

[0057] curl -6 -o / dev / null-sw --- Server IP: [%{remote_ip}"]\n" --- Homepage response status code: %{http_code}"\n" --- DNS resolution time: %{time_namelookup} seconds"\n" --- TCP connection establishment time: %{time_connect} seconds"\n" --- Server response first packet latency: %{time_starttransfer} seconds"\n" --- Page response time: %{time_total} seconds"\n" --- Page load speed: %{speed_download} bytes / second""http: / / www.xxx.cn"

[0058] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of the claims of this patent application.

Claims

1. A network IP monitoring system based on message transmission, characterized in that, The system comprises a monitoring backend, a monitoring middleware, and a monitoring frontend. The monitoring backend includes a data acquisition unit, a data storage unit, a message transmission middleware, and a message service processing module. The monitoring middleware includes a monitoring service module. The data acquisition unit is communicatively connected to the data storage unit and the message transmission middleware. The message transmission middleware is communicatively connected to the message service processing module. The message service processing module is communicatively connected to the monitoring service module via a gateway component. The monitoring service module is communicatively connected to the monitoring frontend. The message transmission middleware is either a NATS-based message service middleware, a Kafuka message middleware, or a RocketMQ message middleware. The configuration component in the message transmission middleware assigns a unique client ID and sets the client ID in the message header. The uniqueness of message data on the client is restricted by the corresponding key, unique value, and a set message repetition time window. The message service processing module reads message service data in a listening mode, consumes messages according to the corresponding stream and subject, and ensures the accuracy of consumption through an ask mode. The data acquisition unit includes a probe component, a crawler component, and an open API interface component.

2. The network IP monitoring system based on message transmission according to claim 1, characterized in that, The monitoring platform is equipped with a monitoring mode configuration module and a monitoring alarm module.

3. The network IP monitoring system based on message transmission according to claim 1, characterized in that, The monitoring front-end has a service subscription module.

4. The network IP monitoring system based on message transmission according to claim 1, characterized in that, In the data storage, when the message transmission middleware is a NATS message service-based middleware, the data is fragmented and partitioned using NATS stream and subject when storing data.

5. The network IP monitoring system based on message transmission according to any one of claims 1 to 4, characterized in that, The monitoring platform also includes a reporting service module, a data interoperability service module, and a data profiling service module. When performing data interoperability, the data interoperability service module divides the scattered monitoring data by analogy and builds the data interoperability mode according to the principle of progressive layering.

6. A method for network IP monitoring using the message transmission-based network IP monitoring system as described in claim 1, characterized in that, Includes the following steps: S1) Based on the monitoring service request, consume the message transmission queue tasks in the middleware and filter and distinguish the data through the gateway component; S2) Load the data filtered in step S1) into the monitoring platform; S3) The monitoring platform obtains the IP address corresponding to the domain name in the data loaded in step S2) by calling the Linux DNS service; S4) The monitoring platform uses the curl command to obtain information about the curl command in the data loaded in step S2); S5) The monitoring platform performs a ping test on the IP address obtained in step S3) by calling the Linux ping service; S6) The monitoring platform verifies whether the IP address obtained in step S3) was successfully telnat by calling the Linux telnat service; S7) The monitoring platform obtains the result of whether the authoritative DNS can resolve the IP address by querying; S8) The monitoring platform obtains monitoring results based on the information obtained in steps S1) to S7) and feeds back the monitoring results to the monitoring front end.

7. The method according to claim 6, characterized in that, In step S4), the curl information is obtained through multiple curl commands to verify whether the information was successfully obtained through the curl command.

8. The method according to claim 6, characterized in that, When crawling data from a web page before filtering in step S1) using a crawler component, the homepage elements are first crawled and useless URLs are filtered out using regular expressions, and then the correct URLs are concatenated.

Images