A multi-dimensional streaming centralized monitoring method and system for comprehensive meteorological data

Abstract

The present invention provides a multi-dimensional streaming centralized monitoring method and system for comprehensive meteorological data. The multi-dimensional streaming centralized monitoring method for comprehensive meteorological data includes: judging the type of meteorological data according to different data timeliness constraints, wherein the types of the meteorological data include periodic timed meteorological data, interval timed meteorological data, and non-fixed time meteorological data; obtaining the effective time according to the meteorological data type; and performing diversity data verification on different types of meteorological data obtained by using different monitoring protocols to complete the data monitoring of various types of meteorological data. The system includes modules corresponding to the method steps.

Description

Technical Field

[0001] The present invention provides a multi - dimensional streaming centralized monitoring method and system for comprehensive meteorological data, belonging to the technical field of comprehensive meteorological data monitoring. Background Art

[0002] In the fields of meteorological observation and data analysis, with the rapid development of sensor technology and network communication technology, the capabilities of meteorological data collection, transmission, and processing have been significantly improved. However, due to the wide variety, extensive sources, and strong timeliness of meteorological data, how to efficiently and accurately monitor these multi - dimensional and streaming meteorological data has become an urgent problem to be solved. Meteorological data can be classified into various types according to their collection methods and update frequencies, mainly including periodic timed meteorological data, interval timed meteorological data, and non - fixed - time meteorological data. These data types have different characteristics respectively:

[0003] Periodic timed meteorological data: Such as daily temperature, humidity, precipitation, etc. These data are collected and updated at fixed time intervals (such as every hour, every day), and have strong regularity and predictability.

[0004] Interval timed meteorological data: Such as wind direction and wind speed within a specific time period. These data are continuously collected within a specific time period, but not strictly at fixed time intervals, and their effective time range is determined according to actual needs.

[0005] Non - fixed - time meteorological data: Such as data of sudden meteorological events (such as tornadoes, hailstones). The occurrence time of these data is unpredictable, but once they occur, they need to be collected and processed immediately.

[0006] Currently, the monitoring methods for meteorological data mainly rely on traditional data verification and storage technologies. These methods have the following limitations when dealing with multi - dimensional and streaming meteorological data:

[0007] Low data verification efficiency: Traditional data verification methods usually need to check all received data one by one. This method is inefficient when dealing with large - scale and high - frequency meteorological data and is difficult to meet the real - time requirements.

[0008] Inaccurate data type identification: Due to the diverse types of meteorological data, traditional monitoring methods often have difficulty accurately identifying different types of data, resulting in reduced accuracy and reliability of data processing.

[0009] Imprecise effective time judgment: For periodic timed and interval timed meteorological data, the judgment of their effective time directly affects the timeliness and availability of data. Traditional monitoring methods may not be able to accurately judge the effective time range of these data, resulting in data waste or delayed processing. Summary of the Invention

[0010] The present invention provides a multi - dimensional streaming centralized monitoring method and system for comprehensive meteorological data to solve the technical problems existing in the above - mentioned prior art. The technical solutions adopted are as follows:

[0011] A multi - dimensional streaming centralized monitoring method for comprehensive meteorological data, the multi - dimensional streaming centralized monitoring method for comprehensive meteorological data includes:

[0012] Judge the type of meteorological data and obtain the valid time according to the meteorological data type;

[0013] Use different monitoring protocols to perform diversity data verification on different types of meteorological data obtained, and complete the data monitoring of various types of meteorological data.

[0014] Further, judging the type of meteorological data and obtaining the valid time according to the meteorological data type includes:

[0015] Judge the type of meteorological data according to different data currency constraints, wherein the types of the meteorological data include periodic timed meteorological data, interval - timed meteorological data, and non - fixed - time meteorological data;

[0016] When the type of the meteorological data is periodic timed meteorological data, determine the valid time corresponding to the periodic timed meteorological data;

[0017] When the type of the meteorological data is interval - timed meteorological data, determine the valid time corresponding to the interval - timed meteorological data.

[0018] Further, when the type of the meteorological data is periodic timed meteorological data, determining the valid time corresponding to the periodic timed meteorological data includes:

[0019] When the type of the meteorological data is periodic timed meteorological data, then judge the data parameter format;

[0020] Obtain the minimum time and query period within the data delay time according to the data parameter format;

[0021] Take the minimum time as the time starting point, and load the meteorological data corresponding to the valid time in units of the data parameter format;

[0022] After the meteorological data is loaded, perform n - loop traversal scans to judge whether there is periodic timed meteorological data within the valid time.

[0023] Further, when the type of the meteorological data is interval - timed meteorological data, determining the valid time corresponding to the interval - timed meteorological data includes:

[0024] When the type of the meteorological data is interval-timed meteorological data, obtain the data valid time after subtracting the data delay time from the current time;

[0025] Determine whether the time interval in which the data valid time is located is less than the minimum interval;

[0026] When the time interval in which the data valid time is located is less than the minimum interval, extract the maximum interval of the previous day as the valid time;

[0027] When the time interval in which the data valid time is located is greater than the minimum interval, extract the time interval in which the data valid time is located as the valid time;

[0028] Load the meteorological data corresponding to the valid time, and determine whether there is interval-timed meteorological data within the valid time.

[0029] Further, the protocol types corresponding to each type of meteorological data include the SMB protocol, the SHH protocol, the TCP protocol, the FTP protocol, and the HTTP protocol; among them, the TCP protocol includes SQL Server, PostgreSQL, MySql, and Oracle.

[0030] Further, the monitoring rules corresponding to each protocol type are as follows:

[0031] The Sql statement rules corresponding to the monitoring principle of the database data corresponding to the monitoring rules include: data server ID monitoring, Sql statement monitoring, data timeliness monitoring, data count monitoring, data table name monitoring, and time field monitoring;

[0032] The file rules corresponding to the monitoring principle of the file data corresponding to the monitoring rules include: data server ID monitoring, file path monitoring or file modification time monitoring, whether it is associated with the element cycle monitoring, data timeliness monitoring, and file size monitoring;

[0033] The rules corresponding to the HTTP data corresponding to the monitoring rules include the DATA set rule, the null value rule, and the 404 error rule;

[0034] Among them, the DATA set rule includes URL monitoring, parameter monitoring, parameter rule monitoring, whether it is associated with the data cycle monitoring, and data timeliness monitoring;

[0035] The null value rule includes URL monitoring, parameter monitoring, parameter rule monitoring, whether it is associated with the data cycle monitoring, and data timeliness monitoring;

[0036] The 404 error rule includes URL monitoring, parameter monitoring, parameter rule monitoring, whether it is associated with the data cycle monitoring, and data timeliness monitoring.

[0037] Furthermore, different monitoring protocols are used to perform diversity data verification on different types of meteorological data to complete data monitoring of various types of meteorological data, including:

[0038] When there is periodic timed meteorological data within the effective time, the data status of the data node corresponding to the periodic timed meteorological data is judged, and the data status of the data node is recorded;

[0039] When there is interval timed meteorological data within the effective time, the data status of the data node corresponding to the interval timed meteorological data is judged, and the data status of the data node is recorded;

[0040] When the type of the meteorological data is non-fixed-time meteorological data, the meteorological data is loaded to judge whether there is non-fixed-time meteorological data; when there is non-fixed-time meteorological data, the data status of the data node corresponding to the non-fixed-time meteorological data is judged, and the data status of the data node is recorded.

[0041] Furthermore, judging the data status of the data nodes corresponding to the periodic timed meteorological data, interval timed meteorological data, and non-fixed-time meteorological data includes:

[0042] Starting from the first node of the data stream in turn by means of a recursive algorithm until the end point of the data stream, the data status corresponding to each layer of data nodes is obtained;

[0043] The data status corresponding to the data nodes is returned layer by layer, and finally the monitoring status of the entire data process is returned.

[0044] Furthermore, the multi-dimensional streaming centralized monitoring method for comprehensive meteorological data further includes:

[0045] Real-time monitoring of the data node operation parameters corresponding to each data node; wherein, the data node operation parameters include memory utilization rate, network bandwidth occupancy rate, data processing error rate, and the number of acquisition processes per unit time;

[0046] Using the memory utilization rate and network bandwidth occupancy rate to obtain the first node operation evaluation coefficient; wherein, the first node operation evaluation coefficient is obtained through the following formula:

[0047]

[0048] where C 01 represents the first node operation evaluation coefficient; n represents the number of unit times experienced by the data node operation; M p represents the average value of the memory utilization rate corresponding to n unit times; M zRepresents the median memory utilization corresponding to n unit time periods; M min Represents the minimum memory utilization corresponding to n unit time periods; M i Represents the memory utilization corresponding to the i-th unit time; B p Represents the average network bandwidth occupancy rate corresponding to n unit time periods; B z Represents the median network bandwidth occupancy rate corresponding to n unit time periods; B min Represents the minimum network bandwidth occupancy rate corresponding to n unit time periods; B i Represents the network bandwidth occupancy rate corresponding to the i-th unit time; S 01 and S 02 Respectively represent the first adjustment coefficient and the second adjustment coefficient, and the first adjustment coefficient and the second adjustment coefficient are obtained through the following formula:

[0049]

[0050] where S 01 and S 02 Respectively represent the first adjustment coefficient and the second adjustment coefficient; k 01 and k 02 Respectively represent the weight values corresponding to the preset memory utilization and network bandwidth occupancy rate; f b Represents the standard deviation of the network bandwidth occupancy rate corresponding to n unit time periods; f m Represents the standard deviation of the memory utilization corresponding to n unit time periods;

[0051] Compare the first node operation evaluation coefficient with a preset first evaluation threshold;

[0052] When the first node operation evaluation coefficient is lower than the preset first evaluation threshold, then retrieve the number of acquisition processes within a unit time;

[0053] Use the number of acquisition processes within a unit time and the first node operation evaluation coefficient to obtain a second node operation evaluation coefficient; where the second node operation evaluation coefficient is obtained through the following formula:

[0054]

[0055] where C 02 Represents the second node operation evaluation coefficient; n represents the number of unit time periods experienced by the data node operation; P i Represents the number of acquisition processes within a unit time corresponding to the i-th unit time; P c Represents a reference value of the number of acquisition processes within a preset unit time; f b Represents the standard deviation of the network bandwidth occupancy rate corresponding to n unit time periods; f mrepresents the standard deviation of memory utilization corresponding to n unit time intervals; f p represents the standard deviation of the number of acquisition processes corresponding to n unit time intervals; C 01 represents the running evaluation coefficient of the first node; C y01 represents a preset first evaluation threshold; P p represents the average value of the number of acquisition processes corresponding to n unit time intervals; P z represents the median value of the number of acquisition processes corresponding to n unit time intervals;

[0056] Compare the running evaluation coefficient of the second node with a preset second evaluation threshold;

[0057] When the running evaluation coefficient of the second node exceeds the preset second evaluation threshold, it is determined that there is an abnormality in the operation of the data node, and an alarm for the abnormal operation of the data node is given.

[0058] A multi-dimensional streaming centralized monitoring system for comprehensive meteorological data, the multi-dimensional streaming centralized monitoring system for comprehensive meteorological data includes:

[0059] An effective time acquisition module, configured to determine the type of meteorological data and acquire the effective time according to the meteorological data type;

[0060] A data monitoring module, configured to perform diversity data verification on different types of meteorological data acquired by using different monitoring protocols, and complete data monitoring of various types of meteorological data.

[0061] Advantages of the present invention:

[0062] A multi-dimensional streaming centralized monitoring method and system for comprehensive meteorological data proposed by the present invention. There are a large number of types, different structures, and diverse data transmission methods for comprehensive meteorological products, resulting in complexity in the daily use of meteorological products from collection to multi-platform applications. Based on a multi-dimensional algorithm for data timeliness judgment and data diversity verification, this project studies the streaming monitoring of diversified comprehensive meteorological data, realizes the chain management of multiple links such as collection, processing, and application of comprehensive products, sorts out and forms a clear product application link network, and data anomalies can be timely fed back through the nodes of the link network, facilitating staff to quickly locate abnormal information points and improving the efficiency of data guarantee. Description of the Drawings

[0063] Figure 1 is a flowchart of the method described in the present invention;

[0064] Figure 2 is a flowchart corresponding to the acquisition of the effective time described in the present invention;

[0065] Figure 3 is a protocol schematic diagram of the present invention;

[0066] Figure 4 This is the schematic diagram of the monitoring rules of the present invention;

[0067] Figure 5 This is the flowchart of the data monitoring process of the present invention;

[0068] Figure 6 This is the system block diagram of the system of the present invention. Detailed implementation manners

[0069] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for the purpose of illustrating and explaining the present invention, and are not used to limit the present invention.

[0070] An embodiment of the present invention provides a multi-dimensional streaming centralized monitoring method for comprehensive meteorological data. As Figure 1 shown, the multi-dimensional streaming centralized monitoring method for comprehensive meteorological data includes:

[0071] S1. Determine the type of meteorological data, and obtain the valid time according to the type of meteorological data;

[0072] S2. Use different monitoring protocols to perform diversity data verification on different types of meteorological data obtained, and complete the data monitoring of various types of meteorological data.

[0073] The working principle of the above technical solution is as follows: For comprehensive meteorological data (including meteorological data, cross-industry data, etc.) of different types, different transmission methods, and different formats, streaming monitoring is realized from each node such as collection, processing, and application, and a data link diagram is formed in a configurable manner, enabling staff to quickly locate the node with data anomalies and discover the problem link in a timely manner.

[0074] The meteorological data transmission methods of the above technical solution mainly include HTTP protocol data, TCP protocol data, SMB protocol data, FTP protocol data, and SSH protocol. Different data types require different request protocols, and for the data formats corresponding to different data types, different reading and parsing methods are used to analyze the data attributes, and finally the verification of diversity data is completed.

[0075] Meteorological data has a diverse structure and large differences. In order to achieve normalized monitoring, this project will determine the timeliness of the data. According to different data timeliness constraints (Currency Constraint), meteorological data will be divided into three types: periodic timing (such as real-time data of automatic stations every 5 minutes), interval timing (such as short-term weather from 4:50 to 21:50 every day), and no fixed time (such as early warning information). In the process of calculating data timeliness, the timeliness calculation function is used to classify the data, and the corresponding relationship between the current monitoring time and the data timeliness and data cycle is matched. When the time point exceeds the maximum monitoring timeliness, the set verification time and the valid time are cyclically verified, and finally the data timeliness monitoring algorithm within the valid time period is realized.

[0076] Through the recursive algorithm, starting from the first node of the data flow and recursively until the end point of the data flow, the data status of the data node is returned layer by layer, and finally the monitoring status of the entire data flow is returned.

[0077] The above technical solutions proposed in this implementation focus on solving technical difficulties and key issues.

[0078] 1. In the timeliness judgment algorithm, how to judge whether the data is effectively updated based on the differences in meteorological products.

[0079] 2. Research on data flow monitoring algorithms, how to use technical means to form a cross-correlated data relationship link diagram.

[0080] The effect of the above technical solution is: by subdividing the meteorological data types (periodic timing, interval timing, no fixed time), and combining the data time constraints (Currency Constraint), the validity and timeliness of the data can be judged more accurately. This helps to reduce decision-making errors caused by outdated or erroneous data and improve the accuracy and reliability of meteorological services.

[0081] By utilizing diverse data verification protocols, we conduct comprehensive and detailed verification of different types of meteorological data to ensure the integrity and accuracy of the data and further improve the quality of data monitoring.

[0082] Supporting multiple data transmission protocols (HTTP, TCP, SMB, FTP, SSH, etc.), the system can flexibly access meteorological data from different sources and in different formats, enhancing the compatibility and scalability of the system. By forming a data link diagram in a configurable way, the staff can intuitively understand the path and status of data flow, facilitate rapid location of problem nodes, and improve the flexibility and operability of data monitoring.

[0083] By adopting the streaming monitoring technology, the real-time collection, processing and monitoring of meteorological data are realized, reducing the delay and waiting time of data processing, and improving the efficiency and performance of data monitoring. By recursively returning the status of data nodes layer by layer, the monitoring results of the entire data process can be quickly summarized, providing timely and comprehensive data support for decision-makers.

[0084] By introducing artificial intelligence technologies such as machine learning and deep learning, the intelligent recognition and analysis of meteorological data are carried out, improving the accuracy and efficiency of data type recognition, and providing a more intelligent means for data monitoring. By constructing a cross-correlated data relationship link diagram, the internal connections and laws between data can be revealed, providing more comprehensive and in-depth information support for meteorological prediction and decision-making. The efficient data monitoring and verification mechanism can detect and process data anomalies in a timely manner, reducing system failures and operation and maintenance costs caused by data errors. The flexible monitoring configuration and visual data link diagram reduce the operation difficulty and error rate of operation and maintenance personnel, and improve the stability and reliability of the system.

[0085] An embodiment of the present invention, as Figure 2 shown, determines the type of meteorological data and obtains the valid time according to the meteorological data type, including:

[0086] S101. Determine the type of meteorological data according to different data currency constraints, where the types of the meteorological data include periodic timed meteorological data, interval timed meteorological data, and non-fixed time meteorological data;

[0087] S102. When the type of the meteorological data is periodic timed meteorological data, determine the valid time corresponding to the periodic timed meteorological data;

[0088] S103. When the type of the meteorological data is interval timed meteorological data, determine the valid time corresponding to the interval timed meteorological data.

[0089] Specifically, when the type of the meteorological data is periodic timed meteorological data, determining the valid time corresponding to the periodic timed meteorological data includes:

[0090] S1021. When the type of the meteorological data is periodic timed meteorological data, judge the data parameter format;

[0091] S1022. Obtain the minimum time and query period within the data delay time according to the data parameter format;

[0092] S1023. Starting from the minimum time as the time starting point and taking the data parameter format as the unit, load the meteorological data corresponding to the valid time;

[0093] S1024. After the meteorological data is loaded, perform n - loop traversal scans to determine whether there is periodic - timed meteorological data within the valid time.

[0094] Meanwhile, when the type of the meteorological data is interval - timed meteorological data, determining the valid time corresponding to the interval - timed meteorological data includes:

[0095] S1031. When the type of the meteorological data is interval - timed meteorological data, obtain the data valid time after subtracting the data delay time from the current time;

[0096] S1032. Judge whether the time interval where the data valid time is located is less than the minimum interval;

[0097] S1033. When the time interval where the data valid time is located is less than the minimum interval, extract the maximum interval of the previous day as the valid time;

[0098] S1034. When the time interval where the data valid time is located is greater than the minimum interval, extract the time interval where the data valid time is located as the valid time;

[0099] S1035. Load the meteorological data corresponding to the valid time, and judge whether there is interval - timed meteorological data within the valid time.

[0100] The working principle of the above - mentioned technical solution is as follows: Figure 2 As shown, meteorological data has characteristics such as complex sources and diverse data structures, making it increasingly difficult to effectively manage the data. This project aims to study and implement the streaming monitoring of comprehensive meteorological data, and needs to solve problems such as data timeliness determination algorithms, diverse data verification algorithms, and data stream monitoring algorithms.

[0101] In the data timeliness determination algorithm, according to different data currency constraints, meteorological data is divided into three types: periodic - timed (such as automatic weather station live data every 5 minutes), interval - timed (such as short - term weather from 4:50 to 21:50 every day), and no fixed time (such as warning information). During the data timeliness calculation process, use the timeliness calculation function cyc() to classify the data, and match the corresponding relationship between the current monitoring time, data timeliness, and data period. For time points beyond the maximum monitoring timeliness, through the set verification times and cyclic verification of the valid times, finally, the data timeliness monitoring algorithm within the valid time period is realized.

[0102] It is necessary to determine the data type. For real-time type data, directly load the data connection driver to obtain real-time meteorological data; for the periodic timing type, first calculate the time format of the data parameters. Generally, the data time parameters are divided into three types: seconds, minutes, and hours. Then, subtract the data aging from the current time to calculate the minimum valid time effecttime. Finally, loop through the data within the valid time from the minimum time to the current time to check if it exists. If it exists, it indicates that the data is normal; otherwise, it is abnormal. For the interval data type, first subtract the data delay from the current time to obtain the data valid time, and then calculate the data interval range where the data valid time is located. If it is less than the minimum interval, take the maximum interval of the previous day as the final valid time; otherwise, take the interval where the valid time is located as the final valid time.

[0103] The effects of the above technical solution are as follows: By distinguishing different types of meteorological data (periodic timing, interval timing, no fixed time) according to the currency constraint of data aging, it is possible to more accurately grasp the data update cycle and the valid time range. This helps to reduce misjudgments or omissions caused by inconsistent data currency and improve the accuracy of data processing.

[0104] For periodic timing meteorological data, by judging the data parameter format and obtaining the minimum time and query cycle within the data delay time, it is possible to start data loading at a more reasonable time point, reducing unnecessary waiting time. At the same time, by scanning the data within the valid time through n times of loop traversal, it is possible to ensure that all timed data within the cycle is accurately captured and verified. For interval timing meteorological data, by calculating the current time minus the data delay time to determine the data valid time and flexibly adjusting the valid time range according to the size of the time interval (such as extracting the maximum interval of the previous day or the current interval), it is possible to avoid data loss caused by too small time intervals and improve the efficiency of data loading and verification.

[0105] This technical solution can process various types of meteorological data, including periodic timing, interval timing, and data without a fixed time, with strong flexibility and adaptability. This enables the system to easily handle meteorological data from different data sources and with different update frequencies, meeting diverse business needs. By accurately judging the type and valid time of meteorological data, the system can provide more timely and accurate data support, providing a better user experience for users. At the same time, these data also provide strong support for meteorological prediction, disaster warning and other decision-making, helping to improve the scientificity and accuracy of decision-making. Through the automated data classification, loading, and verification process, the possibility of manual intervention and errors is reduced, and the operation and maintenance costs are lowered. At the same time, timely data monitoring and anomaly detection help to quickly discover and solve problems, reducing the risks caused by data errors or omissions.

[0106] An embodiment of the present invention is asFigure 3 and Figure 4 As shown in Figure 4 , the protocol types corresponding to each meteorological data include SMB protocol, SHH protocol, TCP protocol, FTP protocol, and HTTP protocol; among them, the TCP protocol includes SQL Server, PostgreSQL, MySql, and Oracle.

[0107] At the same time, the monitoring rules corresponding to each protocol type are as follows:

[0108] The Sql statement rules corresponding to the monitoring principles of the database data corresponding to the monitoring rules include: data server ID monitoring, Sql statement monitoring, data timeliness monitoring, data count monitoring, data table name monitoring, and time field monitoring;

[0109] The file rules corresponding to the monitoring principles of the file data corresponding to the monitoring rules include: data server ID monitoring, file path monitoring or file modification time monitoring, whether it is associated with the element cycle monitoring, data timeliness monitoring, and file size monitoring;

[0110] The rules corresponding to the HTTP data corresponding to the monitoring rules include DATA set rules, null value rules, and 404 error rules;

[0111] Among them, the DATA set rules include URL monitoring, parameter monitoring, parameter rule monitoring, whether it is associated with the data cycle monitoring, and data timeliness monitoring;

[0112] The null value rules include URL monitoring, parameter monitoring, parameter rule monitoring, whether it is associated with the data cycle monitoring, and data timeliness monitoring;

[0113] The 404 error rules include URL monitoring, parameter monitoring, parameter rule monitoring, whether it is associated with the data cycle monitoring, and data timeliness monitoring.

[0114] The working principle of the above technical solution is: The diversity data verification mainly completes the comprehensive monitoring of different data types. The types of monitored data mainly include data types such as HTTP protocol data, TCP protocol data, SMB protocol data, FTP protocol data, and SSH protocol data. Among them, the TCP protocol mainly realizes the access and monitoring of different database data, such as SQL Server, Oracel, Mysql, etc.; the SMB protocol realizes the monitoring of file data in the Windows operating system. The monitoring protocols supported by the diversity data verification are as Figure 3 shown.

[0115] Since the operation methods and data formats of each protocol are different during connection and monitoring, it is necessary to classify and customize the data reading rules for each protocol. For the data information of different protocols, different monitoring rules will be adopted. The overview of the monitoring rules is as Figure 4 shown.

[0116] The effects of the above technical solutions are as follows: The technical solutions support the monitoring of multiple meteorological data types (including data transmitted through protocols such as SMB, SSH, TCP, FTP, HTTP, etc.), ensuring that regardless of the protocol or system from which the data originates, it can be effectively included in the monitoring scope. This comprehensive coverage ability improves the compatibility and flexibility of the system.

[0117] For different types of data (database data, file data, HTTP data), detailed monitoring rules have been formulated. For example, database data monitoring includes multiple dimensions such as data server ID, Sql statements, data timeliness, and the number of data records; file data monitoring focuses on file paths, modification times, associated data cycles, etc.; HTTP data monitoring covers multiple aspects such as URLs, parameters, null values, 404 errors, etc. These refined monitoring rules help to discover and solve problems that may occur in various links such as data collection, transmission, and storage.

[0118] By implementing measures such as data timeliness monitoring and the number of data records monitoring, expired or abnormal data can be discovered and processed in a timely manner, ensuring the accuracy and timeliness of the data. This is particularly important for application scenarios with extremely high timeliness requirements such as meteorological data. Comprehensive data monitoring helps to discover and solve potential system failures or data problems in a timely manner, thereby avoiding adverse effects of these problems on system stability and business continuity. In addition, through an automated monitoring and alarm mechanism, the possibility of manual intervention and errors can be reduced, further improving the reliability of the system. Refined monitoring rules can help the system more accurately identify and process data problems, avoiding unnecessary resource waste. At the same time, the automated monitoring and alarm mechanism can reduce the burden on operation and maintenance personnel and lower operation and maintenance costs. Accurate and real-time data monitoring provides more reliable data support for users, helping to improve the user experience. At the same time, these data also provide strong support for decision-making such as meteorological forecasting and disaster warning, helping to improve the scientificity and accuracy of decision-making.

[0119] In summary, the above technical solutions achieve comprehensive monitoring of diverse data such as meteorological data through measures such as comprehensively covering multiple data types and protocols, formulating refined monitoring rules, and improving the accuracy and timeliness of data monitoring, thereby enhancing the reliability, stability, and user experience of the system.

[0120] In an embodiment of the present invention, different monitoring protocols are utilized to obtain diverse data verification for different types of meteorological data, and data monitoring of various types of meteorological data is completed, including:

[0121] S201. When there is periodic timed meteorological data within the effective time, determine the data status of the data node corresponding to the periodic timed meteorological data, and record the data status of the data node.

[0122] S202. When there is interval timed meteorological data within the effective time, determine the data status of the data node corresponding to the interval timed meteorological data, and record the data status of the data node.

[0123] S203. When the type of the meteorological data is non-fixed-time meteorological data, load the meteorological data and determine whether there is non-fixed-time meteorological data; when there is non-fixed-time meteorological data, determine the data status of the data node corresponding to the non-fixed-time meteorological data, and record the data status of the data node.

[0124] Specifically, determining the data status of the data nodes corresponding to the periodic timed meteorological data, interval timed meteorological data, and non-fixed-time meteorological data includes:

[0125] Starting from the first node of the data stream, recursively traverse layer by layer until the end point of the data stream through a recursive algorithm to obtain the data status corresponding to each data node.

[0126] Return the data status corresponding to the data node layer by layer, and finally return the monitoring status of the entire data process.

[0127] The working principle of the above technical solution is as follows: Starting from the first node N(1) of the data stream, recursively traverse layer by layer until the end point N(n) of the data stream, and return the data status of the data node layer by layer. Finally, return the monitoring status of the entire data process. To ensure that users can customize the monitoring process conveniently and quickly, jQuery drag-and-drop technology, SVG graphic drawing technology, etc. are introduced in the front-end page, which improves the visualization and visualization of the monitoring process configuration. Moreover, during the execution of the background monitoring process, the Quartz open-source framework is introduced to implement the concurrent execution of multiple data processes, and at the same time, recursive indexing is implemented for each data process to sequentially retrieve the data status of each data node. Data stream node monitoring process Figure 5 as shown.

[0128] The effects of the above technical solution are as follows: By monitoring and recording the data status for three different types of meteorological data, namely periodic timing, interval timing, and no fixed time, the comprehensiveness of data monitoring is ensured. Whether it is regularly updated data or randomly appearing data, it can be effectively included in the monitoring scope, thereby improving the accuracy and reliability of data monitoring. Through the recursive algorithm, starting from the first node of the data stream and recursively until the end point of the data stream, the data status of the data nodes is returned layer by layer, and finally the monitoring status of the entire data process is returned. This real-time monitoring and dynamic feedback mechanism enables the system to promptly detect and handle data anomalies or problems, improving the real-time and dynamic nature of data monitoring. The application of the recursive algorithm makes the data monitoring process more efficient and orderly. The system can automatically complete the traversal and status judgment of data nodes, reducing the possibility of manual intervention and errors, and improving the efficiency and performance of data monitoring.

[0129] Comprehensive data monitoring and timely exception handling help reduce system failures or instability caused by data errors or anomalies. This enhances the stability and reliability of the entire system, ensuring the continuity and availability of meteorological data services. Accurate and comprehensive data monitoring results provide strong support for decision-making such as meteorological forecasting and disaster warning. Decision-makers can adjust strategies and optimize resource allocation in a timely manner based on the monitoring results to respond to possible meteorological changes or disaster risks. The implementation of this technical solution helps to promote the standardization and standardization of meteorological data governance and quality management. By monitoring and recording the data status, data quality problems can be promptly discovered, and corresponding measures can be taken for improvement and optimization to improve the quality and availability of data.

[0130] In one embodiment of the present invention, the comprehensive meteorological data multi-dimensional streaming centralized monitoring method further includes:

[0131] Step 1: Real-time monitor the data node operation parameters corresponding to each data node; wherein, the data node operation parameters include memory utilization rate, network bandwidth occupancy rate, data processing error rate, and the number of acquisition processes per unit time.

[0132] Step 2: Obtain the first node operation evaluation coefficient by using the memory utilization rate and the network bandwidth occupancy rate; wherein, the first node operation evaluation coefficient is obtained through the following formula:

[0133]

[0134] wherein, C 01 represents the first node operation evaluation coefficient; n represents the number of unit times experienced by the data node operation; M p represents the average memory utilization rate corresponding to n unit times; M zRepresents the median memory utilization rate corresponding to n unit times; M min Represents the minimum memory utilization rate corresponding to n unit times; M i Represents the memory utilization rate corresponding to the i-th unit time; B p Represents the average network bandwidth occupancy rate corresponding to n unit times; B z Represents the median network bandwidth occupancy rate corresponding to n unit times; B min Represents the minimum network bandwidth occupancy rate corresponding to n unit times; B i Represents the network bandwidth occupancy rate corresponding to the i-th unit time; S 01 And S 02 Respectively represent the first adjustment coefficient and the second adjustment coefficient, and the first adjustment coefficient and the second adjustment coefficient are obtained through the following formula:

[0135]

[0136] Among them, S 01 And S 02 Respectively represent the first adjustment coefficient and the second adjustment coefficient; k 01 And k 02 Respectively represent the weight values corresponding to the preset memory utilization rate and network bandwidth occupancy rate; f b Represents the standard deviation of the network bandwidth occupancy rate corresponding to n unit times; f m Represents the standard deviation of the memory utilization rate corresponding to n unit times;

[0137] Step 3: Compare the first node operation evaluation coefficient with a preset first evaluation threshold;

[0138] Step 4: When the first node operation evaluation coefficient is lower than the preset first evaluation threshold, then retrieve the number of acquisition processes per unit time;

[0139] Step 5: Combine the number of acquisition processes per unit time with the first node operation evaluation coefficient to obtain a second node operation evaluation coefficient; among them, the second node operation evaluation coefficient is obtained through the following formula:

[0140]

[0141] Among them, C 02 Represents the second node operation evaluation coefficient; n represents the number of unit times experienced by the data node operation; P i Represents the number of acquisition processes per unit time corresponding to the i-th unit time; P c Represents the reference value of the number of acquisition processes per unit time preset; f b Represents the standard deviation of the network bandwidth occupancy rate corresponding to n unit times; fm represents the standard deviation of memory utilization corresponding to n unit time intervals; f p represents the standard deviation of the number of acquisition processes corresponding to n unit time intervals; C 01 represents the running evaluation coefficient of the first node; C y01 represents a preset first evaluation threshold; P p represents the average value of the number of acquisition processes corresponding to n unit time intervals; P z represents the median value of the number of acquisition processes corresponding to n unit time intervals;

[0142] Step 6, compare the running evaluation coefficient of the second node with a preset second evaluation threshold;

[0143] Step 7, when the running evaluation coefficient of the second node exceeds the preset second evaluation threshold, it is determined that there is an abnormality in the operation of the data node, and an alarm for the abnormal operation of the data node is given.

[0144] The working principle of the above technical solution is as follows: The system continuously monitors the key running parameters of each data node, including memory utilization, network bandwidth occupancy, data processing error rate, and the number of acquisition processes per unit time. Based on the average value, median value, and minimum value of memory utilization and network bandwidth occupancy, as well as the preset weight values and standard deviations, the running evaluation coefficient (C01) of the first node is calculated through a specific formula. This coefficient reflects the comprehensive performance of the data node in terms of memory and network resource utilization. Compare the running evaluation coefficient of the first node with the preset first evaluation threshold. If it is lower than the threshold, it indicates that there may be performance bottlenecks or deficiencies in the data node in some aspects.

[0145] When the running evaluation coefficient of the first node is lower than the threshold, the system further considers the number of acquisition processes per unit time as an evaluation factor. Combining the running evaluation coefficient of the first node and the number of acquisition processes per unit time, the running evaluation coefficient of the second node is calculated through another formula. This coefficient more comprehensively reflects the running state of the data node, including resource utilization and process handling capabilities. Compare the running evaluation coefficient of the second node with the preset second evaluation threshold. If it exceeds the threshold, it indicates that the running state of the data node is abnormal and alarm processing is required. When an abnormality in the operation of the data node is detected, the system immediately gives an alarm so that the operation and maintenance personnel can intervene in a timely manner and solve the problem.

[0146] The effects of the above technical solutions are as follows: By monitoring the operating parameters in multiple dimensions and combining complex calculation models, the operating status of data nodes can be evaluated more accurately, improving the comprehensiveness and accuracy of monitoring. Through real-time monitoring and the abnormal alarm mechanism, abnormal situations during the operation of data nodes can be detected in a timely manner, and corresponding handling measures can be taken to prevent problems from escalating. By evaluating key indicators such as memory utilization rate, network bandwidth occupancy rate, and the number of processes obtained, it can help operation and maintenance personnel understand the resource utilization of data nodes, thereby making more reasonable resource allocation and optimization. By promptly discovering and handling abnormal problems during the operation of data nodes, the stability and reliability of the entire system can be effectively improved, ensuring the continuity and efficiency of meteorological data services. The automated monitoring and alarm mechanism can reduce the frequency and error rate of manual intervention, thereby reducing operation and maintenance costs and improving work efficiency.

[0147] An embodiment of the present invention proposes a comprehensive meteorological data multi-dimensional streaming centralized monitoring system, as Figure 5 shown, the comprehensive meteorological data multi-dimensional streaming centralized monitoring system includes:

[0148] An effective time acquisition module, configured to determine the type of meteorological data and obtain the effective time according to the meteorological data type;

[0149] A data monitoring module, configured to perform diversity data verification on different types of meteorological data obtained by using different monitoring protocols, and complete the data monitoring of various types of meteorological data.

[0150] The working principle of the above technical solutions is as follows: For comprehensive meteorological data of different types, different transmission methods, and different formats (including meteorological data, cross-industry data, etc.), streaming monitoring is realized from each node such as collection, processing, and application, and through a configurable method, a data link diagram is formed, enabling staff to quickly locate the data node with anomalies and promptly discover problem links.

[0151] The meteorological data transmission methods of the above technical solutions mainly include HTTP protocol data, TCP protocol data, SMB protocol data, FTP protocol data, and SSH protocol. Different data types require different request protocols, and for the data formats corresponding to different data types, different reading and parsing methods are used to analyze the data attributes, and finally the verification of diversity data is completed.

[0152] Meteorological data has diverse structures and significant differences. To achieve normalized monitoring, this project will determine the timeliness of data. According to different data timeliness constraints (Currency Constraint), meteorological data is divided into three types: periodic timing (such as automatic station live data every 5 minutes), interval timing (such as short-term weather from 4:50 to 21:50 every day), and no fixed time (such as warning information). During the calculation of data timeliness, the timeliness calculation function is used to classify the data and match the corresponding relationship between the current monitoring time, data timeliness, and data cycle. For time points exceeding the maximum monitoring timeliness, cyclic verification is performed through the set verification time intervals and valid time intervals, and finally, the data timeliness monitoring algorithm within the valid time period is realized.

[0153] In the way of recursive algorithm, starting from the first node of the data stream, recursive operations are carried out in sequence until the end point of the data stream, and the data status of the data nodes is returned layer by layer, and finally the monitoring status of the entire data process is returned.

[0154] The effects of the above technical solutions are as follows: By subdividing meteorological data types (periodic timing, interval timing, no fixed time) and combining data timeliness constraints (Currency Constraint), the validity and timeliness of data can be judged more accurately. This helps to reduce decision-making errors caused by outdated or incorrect data and improve the accuracy and reliability of meteorological services.

[0155] Using the diversity data verification protocol, comprehensive and detailed verification of different types of meteorological data is carried out to ensure the integrity and accuracy of the data, and further improve the quality of data monitoring.

[0156] Supporting multiple data transmission protocols (HTTP, TCP, SMB, FTP, SSH, etc.) enables the system to flexibly access meteorological data from different sources and in different formats, enhancing the compatibility and scalability of the system. By forming a data link diagram in a configurable manner, staff can intuitively understand the path and status of data flow, facilitating quick positioning of problem nodes and improving the flexibility and operability of data monitoring.

[0157] Adopting the streaming monitoring technology, real-time collection, processing, and monitoring of meteorological data are realized, reducing data processing delay and waiting time, and improving the efficiency and performance of data monitoring. By recursively returning the status of data nodes layer by layer, the monitoring results of the entire data process can be quickly summarized, providing timely and comprehensive data support for decision-makers.

[0158] By introducing artificial intelligence technologies such as machine learning and deep learning, intelligent recognition and analysis of meteorological data are carried out, improving the accuracy and efficiency of data type recognition and providing a more intelligent means for data monitoring. By constructing a cross-correlated data relationship link graph, the internal connections and laws between data can be revealed, providing more comprehensive and in-depth information support for meteorological forecasting and decision-making. An efficient data monitoring and verification mechanism can timely detect and handle data anomalies, reducing system failures and operation and maintenance costs caused by data errors. Flexible monitoring configuration and visual data link graphs reduce the operation difficulty and error rate of operation and maintenance personnel, improving the stability and reliability of the system.

[0159] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A multi-dimensional streaming centralized monitoring method for comprehensive meteorological data, characterized in that, The comprehensive meteorological data multi-dimensional streaming centralized monitoring method includes: Judge the type of meteorological data, and obtain the effective time according to the type of the meteorological data; Use different monitoring protocols to perform diversity data verification on different types of meteorological data obtained, and complete the data monitoring of various types of meteorological data; The comprehensive meteorological data multi-dimensional streaming centralized monitoring method further includes: Real-time monitor the data node operation parameters corresponding to each data node; wherein, the data node operation parameters include memory utilization rate, network bandwidth occupancy rate, data processing error rate, and the number of acquisition processes per unit time; Obtain the first node operation evaluation coefficient by using the memory utilization rate and the network bandwidth occupancy rate; wherein, the first node operation evaluation coefficient is obtained through the following formula: Among them, C 01 represents the running evaluation coefficient of the first node; n represents the number of unit time periods experienced by the data node during operation; M p represents the average memory utilization rate corresponding to n unit time periods; M z represents the median memory utilization rate corresponding to n unit time periods; M min represents the minimum memory utilization rate corresponding to n unit time periods; M i represents the memory utilization rate corresponding to the i-th unit time period; B p represents the average network bandwidth occupancy rate corresponding to n unit time periods; B z represents the median network bandwidth occupancy rate corresponding to n unit time periods; B min represents the minimum network bandwidth occupancy rate corresponding to n unit time periods; B i represents the network bandwidth occupancy rate corresponding to the i-th unit time period; S 01 and S 02 represent the first adjustment coefficient and the second adjustment coefficient respectively, and the first adjustment coefficient and the second adjustment coefficient are obtained through the following formula: where k 01 and k 02 respectively represent the weight values corresponding to the preset memory utilization rate and network bandwidth occupancy rate; f b represents the standard deviation of the network bandwidth occupancy rate corresponding to n unit times; f m represents the standard deviation of the memory utilization rate corresponding to n unit times; Compare the first node operation evaluation coefficient with a preset first evaluation threshold; When the first node operation evaluation coefficient is lower than the preset first evaluation threshold, then retrieve the number of acquisition processes per unit time; Obtain the second node operation evaluation coefficient by combining the number of acquisition processes per unit time with the first node operation evaluation coefficient; wherein, the second node operation evaluation coefficient is obtained through the following formula: Among them, C 02 represents the running evaluation coefficient of the second node; P i represents the number of acquisition processes within the unit time corresponding to the i-th unit time; P c represents the reference value of the number of acquisition processes within the preset unit time; f p represents the standard deviation of the number of acquisition processes corresponding to n unit times; C y01 represents the preset first evaluation threshold; P p represents the average value of the number of acquisition processes corresponding to n unit times; P z represents the median value of the number of acquisition processes corresponding to n unit times; Compare the second node operation evaluation coefficient with a preset second evaluation threshold; When the second node operation evaluation coefficient exceeds the preset second evaluation threshold, it is determined that the data node operation is abnormal, and an alarm for abnormal data node operation is given.

2. The multi-dimensional streaming centralized monitoring method for comprehensive meteorological data according to claim 1, characterized in that, Judging the type of meteorological data and obtaining the effective time according to the type of the meteorological data includes: Judge the type of meteorological data according to different data timeliness constraints, wherein the type of the meteorological data includes periodic timed meteorological data, interval timed meteorological data, and non-fixed time meteorological data; When the type of the meteorological data is periodic timed meteorological data, determine the effective time corresponding to the periodic timed meteorological data; When the type of the meteorological data is interval timed meteorological data, determine the effective time corresponding to the interval timed meteorological data.

3. The multi-dimensional streaming centralized monitoring method for comprehensive meteorological data according to claim 2, characterized in that When the type of the meteorological data is periodic timed meteorological data, determining the effective time corresponding to the periodic timed meteorological data includes: When the type of the meteorological data is periodic timed meteorological data, then judge the data parameter format; Obtain the minimum time and the query period within the data delay time according to the data parameter format; Taking the minimum time as the time starting point and using the data parameter format as the unit, load the meteorological data corresponding to the effective time; After the meteorological data is loaded, perform n times of cyclic traversal scanning to judge whether there is periodic timed meteorological data within the effective time.

4. The multi-dimensional streaming centralized monitoring method for comprehensive meteorological data according to claim 2, characterized in that, When the type of the meteorological data is interval timed meteorological data, determining the effective time corresponding to the interval timed meteorological data includes: When the type of the meteorological data is interval timed meteorological data, then obtain the data effective time after subtracting the data delay time from the current time; Judge whether the time interval where the data effective time is located is less than the minimum interval; When the time interval where the data effective time is located is less than the minimum interval, then extract the maximum interval of the previous day as the effective time; When the time interval in which the data valid time is located is greater than the minimum interval, the time interval in which the data valid time is located is extracted as the valid time; Load the meteorological data corresponding to the valid time, and determine whether there is interval-timed meteorological data within the valid time.

5. The multi-dimensional streaming centralized monitoring method for comprehensive meteorological data according to claim 1, wherein The protocol types corresponding to each type of meteorological data include SMB protocol, SHH protocol, TCP protocol, FTP protocol, and HTTP protocol; among them, the TCP protocol includes SQL Server, PostgreSQL, MySql, and Oracle.

6. The multi-dimensional streaming centralized monitoring method for comprehensive meteorological data according to claim 5, characterized in that The monitoring rules corresponding to each protocol type are as follows: The Sql statement rules corresponding to the monitoring principles of the database data corresponding to the monitoring rules include: data server ID monitoring, Sql statement monitoring, data timeliness monitoring, data count monitoring, data table name monitoring, and time field monitoring; The file rules corresponding to the monitoring principles of the file data corresponding to the monitoring rules include: data server ID monitoring, file path monitoring or file modification time monitoring, whether it is associated with the element period monitoring, data timeliness monitoring, and file size monitoring; The rules corresponding to the HTTP data corresponding to the monitoring rules include the DATA set rule, the null value rule, and the 404 error rule; Among them, the DATA set rule includes URL monitoring, parameter monitoring, parameter rule monitoring, whether it is associated with the data period monitoring, and data timeliness monitoring; The null value rule includes URL monitoring, parameter monitoring, parameter rule monitoring, whether it is associated with the data period monitoring, and data timeliness monitoring; The 404 error rule includes URL monitoring, parameter monitoring, parameter rule monitoring, whether it is associated with the data period monitoring, and data timeliness monitoring.

7. The multi-dimensional streaming centralized monitoring method for comprehensive meteorological data according to claim 1, wherein Use different monitoring protocols to perform diversity data verification on different types of meteorological data, and complete the data monitoring of various types of meteorological data, including: When there is periodic timed meteorological data within the valid time, judge the data status of the data node corresponding to the periodic timed meteorological data, and record the data status of the data node; When there is interval-timed meteorological data within the valid time, judge the data status of the data node corresponding to the interval-timed meteorological data, and record the data status of the data node; When the type of the meteorological data is non-fixed-time meteorological data, load the meteorological data and judge whether there is non-fixed-time meteorological data; when there is non-fixed-time meteorological data, judge the data status of the data node corresponding to the non-fixed-time meteorological data, and record the data status of the data node.

8. The multi-dimensional streaming centralized monitoring method for comprehensive meteorological data according to claim 7, characterized in that Judging the data status of the data nodes corresponding to the periodic timed meteorological data, interval-timed meteorological data, and non-fixed-time meteorological data includes: Starting from the first node of the data stream in the way of a recursive algorithm, recursively until the end point of the data stream, and obtain the data status corresponding to each layer of data nodes; Return the data status corresponding to the data nodes layer by layer, and finally return the monitoring status of the entire data process.

9. A multi-dimensional streaming centralized monitoring system for comprehensive meteorological data adopts the multi-dimensional streaming centralized monitoring method for comprehensive meteorological data described in any one of claims 1 to 8, characterized in that, The comprehensive meteorological data multi-dimensional streaming centralized monitoring system includes: An effective time acquisition module, which is used to judge the type of meteorological data and acquire the effective time according to the meteorological data type; a data monitoring module, which is used to perform diversity data verification on different types of acquired meteorological data by using different monitoring protocols to complete the data monitoring of various types of meteorological data.

Images