Comprehensive meteorological data multi-dimensional streaming centralized monitoring method and system

Abstract

The invention provides a comprehensive meteorological data multi-dimensional streaming centralized monitoring method and a comprehensive meteorological data multi-dimensional streaming centralized monitoring system. The comprehensive meteorological data multi-dimensional streaming centralized monitoring method comprises the steps that the types of meteorological data are judged according to different data timeliness constraints, and the types of the meteorological data comprise periodic timing meteorological data, interval timing meteorological data and meteorological data without fixed time; obtaining effective time according to the meteorological data type; and different monitoring protocols are utilized to obtain different types of meteorological data to carry out diversity data verification, and data monitoring of various types of meteorological data is completed. The system comprises modules corresponding to the steps of the method.

Description

Technical Field

[0001] The present invention proposes a method and system for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming manner, belonging to the technical field of comprehensive meteorological data monitoring. Background Art

[0002] In the field of meteorological observation and data analysis, with the rapid development of sensor technology and network communication technology, the ability to collect, transmit and process meteorological data has been significantly improved. However, meteorological data is of various types, from a wide range of sources, and has strong timeliness. How to efficiently and accurately monitor these multi-dimensional, streaming meteorological data has become an urgent problem to be solved. Meteorological data can be divided into many types according to its collection method and update frequency, mainly including periodic timed meteorological data, interval timed meteorological data, and non-fixed time meteorological data. These data types each have different characteristics: Periodic meteorological data: such as daily temperature, humidity, precipitation, etc. These data are collected and updated at fixed time intervals (such as every hour, every day) and have strong regularity and predictability.

[0003] Interval timed meteorological data: such as wind direction and wind speed within a specific time period. These data are collected continuously within a specific time period, but not strictly at fixed time intervals. The effective time range depends on actual needs.

[0004] Weather data without fixed time: such as data on sudden meteorological events (such as tornadoes and hail), the time of occurrence of these data is unpredictable, but once they occur, they need to be collected and processed immediately.

[0005] At present, the monitoring methods for meteorological data mainly rely on traditional data verification and storage technologies. These methods have the following limitations when processing multi-dimensional and streaming meteorological data: Inefficient data verification: Traditional data verification methods usually require checking all received data one by one. This method is inefficient when processing large-scale, high-frequency meteorological data and is difficult to meet real-time requirements.

[0006] Inaccurate identification of data types: Due to the diversity of meteorological data types, traditional monitoring methods often have difficulty accurately identifying different types of data, resulting in reduced accuracy and reliability of data processing.

[0007] Inaccurate determination of effective time: For periodic and interval timing meteorological data, the determination of effective time directly affects the timeliness and availability of data. Traditional monitoring methods may not be able to accurately determine the effective time range of these data, resulting in data waste or delayed processing. Summary of the invention

[0008] The present invention provides a method and system for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming manner, which is used to solve the technical problems existing in the above-mentioned prior art. The technical solutions adopted are as follows: A method for centralized monitoring of integrated meteorological data in a multi-dimensional streaming manner, the method comprising: Determining the type of meteorological data according to different data time constraints, wherein the types of meteorological data include periodic timing meteorological data, interval timing meteorological data and non-fixed time meteorological data; Acquire the effective time according to the meteorological data type; Different monitoring protocols are used to perform diverse data verification on different types of meteorological data, and complete data monitoring of various types of meteorological data.

[0009] Further, obtaining the effective time according to the meteorological data type includes: When the type of the meteorological data is periodic timing meteorological data, determining the effective time corresponding to the periodic timing meteorological data; When the type of the meteorological data is interval timing meteorological data, the effective time corresponding to the interval timing meteorological data is determined.

[0010] Further, when the type of the meteorological data is periodic timing meteorological data, determining the effective time corresponding to the periodic timing meteorological data includes: When the type of the meteorological data is periodic timing meteorological data, the data parameter format is determined; Obtaining the minimum time and query cycle that meet the data delay time according to the data parameter format; Take the minimum time as the time starting point and the data parameter format as the unit to load the meteorological data corresponding to the effective time; After the meteorological data is loaded, n cycles of scanning are performed to determine whether periodic timing meteorological data exists within the effective time.

[0011] Further, when the type of the meteorological data is interval timing meteorological data, determining the effective time corresponding to the interval timing meteorological data includes: When the type of the meteorological data is interval timing meteorological data, the data validity time after subtracting the data delay time from the current time is obtained; Determine whether the time interval of the data validity time is less than the minimum interval; When the time interval of the data validity time is less than the minimum interval, the maximum interval of the previous day is extracted as the valid time; When the time interval of the data valid time is greater than the minimum interval, the time interval of the data valid time is extracted as the valid time; The meteorological data corresponding to the effective time is loaded to determine whether there is interval timing meteorological data within the effective time.

[0012] Furthermore, the protocol types corresponding to each type of meteorological data include SMB protocol, SHH protocol, TCP protocol, FTP protocol and HTTP protocol; wherein the TCP protocol includes SQL Server, PostgreSQL, MySql and Oracle.

[0013] Furthermore, the monitoring rules corresponding to each protocol type are as follows: The Sql statement rules corresponding to the monitoring principles of the database data corresponding to the monitoring rules include: data server ID monitoring, Sql statement monitoring, data timeliness monitoring, data number monitoring, data table name monitoring and time field monitoring; The file rules corresponding to the monitoring principles of the file data corresponding to the monitoring rules include: data server ID monitoring, file path monitoring or file modification time monitoring, whether the associated element has a cycle monitoring, data timeliness monitoring and file size monitoring; The rules for HTTP data corresponding to the monitoring rules include DATA set rules, null value rules and 404 error rules; The DATA set rules include URL monitoring, parameter monitoring, parameter rule monitoring, whether the data cycle is associated, and data timeliness monitoring; The null value rules include URL monitoring, parameter monitoring, parameter rule monitoring, whether the data cycle is associated and data timeliness monitoring; The 404 error rules include URL monitoring, parameter monitoring, parameter rule monitoring, associated data cycle monitoring and data timeliness monitoring.

[0014] Furthermore, different monitoring protocols are used to perform diversified data verification on different types of meteorological data, and data monitoring of various types of meteorological data is completed, including: When periodic timing meteorological data exists within the effective time, the data state of the data node corresponding to the periodic timing meteorological data is determined, and the data state of the data node is recorded; When there is interval timing meteorological data within the effective time, the data state of the data node corresponding to the interval timing meteorological data is determined, and the data state of the data node is recorded; When the type of the meteorological data is meteorological data without fixed time, the meteorological data is loaded to determine whether there is meteorological data without fixed time; when there is meteorological data without fixed time, the data status of the data node corresponding to the meteorological data without fixed time is determined, and the data status of the data node is recorded.

[0015] Further, judging the data status of the data nodes corresponding to the periodic timing meteorological data, the interval timing meteorological data and the meteorological data without fixed time includes: Using a recursive algorithm, the data state corresponding to each layer of data nodes is obtained by recursively starting from the first node of the data stream and recursively until the end point of the data stream. The data status corresponding to the data node is returned layer by layer, and finally the monitoring status of the entire data flow is returned.

[0016] Furthermore, the method for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming manner further includes: Real-time monitoring of the data node operating parameters corresponding to each data node; wherein the data node operating parameters include memory utilization, network bandwidth occupancy, data processing error rate and the number of acquisition processes per unit time; The memory utilization rate and the network bandwidth occupancy rate are used to obtain the first node operation evaluation coefficient; wherein the first node operation evaluation coefficient is obtained by the following formula:

[0017] in, C 01 Indicates the first node operation evaluation coefficient; n Indicates the number of unit time experienced by the data node; M p express n The average memory utilization corresponding to the unit time; M z express n The median memory utilization corresponding to the unit time; M min express n The minimum memory utilization corresponding to the unit time; M i Indicates i Memory utilization per unit time; B p express n The average value of network bandwidth usage corresponding to each unit time; B z express n The median value of network bandwidth usage corresponding to a unit of time; B min express n The minimum network bandwidth usage corresponding to a unit of time; B i Indicates i The network bandwidth usage corresponding to the unit time; S01 and S 02 represent the first adjustment coefficient and the second adjustment coefficient respectively, and the first adjustment coefficient and the second adjustment coefficient are obtained by the following formula:

[0018] in, S 01 and S 02 represent the first adjustment coefficient and the second adjustment coefficient respectively; k 01 and k 02 Respectively represent the weight values ​​corresponding to the preset memory utilization and network bandwidth occupancy; f b express n The standard deviation of network bandwidth utilization corresponding to each unit of time; f m express n The standard deviation of memory utilization corresponding to unit time; Comparing the first node operation evaluation coefficient with a preset first evaluation threshold; When the first node operation evaluation coefficient is lower than a preset first evaluation threshold, the number of acquisition processes per unit time is retrieved; The second node operation evaluation coefficient is obtained by combining the number of acquired processes in the unit time with the first node operation evaluation coefficient; wherein the second node operation evaluation coefficient is obtained by the following formula:

[0019] in, C 02 represents the second node operation evaluation coefficient; n Indicates the number of unit time experienced by the data node; P i Indicates i The number of acquisition processes per unit time corresponding to the unit time; P c Indicates the reference value of the number of processes obtained within a preset unit time; f b express n The standard deviation of network bandwidth utilization corresponding to each unit of time; f m express n The standard deviation of memory utilization corresponding to unit time; f p express n The standard deviation of the number of processes obtained per unit time;C 01 Indicates the first node operation evaluation coefficient; C y01 represents a preset first evaluation threshold; P p express n The average number of processes obtained per unit time; P z express n Get the median value of the number of processes corresponding to the unit time; Comparing the second node operation evaluation coefficient with a preset second evaluation threshold; When the second node operation evaluation coefficient exceeds a preset second evaluation threshold, it is determined that the data node operation is abnormal, and an abnormal data node operation alarm is issued.

[0020] A multi-dimensional streaming centralized monitoring system for comprehensive meteorological data, the multi-dimensional streaming centralized monitoring system for comprehensive meteorological data comprising: A type determination module, used to determine the type of meteorological data according to different data time constraints, wherein the types of meteorological data include periodic timing meteorological data, interval timing meteorological data and non-fixed time meteorological data; An effective time acquisition module, used for acquiring effective time according to the meteorological data type; The data monitoring module is used to utilize different monitoring protocols to perform diverse data verification on different types of meteorological data, thereby completing data monitoring of various types of meteorological data.

[0021] Beneficial effects of the present invention: The present invention proposes a method and system for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming format. Comprehensive meteorological products are of various types, with different structures and diversified data transmission methods, which leads to the complexity of meteorological products in daily use, from collection to multi-platform application. This project is based on a multi-dimensional algorithm for data timeliness judgment and data diversity verification, and studies the streaming monitoring of diversified comprehensive meteorological data, realizing the chain management of comprehensive products from collection, processing to application, and forming a product application link network with clear context. Data anomalies can be fed back in time through the nodes of the link network, which is convenient for staff to quickly locate abnormal information points and improve the efficiency of data protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the method of the present invention; Figure 2 A flowchart corresponding to the effective time acquisition of the present invention; Figure 3 This is a schematic diagram of the protocol of the present invention; Figure 4 This is a schematic diagram of the monitoring rules of the present invention; Figure 5 A flow chart of the data monitoring process of the present invention; Figure 6 The system block diagram of the system of the present invention. DETAILED DESCRIPTION

[0023] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0024] The embodiment of the present invention proposes a method for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming manner, such as Figure 1 As shown, the method for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming manner includes: S1. Determine the type of meteorological data according to different data time constraints, wherein the types of meteorological data include periodic timing meteorological data, interval timing meteorological data and non-fixed time meteorological data; S2. Obtaining effective time according to the meteorological data type; S3. Use different monitoring protocols to perform diverse data verification on different types of meteorological data to complete data monitoring of multiple types of meteorological data.

[0025] The working principle of the above technical solution is: for comprehensive meteorological data of different types, different transmission methods and different formats (including meteorological data, cross-industry data, etc.), streaming monitoring of various nodes such as collection, processing and application is realized, and a data link diagram is formed in a configurable way, so that the staff can locate the nodes with abnormal data in time and find the problem links in time. In the data timeliness determination algorithm, according to the different data timeliness constraints (Currency Constraint), meteorological data is divided into three types: periodic timing (such as automatic station real-time data every 5 minutes), interval timing (such as short-term weather from 4:50 to 21:50 every day), and no fixed time (such as early warning information). In the data timeliness calculation process, the timeliness calculation function cyc() is used to classify the data, and the corresponding relationship between the current monitoring time and the data timeliness and data cycle is matched. When the time point exceeds the maximum monitoring timeliness, the set verification time and the valid time are cyclically verified, and finally the data timeliness monitoring algorithm within the valid time period is realized.

[0026] The meteorological data transmission methods of the above technical solution mainly include HTTP protocol data, TCP protocol data, SMB protocol data, FTP protocol data, and SSH protocol. Different data types require different request protocols, and different reading and parsing methods are used for data attribute analysis according to the data formats corresponding to different data types, and finally the verification of diverse data is completed.

[0027] Meteorological data has a diverse structure and large differences. In order to achieve normalized monitoring, this project will determine the timeliness of the data. According to different data timeliness constraints (Currency Constraint), meteorological data will be divided into three types: periodic timing (such as real-time data of automatic stations every 5 minutes), interval timing (such as short-term weather from 4:50 to 21:50 every day), and no fixed time (such as early warning information). In the process of calculating data timeliness, the timeliness calculation function is used to classify the data, and the corresponding relationship between the current monitoring time and the data timeliness and data cycle is matched. When the time point exceeds the maximum monitoring timeliness, the set verification time and the valid time are cyclically verified, and finally the data timeliness monitoring algorithm within the valid time period is realized.

[0028] Through the recursive algorithm, starting from the first node of the data flow and recursively until the end point of the data flow, the data status of the data node is returned layer by layer, and finally the monitoring status of the entire data flow is returned.

[0029] The above technical solutions proposed in this implementation focus on solving technical difficulties and key issues.

[0030] 1. In the timeliness judgment algorithm, how to judge whether the data is effectively updated based on the differences in meteorological products.

[0031] 2. Research on data flow monitoring algorithms, how to form a cross-correlated data relationship link diagram through technical means.

[0032] The effect of the above technical solution is: by subdividing the meteorological data types (periodic timing, interval timing, no fixed time), and combining the data timeliness constraint (Currency Constraint), the validity and timeliness of the data can be judged more accurately. This helps to reduce decision-making errors caused by outdated or erroneous data and improve the accuracy and reliability of meteorological services.

[0033] By utilizing diverse data verification protocols, we conduct comprehensive and detailed verification of different types of meteorological data to ensure the integrity and accuracy of the data and further improve the quality of data monitoring.

[0034] Supports multiple data transmission protocols (HTTP, TCP, SMB, FTP, SSH, etc.), allowing the system to flexibly access meteorological data from different sources and in different formats, enhancing the system's compatibility and scalability. A data link diagram is formed in a configurable manner, allowing staff to intuitively understand the path and status of data flow, facilitating rapid location of problem nodes, and improving the flexibility and operability of data monitoring.

[0035] The use of streaming monitoring technology enables real-time collection, processing and monitoring of meteorological data, reduces data processing delays and waiting time, and improves the efficiency and performance of data monitoring. By returning the status of data nodes layer by layer through a recursive algorithm, the monitoring results of the entire data process can be quickly summarized, providing timely and comprehensive data support for decision makers.

[0036] The introduction of artificial intelligence technologies such as machine learning and deep learning can intelligently identify and analyze meteorological data, improve the accuracy and efficiency of data type identification, and provide a more intelligent means for data monitoring. By constructing a cross-correlated data relationship link diagram, the inherent connections and laws between data can be revealed, providing more comprehensive and in-depth information support for meteorological forecasting and decision-making. Efficient data monitoring and verification mechanisms can promptly detect and handle data anomalies, reducing system failures and operation and maintenance costs caused by data errors. Flexible monitoring configuration and visual data link diagrams reduce the difficulty and error rate of operation and maintenance personnel, and improve the stability and reliability of the system.

[0037] One embodiment of the present invention, as Figure 2 As shown, obtaining the effective time according to the meteorological data type includes: S201, when the type of the meteorological data is periodic timing meteorological data, determining the effective time corresponding to the periodic timing meteorological data; S202: When the type of the meteorological data is interval-timed meteorological data, determine the effective time corresponding to the interval-timed meteorological data.

[0038] Specifically, when the type of the meteorological data is periodic timing meteorological data, determining the effective time corresponding to the periodic timing meteorological data includes: S2011. When the type of the meteorological data is periodic timing meteorological data, determining the data parameter format; S2012, obtaining a minimum time and a query period that meet the data delay time according to the data parameter format; S2013, taking the minimum time as the time starting point and the data parameter format as the unit, loading the meteorological data corresponding to the effective time; S2014: After the meteorological data is loaded, perform n cycles of scanning to determine whether there is periodic timing meteorological data within the effective time.

[0039] Meanwhile, when the type of the meteorological data is interval timing meteorological data, determining the effective time corresponding to the interval timing meteorological data includes: S2021. When the type of the meteorological data is interval timing meteorological data, obtaining the data validity time after subtracting the data delay time from the current time; S2022, determining whether the time interval of the data validity time is less than the minimum interval; S2023. When the time interval of the data validity time is smaller than the minimum interval, extract the maximum interval of the previous day as the validity time; S2024. When the time interval of the data valid time is greater than the minimum interval, extracting the time interval of the data valid time as the valid time; S2025. Load the meteorological data corresponding to the effective time, and determine whether there is interval timing meteorological data within the effective time.

[0040] The working principle of the above technical solution is as follows: Figure 2 As shown, meteorological data has the characteristics of complex sources and diverse data structures, and effective data management is becoming increasingly difficult. In order to study and realize streaming monitoring of comprehensive meteorological data, this project needs to solve the problems of data timeliness determination algorithm, diversity data verification algorithm and data flow monitoring algorithm.

[0041] It is necessary to determine the data type. Real-time data is directly loaded into the data connection driver to obtain real-time meteorological data. The periodic timing type first calculates the time format of the data parameters. Generally, the data time parameters are divided into three types: seconds, minutes, and hours. Then, the data timeliness is subtracted from the current time to calculate the minimum effective time effecttime. Finally, it loops from the minimum time to the current time to see if the data of the effective time exists. If it exists, it indicates that the data is normal, otherwise it is abnormal. The interval data type first obtains the data effective time by subtracting the data delay from the current time, and then calculates the data interval range where the data effective time is located. If it is less than the minimum interval, the maximum interval of the previous day is taken as the final effective time, otherwise the interval where the effective time is located is taken as the final effective time.

[0042] The effect of the above technical solution is: by distinguishing different types of meteorological data (periodic timing, interval timing, no fixed time) according to the data timeliness constraint (Currency Constraint), it is possible to more accurately grasp the data update cycle and effective time range. This helps to reduce misjudgments or omissions caused by inconsistent data timeliness and improve the accuracy of data processing.

[0043] For periodic timing meteorological data, by judging the data parameter format and obtaining the minimum time and query period that meet the data delay time, data loading can be started at a more reasonable time point to reduce unnecessary waiting time. At the same time, by looping through the data within the effective time for n times, it can ensure that all timing data within the period are accurately captured and verified. For interval timing meteorological data, the data effective time is determined by calculating the current time minus the data delay time, and the effective time range is flexibly adjusted according to the size of the time interval (such as extracting the maximum interval of the previous day or the current interval), avoiding data loss caused by too small a time interval and improving the efficiency of data loading and verification.

[0044] This technical solution can handle various types of meteorological data, including periodic timing, interval timing and data without fixed time, with strong flexibility and adaptability. This enables the system to easily cope with meteorological data from different data sources and different update frequencies, and meet diverse business needs. By accurately judging the type and effective time of meteorological data, the system can provide more timely and accurate data support, and provide users with a better user experience. At the same time, these data also provide strong support for decision-making such as meteorological forecasts and disaster warnings, which helps to improve the scientificity and accuracy of decision-making. Through automated data classification, loading and verification processes, the possibility of manual intervention and errors is reduced, and operation and maintenance costs are reduced. At the same time, timely data monitoring and anomaly detection help to quickly discover and solve problems, reducing the risks caused by data errors or omissions.

[0045] One embodiment of the present invention, as Figure 3 and Figure 4 As shown, the protocol type corresponding to each meteorological data includes SMB protocol, SHH protocol, TCP protocol, FTP protocol and HTTP protocol; wherein the TCP protocol includes SQL Server, PostgreSQL, MySql and Oracle.

[0046] At the same time, the monitoring rules corresponding to each protocol type are as follows: The Sql statement rules corresponding to the monitoring principles of the database data corresponding to the monitoring rules include: data server ID monitoring, Sql statement monitoring, data timeliness monitoring, data number monitoring, data table name monitoring and time field monitoring; The file rules corresponding to the monitoring principles of the file data corresponding to the monitoring rules include: data server ID monitoring, file path monitoring or file modification time monitoring, whether the associated element has a cycle monitoring, data timeliness monitoring and file size monitoring; The rules for HTTP data corresponding to the monitoring rules include DATA set rules, null value rules and 404 error rules; The DATA set rules include URL monitoring, parameter monitoring, parameter rule monitoring, whether the data cycle is associated, and data timeliness monitoring; The null value rules include URL monitoring, parameter monitoring, parameter rule monitoring, whether the data cycle is associated and data timeliness monitoring; The 404 error rules include URL monitoring, parameter monitoring, parameter rule monitoring, associated data cycle monitoring and data timeliness monitoring.

[0047] The working principle of the above technical solution is as follows: Diversity data verification mainly completes all-round monitoring of different data types. The types of monitored data mainly include HTTP protocol data, TCP protocol data, SMB protocol data, FTP protocol data, SSH protocol data and other data types. Among them, the TCP protocol mainly realizes the access and monitoring of different database data, such as SQL Server, Oracle, Mysql, etc.; the SMB protocol realizes the file data monitoring of the Windows operating system. The monitoring protocols supported by diversity data verification are as follows: Figure 3 shown.

[0048] Since each protocol has different operation modes and data formats when connected and monitored, it is necessary to classify and customize the data reading rules for each protocol. Different monitoring rules will be used for data information of different protocols. The monitoring rules are summarized as follows: Figure 4 shown.

[0049] The effect of the above technical solution is: the technical solution supports the monitoring of various types of meteorological data (including data transmitted through protocols such as SMB, SSH, TCP, FTP, HTTP, etc.), ensuring that no matter what protocol or system the data comes from, it can be effectively included in the monitoring scope. This comprehensive coverage capability improves the compatibility and flexibility of the system.

[0050] Detailed monitoring rules have been developed for different types of data (database data, file data, HTTP data). For example, database data monitoring includes multiple dimensions such as data server ID, SQL statements, data timeliness, and number of data items; file data monitoring focuses on file path, modification time, and associated data cycle; HTTP data monitoring covers multiple aspects such as URL, parameters, null values, and 404 errors. These refined monitoring rules help discover and solve problems that may arise in various links such as data collection, transmission, and storage.

[0051] By implementing measures such as data timeliness monitoring and data number monitoring, it is possible to promptly discover and process expired or abnormal data to ensure data accuracy and real-time performance. This is particularly important for application scenarios with extremely high timeliness requirements, such as meteorological data. Comprehensive data monitoring helps to promptly discover and resolve potential system failures or data problems, thereby preventing these problems from adversely affecting system stability and business continuity. In addition, through automated monitoring and alarm mechanisms, the possibility of manual intervention and errors can be reduced, further improving system reliability. Refined monitoring rules can help the system identify and handle data problems more accurately and avoid unnecessary waste of resources. At the same time, automated monitoring and alarm mechanisms can reduce the burden on operation and maintenance personnel and reduce operation and maintenance costs. Accurate and real-time data monitoring provides users with more reliable data support, which helps to improve user experience. At the same time, these data also provide strong support for decision-making such as weather forecasts and disaster warnings, helping to improve the scientificity and accuracy of decision-making.

[0052] In summary, the above technical solution achieves all-round monitoring of diverse data such as meteorological data by comprehensively covering multiple data types and protocols, formulating refined monitoring rules, and improving the accuracy and real-time performance of data monitoring, thereby enhancing the reliability, stability and user experience of the system.

[0053] In one embodiment of the present invention, different monitoring protocols are used to perform diversified data verification on different types of meteorological data to complete data monitoring of multiple types of meteorological data, including: S301, when there is periodic timing meteorological data within the effective time, determining the data state of the data node corresponding to the periodic timing meteorological data, and recording the data state of the data node; S302: When there is interval timing meteorological data within the effective time, determine the data status of the data node corresponding to the interval timing meteorological data, and record the data status of the data node; S303. When the type of the meteorological data is meteorological data without fixed time, the meteorological data is loaded to determine whether there is meteorological data without fixed time; when there is meteorological data without fixed time, the data status of the data node corresponding to the meteorological data without fixed time is determined, and the data status of the data node is recorded.

[0054] Specifically, determining the data status of the data nodes corresponding to the periodic timing meteorological data, the interval timing meteorological data, and the meteorological data without fixed time includes: Using a recursive algorithm, the data state corresponding to each layer of data nodes is obtained by recursively starting from the first node of the data stream and recursively until the end point of the data stream. The data status corresponding to the data node is returned layer by layer, and finally the monitoring status of the entire data flow is returned.

[0055] The working principle of the above technical solution is: starting from the first node N(1) of the data flow, it recursively returns to the data flow endpoint N(n), returns the data status of the data node layer by layer, and finally returns the monitoring status of the entire data flow. In order to ensure that users can quickly and conveniently customize the monitoring process, jQuery drag technology and SVG graphic painting technology are introduced in the front-end page, and the visualization and image of the monitoring process configuration are improved. In addition, during the execution of the background monitoring process, the Quartz open source framework is introduced to realize the concurrent execution of multiple data processes, and recursive indexing is implemented for each data flow to retrieve the data status of each data node in turn. Data flow node monitoring process Figure 5 shown.

[0056] The effect of the above technical solution is: the technical solution ensures the comprehensiveness of data monitoring by monitoring and recording the data status for three different types of meteorological data: periodic timing, interval timing and non-fixed time. Both regularly updated data and randomly appearing data can be effectively included in the monitoring scope, thereby improving the accuracy and reliability of data monitoring. By means of a recursive algorithm, starting from the first node of the data flow and recursively until the end point of the data flow, the data status of the data node is returned layer by layer, and finally the monitoring status of the entire data flow is returned. This real-time monitoring and dynamic feedback mechanism enables the system to detect and handle data anomalies or problems in a timely manner, improving the real-time and dynamic nature of data monitoring. The application of the recursive algorithm makes the data monitoring process more efficient and orderly. The system can automatically complete the traversal and status judgment of data nodes, reducing the possibility of manual intervention and errors, and improving the efficiency and performance of data monitoring.

[0057] Comprehensive data monitoring and timely exception handling can help reduce system failures or instability caused by data errors or anomalies. This improves the stability and reliability of the entire system and ensures the continuity and availability of meteorological data services. Accurate and comprehensive data monitoring results provide strong support for decisions such as meteorological forecasts and disaster warnings. Decision makers can adjust strategies and optimize resource allocation in a timely manner based on monitoring results to cope with possible meteorological changes or disaster risks. The implementation of this technical solution will help promote the standardization and standardization of meteorological data governance and quality management. By monitoring and recording data status, data quality problems can be discovered in a timely manner, and corresponding measures can be taken to improve and optimize, thereby improving data quality and availability.

[0058] In one embodiment of the present invention, the method for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming manner further includes: Step 1: monitor the data node operation parameters corresponding to each data node in real time; wherein the data node operation parameters include memory utilization, network bandwidth occupancy, data processing error rate and the number of acquisition processes per unit time; Step 2: Obtain the first node operation evaluation coefficient using the memory utilization rate and the network bandwidth occupancy rate; wherein the first node operation evaluation coefficient is obtained by the following formula:

[0059] in, C 01 Indicates the first node operation evaluation coefficient; n Indicates the number of unit time experienced by the data node; M p express n The average memory utilization corresponding to the unit time; M z express n The median memory utilization corresponding to the unit time; M min express n The minimum memory utilization corresponding to the unit time; M i Indicates i Memory utilization per unit time; B p express n The average value of network bandwidth usage corresponding to each unit time; B z express n The median value of network bandwidth usage corresponding to a unit of time; B min express n The minimum network bandwidth usage corresponding to a unit of time; B i Indicates i The network bandwidth usage corresponding to the unit time; S 01 and S 02 represent the first adjustment coefficient and the second adjustment coefficient respectively, and the first adjustment coefficient and the second adjustment coefficient are obtained by the following formula:

[0060] in, S 01 and S 02 represent the first adjustment coefficient and the second adjustment coefficient respectively; k 01 andk 02 Respectively represent the weight values ​​corresponding to the preset memory utilization and network bandwidth occupancy; f b express n The standard deviation of network bandwidth utilization corresponding to each unit of time; f m express n The standard deviation of memory utilization corresponding to unit time; Step 3: Compare the first node operation evaluation coefficient with a preset first evaluation threshold; Step 4: when the first node operation evaluation coefficient is lower than a preset first evaluation threshold, the number of acquisition processes per unit time is retrieved; Step 5: derive the second node operation evaluation coefficient by combining the number of processes obtained in the unit time with the first node operation evaluation coefficient; wherein the second node operation evaluation coefficient is obtained by the following formula:

[0061] in, C 02 represents the second node operation evaluation coefficient; n Indicates the number of unit time experienced by the data node; P i Indicates i The number of acquisition processes per unit time corresponding to the unit time; P c Indicates the reference value of the number of processes obtained within a preset unit time; f b express n The standard deviation of network bandwidth utilization corresponding to each unit of time; f m express n The standard deviation of memory utilization corresponding to unit time; f p express n The standard deviation of the number of processes obtained per unit time; C 01 Indicates the first node operation evaluation coefficient; C y01 represents a preset first evaluation threshold; P p express n The average number of processes obtained per unit time; P z express n Get the median value of the number of processes corresponding to the unit time; Step 6: Compare the second node operation evaluation coefficient with a preset second evaluation threshold; Step 7: When the second node operation evaluation coefficient exceeds a preset second evaluation threshold, it is determined that the data node operation is abnormal, and an abnormal data node operation alarm is issued.

[0062] The working principle of the above technical solution is as follows: the system continuously monitors the key operating parameters of each data node, including memory utilization, network bandwidth occupancy, data processing error rate, and the number of acquisition processes per unit time. Based on the average, median, and minimum values ​​of memory utilization and network bandwidth occupancy, as well as the preset weight value and standard deviation, the first node operation evaluation coefficient (C01) is calculated by a specific formula. This coefficient reflects the comprehensive performance of the data node in terms of memory and network resource utilization. The first node operation evaluation coefficient is compared with the preset first evaluation threshold. If it is lower than the threshold, it means that the data node may have performance bottlenecks or deficiencies in some aspects.

[0063] When the first node operation evaluation coefficient is lower than the threshold, the system further considers the number of acquisition processes per unit time as an evaluation factor. Combining the first node operation evaluation coefficient and the number of acquisition processes per unit time, the second node operation evaluation coefficient is calculated by another formula. This coefficient more comprehensively reflects the operation status of the data node, including resource utilization and process processing capacity. The second node operation evaluation coefficient is compared with the preset second evaluation threshold. If it exceeds the threshold, it means that the operation status of the data node is abnormal and an alarm is required. When an abnormal operation of a data node is detected, the system immediately issues an alarm so that the operation and maintenance personnel can intervene and solve the problem in time.

[0064] The effect of the above technical solution is: by monitoring the operating parameters of multiple dimensions and combining complex calculation models, the operating status of data nodes can be evaluated more accurately, and the comprehensiveness and accuracy of monitoring can be improved. Through real-time monitoring and abnormal alarm mechanisms, abnormal conditions in the operation of data nodes can be discovered in time, and corresponding treatment measures can be taken to avoid the expansion of problems. By evaluating key indicators such as memory utilization, network bandwidth occupancy, and the number of processes, operation and maintenance personnel can understand the resource utilization of data nodes, so as to make more reasonable resource allocation and optimization. By timely discovering and handling abnormal problems in the operation of data nodes, the stability and reliability of the entire system can be effectively improved, ensuring the continuity and efficiency of meteorological data services. Automated monitoring and alarm mechanisms can reduce the frequency and error rate of manual intervention, thereby reducing operation and maintenance costs and improving work efficiency.

[0065] The embodiment of the present invention proposes a comprehensive meteorological data multi-dimensional streaming centralized monitoring system, such as Figure 5As shown, the comprehensive meteorological data multi-dimensional streaming centralized monitoring system includes: An effective time acquisition module, used for acquiring effective time according to the meteorological data type; The data monitoring module is used to utilize different monitoring protocols to perform diverse data verification on different types of meteorological data, thereby completing data monitoring of various types of meteorological data.

[0066] The working principle of the above technical solution is: for comprehensive meteorological data of different types, different transmission methods and different formats (including meteorological data, cross-industry data, etc.), streaming monitoring of each node from collection, processing and application is realized, and a data link diagram is formed in a configurable way, so that staff can locate nodes with abnormal data in time and discover problem links in time.

[0067] The meteorological data transmission methods of the above technical solution mainly include HTTP protocol data, TCP protocol data, SMB protocol data, FTP protocol data, and SSH protocol. Different data types require different request protocols, and different reading and parsing methods are used for data attribute analysis according to the data formats corresponding to different data types, and finally the verification of diverse data is completed.

[0068] Meteorological data has a diverse structure and large differences. In order to achieve normalized monitoring, this project will determine the timeliness of the data. According to different data timeliness constraints (Currency Constraint), meteorological data will be divided into three types: periodic timing (such as real-time data of automatic stations every 5 minutes), interval timing (such as short-term weather from 4:50 to 21:50 every day), and no fixed time (such as early warning information). In the process of calculating data timeliness, the timeliness calculation function is used to classify the data, and the corresponding relationship between the current monitoring time and the data timeliness and data cycle is matched. When the time point exceeds the maximum monitoring timeliness, the set verification time and the valid time are cyclically verified, and finally the data timeliness monitoring algorithm within the valid time period is realized.

[0069] Through the recursive algorithm, starting from the first node of the data flow and recursively until the end point of the data flow, the data status of the data node is returned layer by layer, and finally the monitoring status of the entire data flow is returned.

[0070] The effect of the above technical solution is: by subdividing the meteorological data types (periodic timing, interval timing, no fixed time), and combining the data timeliness constraint (Currency Constraint), the validity and timeliness of the data can be judged more accurately. This helps to reduce decision-making errors caused by outdated or erroneous data and improve the accuracy and reliability of meteorological services.

[0071] By utilizing diverse data verification protocols, we conduct comprehensive and detailed verification of different types of meteorological data to ensure the integrity and accuracy of the data and further improve the quality of data monitoring.

[0072] Supports multiple data transmission protocols (HTTP, TCP, SMB, FTP, SSH, etc.), allowing the system to flexibly access meteorological data from different sources and in different formats, enhancing the system's compatibility and scalability. A data link diagram is formed in a configurable manner, allowing staff to intuitively understand the path and status of data flow, facilitating rapid location of problem nodes, and improving the flexibility and operability of data monitoring.

[0073] The use of streaming monitoring technology enables real-time collection, processing and monitoring of meteorological data, reduces data processing delays and waiting time, and improves the efficiency and performance of data monitoring. By returning the status of data nodes layer by layer through a recursive algorithm, the monitoring results of the entire data process can be quickly summarized, providing timely and comprehensive data support for decision makers.

[0074] The introduction of artificial intelligence technologies such as machine learning and deep learning can intelligently identify and analyze meteorological data, improve the accuracy and efficiency of data type identification, and provide a more intelligent means for data monitoring. By constructing a cross-correlated data relationship link diagram, the inherent connections and laws between data can be revealed, providing more comprehensive and in-depth information support for meteorological forecasting and decision-making. Efficient data monitoring and verification mechanisms can promptly detect and handle data anomalies, reducing system failures and operation and maintenance costs caused by data errors. Flexible monitoring configuration and visual data link diagrams reduce the difficulty and error rate of operation and maintenance personnel, and improve the stability and reliability of the system.

[0075] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming manner, characterized in that: The method for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming manner includes: Determining the type of meteorological data according to different data time constraints, wherein the types of meteorological data include periodic timing meteorological data, interval timing meteorological data and non-fixed time meteorological data; Acquire the effective time according to the meteorological data type; Different monitoring protocols are used to perform diverse data verification on different types of meteorological data, and complete data monitoring of various types of meteorological data.

2. The method for centralized monitoring of comprehensive meteorological data in multi-dimensional streaming format according to claim 1 is characterized in that: Obtaining the effective time according to the meteorological data type includes: When the type of the meteorological data is periodic timing meteorological data, determining the effective time corresponding to the periodic timing meteorological data; When the type of the meteorological data is interval timing meteorological data, the effective time corresponding to the interval timing meteorological data is determined.

3. The method for centralized monitoring of comprehensive meteorological data in multi-dimensional streaming format according to claim 2 is characterized in that: When the type of the meteorological data is periodic timing meteorological data, determining the effective time corresponding to the periodic timing meteorological data includes: When the type of the meteorological data is periodic timing meteorological data, the data parameter format is determined; Obtaining the minimum time and query cycle that meet the data delay time according to the data parameter format; Take the minimum time as the time starting point and the data parameter format as the unit to load the meteorological data corresponding to the effective time; After the meteorological data is loaded, n cycles of scanning are performed to determine whether periodic timing meteorological data exists within the effective time.

4. The method for centralized monitoring of comprehensive meteorological data in multi-dimensional streaming format according to claim 2 is characterized in that: When the type of the meteorological data is interval timing meteorological data, determining the effective time corresponding to the interval timing meteorological data includes: When the type of the meteorological data is interval timing meteorological data, the data validity time after subtracting the data delay time from the current time is obtained; Determine whether the time interval of the data validity time is less than the minimum interval; When the time interval of the data validity time is less than the minimum interval, the maximum interval of the previous day is extracted as the valid time; When the time interval of the data valid time is greater than the minimum interval, the time interval of the data valid time is extracted as the valid time; The meteorological data corresponding to the effective time is loaded to determine whether there is interval timing meteorological data within the effective time.

5. The method for centralized monitoring of comprehensive meteorological data in multi-dimensional streaming format according to claim 1 is characterized in that: The protocol types corresponding to each type of meteorological data include SMB protocol, SHH protocol, TCP protocol, FTP protocol and HTTP protocol; wherein the TCP protocol includes SQL Server, PostgreSQL, MySql and Oracle.

6. The method for centralized monitoring of comprehensive meteorological data in multi-dimensional streaming format according to claim 5 is characterized in that: The monitoring rules corresponding to each protocol type are as follows: The Sql statement rules corresponding to the monitoring principles of the database data corresponding to the monitoring rules include: data server ID monitoring, Sql statement monitoring, data timeliness monitoring, data number monitoring, data table name monitoring and time field monitoring; The file rules corresponding to the monitoring principles of the file data corresponding to the monitoring rules include: data server ID monitoring, file path monitoring or file modification time monitoring, whether the associated element has a cycle monitoring, data timeliness monitoring and file size monitoring; The rules for HTTP data corresponding to the monitoring rules include DATA set rules, null value rules and 404 error rules; The DATA set rules include URL monitoring, parameter monitoring, parameter rule monitoring, whether the data cycle is associated, and data timeliness monitoring; The null value rules include URL monitoring, parameter monitoring, parameter rule monitoring, whether the data cycle is associated and data timeliness monitoring; The 404 error rules include URL monitoring, parameter monitoring, parameter rule monitoring, associated data cycle monitoring and data timeliness monitoring.

7. The method for centralized monitoring of comprehensive meteorological data in multi-dimensional streaming format according to claim 1 is characterized in that: Different monitoring protocols are used to perform diverse data verification on different types of meteorological data, and complete data monitoring of various types of meteorological data, including: When periodic timing meteorological data exists within the effective time, the data state of the data node corresponding to the periodic timing meteorological data is determined, and the data state of the data node is recorded; When there is interval timing meteorological data within the effective time, the data state of the data node corresponding to the interval timing meteorological data is determined, and the data state of the data node is recorded; When the type of the meteorological data is meteorological data without fixed time, the meteorological data is loaded to determine whether there is meteorological data without fixed time; when there is meteorological data without fixed time, the data status of the data node corresponding to the meteorological data without fixed time is determined, and the data status of the data node is recorded.

8. The method for centralized monitoring of comprehensive meteorological data in multi-dimensional streaming format according to claim 7 is characterized in that: Determining the data status of the data nodes corresponding to the periodic timing meteorological data, the interval timing meteorological data and the meteorological data without fixed time includes: Using a recursive algorithm, the data state corresponding to each layer of data nodes is obtained by recursively starting from the first node of the data stream and recursively until the end point of the data stream. The data status corresponding to the data node is returned layer by layer, and finally the monitoring status of the entire data flow is returned.

9. The method for centralized monitoring of comprehensive meteorological data in multi-dimensional streaming format according to claim 1 is characterized in that: The method for centralized monitoring of comprehensive meteorological data in a multi-dimensional streaming manner also includes: Real-time monitoring of the data node operating parameters corresponding to each data node; wherein the data node operating parameters include memory utilization, network bandwidth occupancy, data processing error rate and the number of acquisition processes per unit time; The memory utilization rate and the network bandwidth occupancy rate are used to obtain the first node operation evaluation coefficient; wherein the first node operation evaluation coefficient is obtained by the following formula: in, C 01 Indicates the first node operation evaluation coefficient; n Indicates the number of unit time experienced by the data node; M p express n The average memory utilization corresponding to the unit time; M z express n The median memory utilization corresponding to the unit time; M min express n The minimum memory utilization corresponding to the unit time; M i Indicates i Memory utilization per unit time; B p express n The average value of network bandwidth usage corresponding to each unit time; B z express n The median value of network bandwidth usage corresponding to a unit of time; B min express n The minimum network bandwidth usage corresponding to a unit of time; B i Indicates i The network bandwidth usage corresponding to the unit time; S 01 and S 02 represent the first adjustment coefficient and the second adjustment coefficient respectively, and the first adjustment coefficient and the second adjustment coefficient are obtained by the following formula: in, S 01 and S 02 represent the first adjustment coefficient and the second adjustment coefficient respectively; k 01 and k 02 Respectively represent the weight values ​​corresponding to the preset memory utilization and network bandwidth occupancy; f b express n The standard deviation of network bandwidth utilization corresponding to each unit of time; f m express n The standard deviation of memory utilization corresponding to unit time; Comparing the first node operation evaluation coefficient with a preset first evaluation threshold; When the first node operation evaluation coefficient is lower than a preset first evaluation threshold, the number of acquisition processes per unit time is retrieved; The second node operation evaluation coefficient is obtained by combining the number of acquired processes in the unit time with the first node operation evaluation coefficient; wherein the second node operation evaluation coefficient is obtained by the following formula: in, C 02 represents the second node operation evaluation coefficient; n Indicates the number of unit time experienced by the data node; P i Indicates i The number of acquisition processes per unit time corresponding to the unit time; P c Indicates the reference value of the number of processes obtained within a preset unit time; f b express n The standard deviation of network bandwidth utilization corresponding to each unit of time; f m express n The standard deviation of memory utilization corresponding to unit time; f p express n The standard deviation of the number of processes obtained per unit time; C 01 Indicates the first node operation evaluation coefficient; C y01 represents a preset first evaluation threshold; P p express n The average number of processes obtained per unit time; P z express n Get the median value of the number of processes corresponding to the unit time; Comparing the second node operation evaluation coefficient with a preset second evaluation threshold; When the second node operation evaluation coefficient exceeds a preset second evaluation threshold, it is determined that the data node operation is abnormal, and an abnormal data node operation alarm is issued.

10. A comprehensive meteorological data multi-dimensional streaming centralized monitoring system, characterized in that: The comprehensive meteorological data multi-dimensional streaming centralized monitoring system includes: A type determination module, used to determine the type of meteorological data according to different data time constraints, wherein the types of meteorological data include periodic timing meteorological data, interval timing meteorological data and non-fixed time meteorological data; An effective time acquisition module, used for acquiring effective time according to the meteorological data type; The data monitoring module is used to utilize different monitoring protocols to perform diverse data verification on different types of meteorological data, thereby completing data monitoring of various types of meteorological data.

Images