Meteorological data intelligent processing system and method

Abstract

This invention discloses an intelligent meteorological data processing system, comprising: a meteorological data parsing and storage subsystem, which implements two-phase commit based on Kafka middleware for receiving and parsing multi-source heterogeneous meteorological data; a business configuration subsystem, which provides a rule configuration interface for dynamically configuring meteorological data processing rules and synchronizing the processing rules to business nodes in real time; a business monitoring subsystem, which collects business indicators in real time through Kafka log streams; and a distributed data storage subsystem, which uses a distributed MySQL cluster to store meteorological data and caches it through Redis. This invention solves the problems of complex operation and maintenance, data backlog, poor scalability, and low reliability of traditional meteorological decoding systems by dynamically configuring business rules, distributed meteorological data decoding, distributed storage architecture, real-time monitoring and alarm mechanism, and fault tolerance function.

Description

Technical Field

[0001] This invention relates to the field of meteorological data processing technology, specifically to an intelligent meteorological data processing system and method. Background Technology

[0002] Traditional meteorological data processing systems suffer from numerous technical shortcomings: First, given the diverse types of meteorological data, each data point requires a separate decoding program, creating a static binding between the data and the processing process. This necessitates maintaining a large number of processes and ports, and prevents backup functionality between processes within the same group, resulting in high system maintenance complexity. Second, the serial call mechanism between the receiving and parsing processes limits high-speed reception processing to low-speed parsing, easily leading to data backlog. Third, the tight coupling between business logic and system architecture hinders system performance scalability, failing to meet the ever-increasing demands for meteorological data processing. Furthermore, business configuration relies on static configuration files, requiring a service restart for changes to take effect, severely impacting business continuity. Most importantly, existing systems lack effective reliability assurance mechanisms, failing to achieve automatic failover in case of node failures, significantly affecting the timeliness and accuracy of meteorological data processing. These problems severely restrict the efficient operation of meteorological services, urgently requiring an innovative solution to address these technical issues. Therefore, existing technologies urgently need improvement. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a meteorological data intelligent processing system and method to solve the problems existing in the prior art.

[0004] To achieve the above and other related objectives, the present invention provides an intelligent meteorological data processing system, comprising:

[0005] The meteorological data parsing and database entry subsystem is based on a distributed architecture to receive and parse multi-source heterogeneous meteorological data, save intermediate parsing results to a Kafka cluster, and finally store the results in a distributed database.

[0006] The business configuration subsystem is used to provide a rule configuration interface to dynamically configure the processing rules for meteorological data and to synchronize the processing rules to the business nodes in real time.

[0007] The business monitoring subsystem is used to collect business metrics in real time through Kafka log streams.

[0008] The distributed data storage subsystem is used to store meteorological data using a distributed MySQL cluster and cache it using Redis;

[0009] The meteorological data query service subsystem is used to provide query services for detailed information on each processing node and business statistical indicators of the meteorological data that has been entered into the database.

[0010] In one embodiment of this application, the meteorological data parsing and storage subsystem includes:

[0011] The receiving cluster is used to preprocess the received multi-source heterogeneous meteorological data, format it into a unified meteorological data message, allocate independent Kafka message topics according to the meteorological data type, and write it into partitions to achieve parallel processing; the receiving cluster supports FTP files, BUFR format ground automatic station messages based on RabbitMQ message queues, and MDOS quality control messages based on RabbitMQ.

[0012] The parsing cluster subscribes to all message topics in Kafka consumer group mode, achieves parallel parsing of meteorological data through partitioning, and stores the parsing results in the database and files.

[0013] In one embodiment of this application, the meteorological data parsing and database entry subsystem, during the process of receiving and parsing multi-source heterogeneous meteorological data, implements a two-phase commit based on the Kafka middleware, specifically including:

[0014] In the first phase, for structured data, the packaged meteorological data messages are processed and quickly split at the site level. The split business data is then written to the corresponding Kafka topic based on the meteorological message type. Specifically, Kafka topics are created based on different meteorological data, and each topic is configured with multiple partitions. For unstructured data files, messages are generated based on metadata information and written to the corresponding Kafka topic. The metadata information includes: data type, file name, file size, storage location, and archive location.

[0015] In the second phase, for structured data, based on the configured data type, the message messages that have been split into individual stations are consumed from the Kafka topic corresponding to the weather report; based on the configured data type, the corresponding data decoding class is dynamically loaded using Java reflection to complete the data decoding; after the data decoding is completed, the fields of the structured data are written to the configured database storage table; for unstructured data, file metadata messages are consumed from the corresponding Kafka topic, and the files are moved to the archive directory according to the archive location settings for this type of file, and the file name and size information are written to the configured database storage table.

[0016] In one embodiment of this application, the service configuration subsystem includes:

[0017] The business configuration client provides a web interface for dynamically configuring business rules.

[0018] The business configuration service is used to call the configuration service interface to persist configuration data to the configuration database; at the same time, it writes the business rules after configuration changes to a predefined path.

[0019] Business nodes are used to capture change events of the predefined paths through a listening mechanism, dynamically load new configurations, and adjust business rules in real time.

[0020] In one embodiment of this application, the modified business rules are published through ZooKeeper's predefined paths.

[0021] In one embodiment of this application, the business monitoring subsystem includes:

[0022] The data receiving and preprocessing receiver module is used to receive structured and unstructured meteorological data, write the data metadata information to the Kafka log partition, and unpack the packaged surface meteorological data packets at the individual station level.

[0023] The data decoding parser module is used to write the metadata information corresponding to the weather report into the decoding operation log topic of Kafka after decoding the weather report. The metadata information corresponding to the report is obtained by the log storage subprocess consuming log data streams from the Kafka log partition, processing the metadata information and operation logs of each processing step according to the fixed format of the log.

[0024] The business analysis analyzer module is used to record detailed logs of the entire process of data reception, preprocessing, parsing, and archiving in the system. It is also used to consume log data streams from Kafka log partitions, perform time window aggregation statistics through the streaming computing engine, and obtain business indicators on the hourly and daily time scales. It is also used for data monitoring and statistical analysis of the timeliness of reception and processing.

[0025] In one embodiment of this application, the business monitoring subsystem further includes an alarm module, which is used to trigger multi-level alarms based on preset thresholds according to Kafka streaming computation.

[0026] In one embodiment of this application, the meteorological data intelligent processing system further includes:

[0027] A dynamic scheduling cluster is used to allocate tasks to a receiving cluster, the dynamic scheduling cluster comprising:

[0028] The master-slave election module is used to elect active nodes based on the ZooKeeper ephemeral node competition mechanism, while standby nodes continuously monitor the status of active nodes. Among them, active nodes are used to monitor the receiving directory, generate tasks according to round-robin or load balancing strategies, and distribute tasks to the receiving cluster.

[0029] The failover module is used to trigger the election of a new active node when the active node fails, and to redistribute tasks to surviving nodes when the receiving node fails.

[0030] This application provides a method for intelligent processing of meteorological data, including:

[0031] The receiving node formats the received meteorological data into a unified meteorological data message, assigns independent message topics according to the meteorological data type, and writes them into the corresponding Kafka partition to achieve parallel processing.

[0032] The parsing cluster subscribes to all message topics in Kafka consumer group mode, and achieves parallel parsing and storage of parsing results through partitioning.

[0033] In one embodiment of the present invention, the method further includes:

[0034] The receiving node's processing program constructs a thread pool for the first-stage preprocessing of meteorological data, a thread pool for decoding structured meteorological data, a thread pool for operating unstructured meteorological data, and a thread pool for decoding log processing, thereby achieving efficient and concurrent processing of meteorological data.

[0035] As described above, the intelligent meteorological data processing system and method provided by the present invention have the following beneficial effects:

[0036] The present invention provides an intelligent meteorological data processing system that solves the problems of complex operation and maintenance, data backlog, poor scalability, and low reliability of traditional systems by dynamically configuring business rules, distributed processing, distributed storage architecture, real-time monitoring and alarm mechanism, and fault transfer function. It has the advantages of simplifying operation and maintenance complexity, avoiding data backlog, improving system scalability, supporting dynamic configuration adjustment, and ensuring processing reliability.

[0037] It also features flexible distribution, including: 1. A peer-to-peer model for the Receiver and Parser clusters, with all nodes in the cluster being Active; the Scheduler and Analyzer clusters operate in a master-slave mode, ensuring high availability. 2. The decoding cluster can be horizontally scaled and adjusted. In actual production environments, running nodes can be dynamically added or removed based on actual workload. 3. Multiple deployment strategies: a. Both the Receiver and Parser programs can be deployed on each node simultaneously; b. Depending on the node's hardware configuration, the Receiver can be deployed on a high-disk-IO host, while the Parser program can be deployed on a host with high CPU and memory performance.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A schematic diagram of a meteorological data intelligent processing system is shown as an exemplary embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram illustrating a two-phase commit implementation based on Kafka middleware, as an exemplary embodiment of the present invention.

[0042] Figure 3 The flowchart illustrates an intelligent meteorological data processing method as an exemplary embodiment of the present invention. Detailed Implementation

[0043] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0044] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0045] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0046] In existing technologies, meteorological data processing systems generally adopt a statically bound business architecture, resulting in a fixed correspondence between processing processes and port resources. Under this architecture, a large number of processes require individual maintenance and cannot achieve intra-group redundancy backup, leading to difficulties in resource allocation during system expansion. When high-speed data receiving modules and low-speed parsing modules use a synchronous call mechanism, processing efficiency is limited by the slowest step, easily causing data backlog. Business rule configurations rely on static file modifications, requiring a service restart to take effect after updates, making it difficult to meet real-time requirements. Furthermore, the system lacks an effective status monitoring mechanism, making it difficult to detect and handle faults in a timely manner.

[0047] To address these issues, the developers discovered that the core problem lay in the mismatch between the rigid coupling of the system architecture and the asynchronous requirements of the data processing flow. First, they considered decoupling business components into independent service units, using message queues to achieve asynchronous communication between modules. Second, to address the issue of delayed configuration updates, they designed a visual configuration interface and a real-time push mechanism. Third, to improve system reliability, they introduced a distributed storage architecture and a multi-level monitoring system. Ultimately, this resulted in a layered architecture solution with message-driven processing at its core and dynamic configuration loading as support.

[0048] like Figure 1 As shown, this application provides a meteorological data intelligent processing system, including: a meteorological data parsing and storage subsystem, a business configuration subsystem, a business monitoring subsystem, a distributed data storage subsystem, and a meteorological data query service subsystem;

[0049] The meteorological data parsing and database entry subsystem is used to receive and parse multi-source heterogeneous meteorological data based on a distributed architecture, save intermediate parsing results to a Kafka cluster, and finally store the results in a distributed database.

[0050] The meteorological data parsing and database entry subsystem refers to the cluster of modules that realize standardized data processing. It can use a distributed message queue to achieve asynchronous decoupling of data reception and parsing, for example, by dividing the processing channels for different data types through the Kafka message middleware.

[0051] The business configuration subsystem provides a rule configuration interface to dynamically configure the processing rules for meteorological data and synchronizes the processing rules to business nodes in real time.

[0052] The business configuration subsystem refers to the functional module that supports dynamic rule management. It can provide a visual configuration interface through web services and use distributed coordination services to push configuration changes in real time.

[0053] The business monitoring subsystem is used to collect business metrics in real time through Kafka log streams.

[0054] The business monitoring subsystem refers to a monitoring system that collects operational metrics in real time and can perform time-window-based aggregation analysis on log data through a streaming computing engine.

[0055] The distributed data storage subsystem is used to store meteorological data using a distributed MySQL cluster and cache it using Redis;

[0056] The meteorological data query service subsystem is used to provide query services for meteorological data.

[0057] Among them, the meteorological data query service subsystem is a service module that provides data access interfaces to the outside world.

[0058] Specifically, when meteorological data enters the system, the receiving cluster converts the raw data into the standard BUFR message format and distributes it to different message queue partitions according to data type for parallel reception. The parsing cluster processes messages in the queue in parallel using a consumer group mode, and stores the data in the sharded database after parsing. Business rule changes are submitted through the configuration interface and pushed to each business node in real time to take effect. The monitoring system continuously collects the operation logs of the processing nodes and uses streaming computing to calculate key indicators such as throughput and processing latency. The data storage layer designs a sharding strategy based on data type characteristics and combines a caching mechanism to improve the response speed of frequently accessed data. The query service provides a unified data retrieval interface to the outside world through a load balancing mechanism.

[0059] It should be noted that the configuration interface can provide changes to business configurations via a web interface.

[0060] In one embodiment, the meteorological data parsing and storage subsystem includes: a receiving cluster and a parsing cluster;

[0061] The receiving cluster is used to preprocess the received multi-source heterogeneous meteorological data, format it into a unified meteorological data message, allocate independent Kafka message topics according to the meteorological data type, and write it into multiple partitions of the Kafka message topic to achieve parallel processing; the receiving cluster supports pushing unparsed meteorological report files (structured data) and other types of meteorological product files (unstructured data) via FTP file push, BUFR format automatic ground station messages based on RabbitMQ message queue, and MDOS quality control messages based on RabbitMQ;

[0062] The receiving cluster is a distributed cluster composed of multiple receiving nodes. A load balancing strategy can be used to dynamically distribute receiving tasks to different nodes, thereby achieving high-concurrency data receiving capabilities. Kafka partitioning refers to dividing messages of different meteorological data types into independent message queue partitions. Hash algorithms or round-robin strategies can be used for partition allocation to ensure that data of the same type is processed centrally.

[0063] The parsing cluster (also a distributed cluster: multiple parsers belong to the same Kafka Consumer Group, each parser subscribes to all data topics, and each parser has the ability to parse all meteorological business data) is used to subscribe to all message topics in Kafka consumer group mode, realize parallel parsing of meteorological data through partitioning, and store the parsing results in the database and files.

[0064] Specifically, after receiving raw meteorological data, the receiving cluster first converts it into a standardized message structure containing metadata such as reception time and data type. Then, it selects the corresponding Kafka topic based on the data type and writes the messages to different partitions using a partitioning strategy. The parsing cluster's nodes subscribe to all topics as consumer groups. Each node can consume messages from all topics, with Kafka automatically assigning the appropriate partitions. The parsing results are directly stored in the database.

[0065] This invention achieves asynchronous processing through Kafka message queues, with the receiving cluster and parsing cluster running independently, and both clusters can increase processing throughput by adding more nodes.

[0066] The parsing cluster encapsulates common operations involved in decoding, such as file reception, message reception, and log writing to Kafka. When a new type of meteorological data is added: during the development phase, developers only need to focus on implementing the business logic for decoding meteorological reports, without needing to worry about the entire decoding data process; during the deployment phase, users only need to add meteorological data metadata information in the configuration interface and upload the compiled meteorological data decoding class file to the folder of all running nodes. This increases the convenience of meteorological data development and deployment, while also increasing the versatility and adaptability of the decoding cluster.

[0067] In one embodiment, the meteorological data parsing and database entry subsystem implements a two-phase commit based on the Kafka middleware during the process of receiving and parsing multi-source heterogeneous meteorological data. Please refer to [link to relevant documentation]. Figure 2 Implementing two-phase commit based on Kafka middleware specifically includes:

[0068] In the first phase, for structured data, the packaged meteorological data messages are quickly split at the site level, and the split business data is written to the corresponding Kafka topic according to the meteorological message type. Among them, Kafka topics are created according to different meteorological data, and each topic is configured with multiple partitions. For unstructured data files, messages are generated based on metadata information, and the messages are written to the corresponding Kafka topic. Metadata information includes: data type, file name, file size, storage location, and archive location.

[0069] In the second phase, for structured data, based on the configured data type, the message messages that have been split into individual stations are consumed from the Kafka topic corresponding to the weather report; based on the configured data type, the corresponding data decoding class is loaded using Java reflection to complete the data decoding; after the data decoding is completed, the fields of the structured data are written to the configured database storage table; for unstructured data, file metadata messages are consumed from the corresponding Kafka topic, and the files are moved to the archive directory according to the archive location settings for this type of file, and the file name and size information are written to the configured database storage table.

[0070] In one embodiment, the service configuration subsystem includes: a service configuration client, a service configuration service, and a service node;

[0071] The business configuration client provides a web interface for dynamically configuring business rules.

[0072] The business configuration client is an interactive interface built on web technologies. It can be implemented using the React or Vue framework and is used for visual editing of business rule parameters.

[0073] The business configuration service is used to call the configuration service interface to persist configuration data to the configuration database; at the same time, it writes the business rules after configuration changes to a predefined path.

[0074] The business configuration service refers to the backend service module, which can be implemented using a Spring Cloud microservice architecture. It receives configuration data submitted by the client via a RESTful interface and stores the data in a MySQL database, ensuring persistent storage of configuration information. The predefined path is a specified data node, which can be implemented using persistent nodes in ZooKeeper to store configuration change information and trigger event notifications.

[0075] Business nodes are used to capture changes to predefined paths through a listening mechanism, dynamically load new configurations, and adjust business rules in real time.

[0076] Specifically, when a user modifies business rules through the web interface of the business configuration client, the configuration data is transmitted to the business configuration service via the HTTP protocol. The business configuration service persists the data to the configuration database and simultaneously writes the updated rule content to a predefined path in ZooKeeper. Business nodes register Watchers to listen to this path. Upon detecting a data change event, they immediately pull the latest configuration from the predefined path and load it into memory, enabling the business rules to take effect in real time. During this process, business nodes can complete the configuration update without restarting, and each node completes the loading operation independently, avoiding single points of failure affecting the overall system.

[0077] In one embodiment, the modified business rules are published via a predefined path in ZooKeeper.

[0078] ZooKeeper's predefined paths are node paths established based on the distributed coordination service, with each path node storing business rule configuration information.

[0079] Specifically, the business configuration subsystem serializes the updated business rules into data objects and writes them to a predefined persistent node path in the ZooKeeper service. Business nodes monitor data change events on this path in real time by registering Watcher listeners. When a change in node data is detected, the nodes proactively fetch the latest configuration data and load it into memory. This mechanism allows business rule changes to take effect without restarting the service. For example, when the meteorological data processing threshold needs adjustment, operations personnel only need to modify the parameters in the configuration interface, and the modified rules are broadcast to all business nodes via the ZooKeeper node.

[0080] In one embodiment, the business monitoring subsystem includes: a data receiving and preprocessing receiver module, a data decoding parser module, and a business analysis analyst module;

[0081] The data receiving and preprocessing receiver module is used to receive structured and unstructured meteorological data, write the data metadata information into the Kafka log topic, and unpack the packaged surface meteorological data messages at the individual station level.

[0082] The data decoding parser module is used to write the metadata information corresponding to the weather report into the Kafka decoding operation log topic after the weather report is decoded. The metadata information corresponding to the report is obtained by the log storage subprocess consuming log data streams from the Kafka log topic, processing the metadata information and operation logs of each processing step according to the fixed format of the log.

[0083] The business analysis analyst module is used to record detailed logs of the entire process of data reception, preprocessing, parsing, and archiving in the system. It is also used to consume log data streams from Kafka log topics, perform time window aggregation statistics through the streaming computing engine, and obtain business indicators on the hourly and daily time scales. It is also used for data monitoring and statistical analysis of reception and processing timeliness.

[0084] Specifically, the data writing module pushes operation logs generated by business nodes to the Kafka message queue in real time, with each log entry carrying a timestamp to ensure time sequence. The business analysis module obtains the consumer log data stream by subscribing to Kafka topics, and performs time window aggregation statistics through the streaming computing engine to obtain hourly and daily time-scale business metrics.

[0085] In one embodiment, the business monitoring subsystem further includes an alarm module, which is used to trigger multi-level alarms based on preset thresholds according to Kafka streaming computation.

[0086] Among them, preset thresholds refer to pre-defined critical values ​​for business indicators, which can be dynamically configured to adjust the triggering conditions of different indicators according to business needs. Multi-level alarms refer to a tiered alarm mechanism, which can be implemented using a combination of multiple channels such as email, SMS, and system notifications. Different alarm levels correspond to different response times and processing priorities.

[0087] For example, if the false alarm rate is less than the CTS false alarm rate threshold 1, the first alarm rule is used; if the false alarm rate is between the CTS false alarm rate threshold 1 and the CTS false alarm rate threshold 2, the second alarm rule is used; if the false alarm rate is greater than the CTS false alarm rate threshold 2, the third alarm rule is used.

[0088] For example, if the timeliness rate is less than the timeliness rate threshold 1, the third alarm rule is used; if the timeliness rate is greater than or equal to the timeliness rate threshold 1 and less than the timeliness rate threshold 2, the second alarm rule is used; if the timeliness rate is greater than or equal to the timeliness rate threshold 2, the first alarm rule is used.

[0089] Specifically, the business analysis module pushes business metrics generated by the streaming computing engine to the alarm module, which compares the metric data with preset thresholds in real time. When a metric exceeds the first-level threshold, a primary alarm is triggered, and maintenance personnel are notified internally via the system. If the metric continues to deteriorate and reaches the second-level threshold, an SMS alarm is triggered to notify on-duty personnel. When the metric exceeds the third-level threshold, a telephone alarm is triggered and the emergency plan is activated. The alarm levels and threshold ranges use a dynamic mapping mechanism, and the threshold parameters are updated in real time through the configuration center.

[0090] In one embodiment, the meteorological data intelligent processing system further includes: a dynamic scheduling cluster, a master-slave election module, and a failover module;

[0091] A dynamic scheduling cluster is used to allocate tasks to the receiving cluster. The dynamic scheduling cluster includes:

[0092] Among them, the dynamic scheduling cluster is a system architecture that realizes dynamic task allocation based on distributed coordination services. It can use ZooKeeper's ephemeral node mechanism to achieve master-slave election and realize task scheduling through node status monitoring. Its role is to eliminate the static binding relationship between business and processing process and realize dynamic load balancing.

[0093] The master-slave election module is used to elect active nodes based on the ZooKeeper ephemeral node competition mechanism, while standby nodes continuously monitor the status of active nodes. Among them, active nodes are used to monitor the receiving directory, generate tasks according to round-robin or load balancing strategies, and distribute tasks to the receiving cluster.

[0094] The master-slave election module is a functional unit that determines a unique active node through a competition mechanism, which can be implemented using ZooKeeper's temporary ordered node feature. When an active node goes offline abnormally, the backup node can immediately detect this and trigger a re-election, ensuring the continuity of task scheduling. The active node dynamically generates a task allocation strategy through a round-robin algorithm or real-time load metrics, which aims to avoid bottlenecks caused by a single node and improve system throughput.

[0095] The failover module is used to trigger the election of a new active node when the active node fails, and to redistribute tasks to surviving nodes when the receiving node fails.

[0096] Specifically, the failover module determines failures by using heartbeat detection and task status tracking, and then reassigns unfinished tasks to healthy nodes.

[0097] Specifically, during operation, the ZooKeeper cluster triggers a master-slave election by creating temporary ordered nodes to obtain the active node. The active node continuously scans and receives meteorological data files, generating task allocation instructions according to preset channel policies, such as using a round-robin approach to evenly distribute tasks to multiple receiving nodes in the cluster. When a receiving node experiences task processing anomalies due to hardware failure or network interruption, the failover module detects the node's failure status through a heartbeat timeout mechanism and automatically reassigns unfinished tasks to other surviving receiving nodes. If the active node itself fails, ZooKeeper immediately removes its temporary node, triggering a backup node to re-initiate the election process, ensuring uninterrupted operation of the scheduling service.

[0098] This invention decouples task allocation from node status through dynamic scheduling of the cluster. The receiving nodes can be horizontally expanded and serve as backups for each other. When the processing capacity of the parsing cluster is insufficient, the dynamic scheduling cluster can automatically reduce the task allocation rate or increase the number of receiving nodes to avoid data backlog.

[0099] In one embodiment, during the system development process, three base classes are encapsulated for developers: a receiving preprocessing module, a decoding and database insertion module, and a log generation class. These classes implement basic functions such as message preprocessing, SQL statement generation for database insertion, and log writing. This allows developers to focus on and complete the decoding business logic for new meteorological data, greatly reducing the development cycle for decoding new data.

[0100] Please see Figure 3 , Figure 3 This is a flowchart illustrating an intelligent meteorological data processing method according to an embodiment of the present invention. Figure 3 In China, intelligent processing methods for meteorological data include:

[0101] Step S310: The scheduling cluster master node monitors the receiving directory, generates tasks according to the channel policy, and assigns them to the receiving nodes.

[0102] In step S320, the receiving node formats the received meteorological data into a unified meteorological data message, allocates independent message topics according to the meteorological data type, and writes them into the corresponding partition of Kafka to achieve parallel processing.

[0103] Step S330: The parsing cluster subscribes to all message topics in Kafka consumer group mode, and achieves parallel parsing and storage of parsing results through partition allocation.

[0104] Specifically, the scheduling cluster master node continuously monitors newly added meteorological data files in the receiving directory and generates task allocation instructions based on preset channel numbers. For example, it allocates data from channel A to node 1 and data from channel B to node 2. Upon receiving the allocation task, the receiving nodes convert the raw meteorological data into a standardized structure containing metadata and a message body. They then select the corresponding Kafka topic based on the data category; for example, ground data is written to the ground topic, and radar data to the radar topic. Each topic has multiple partitions to achieve horizontal scaling. The parsing cluster's nodes subscribe to all topics in a consumer group mode and automatically obtain data to be processed through Kafka's partition allocation mechanism. For example, node A processes partitions 0-2 of topic 1, and node B processes partitions 3-5 of topic 1, achieving stateless horizontal scaling capabilities.

[0105] This invention removes process binding through a dynamic task allocation mechanism and uses Kafka message queues to achieve asynchronous processing. After the receiving node completes data formatting, it immediately writes the data to the message queue. The parsing cluster consumes the data according to its own processing capacity, avoiding the problem of high-speed reception being limited by low-speed parsing.

[0106] In one embodiment, the method further includes:

[0107] The receiving node's processing program constructs a thread pool for the first-stage preprocessing of meteorological data, a thread pool for decoding structured meteorological data, a thread pool for operating unstructured meteorological data, and a thread pool for decoding log processing, thereby achieving efficient and concurrent processing of meteorological data.

[0108] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0109] It should be noted that the intelligent meteorological data processing method provided in the above embodiments and the intelligent meteorological data processing system provided in the above embodiments belong to the same concept. The specific methods of operation of each module and unit have been described in detail in the embodiments of the intelligent meteorological data processing system, and will not be repeated here. In practical applications, the functions provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0110] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A meteorological data intelligent processing system, characterized in that, include: The meteorological data parsing and database entry subsystem is based on a distributed architecture to receive and parse multi-source heterogeneous meteorological data, save intermediate parsing results to a Kafka cluster, and finally store the results in a distributed database. The business configuration subsystem is used to provide a rule configuration interface to dynamically configure the processing rules for meteorological data and to synchronize the processing rules to the business nodes in real time. The business monitoring subsystem is used to collect business metrics in real time through Kafka log streams. The distributed data storage subsystem is used to store meteorological data using a distributed MySQL cluster and cache it using Redis; The meteorological data query service subsystem is used to provide query services for detailed information on each processing node and business statistical indicators of the meteorological data that has been entered into the database.

2. The meteorological data intelligent processing system according to claim 1, characterized in that: The meteorological data parsing and database entry subsystem includes: The receiving cluster is used to preprocess the received multi-source heterogeneous meteorological data, format it into a unified meteorological data message, allocate independent Kafka message topics according to the meteorological data type, and write it into partitions to achieve parallel processing; the receiving cluster supports FTP files, BUFR format ground automatic station messages based on RabbitMQ message queues, and MDOS quality control messages based on RabbitMQ. The parsing cluster is used to subscribe to all message topics in Kafka consumer group mode, realize parallel parsing of meteorological data through partitioning, and store the parsing results in the database and files.

3. The intelligent meteorological data processing system according to claim 2, characterized in that: The meteorological data parsing and database entry subsystem, in the process of receiving and parsing multi-source heterogeneous meteorological data, implements a two-phase commit based on the Kafka middleware, specifically including: In the first phase, for structured data, the packaged meteorological data messages are quickly split at the site level, and the split business data is written to the corresponding Kafka topic according to the meteorological message type. Specifically, Kafka topics are created based on different meteorological data, and each topic is configured with multiple partitions. For unstructured data files, messages are generated based on metadata information, and the messages are written to the corresponding Kafka topic. The metadata information includes: data type, file name, file size, storage location, and archive location. In the second phase, for structured data, based on the configured data type, the message messages that have been split into individual stations are consumed from the Kafka topic corresponding to the weather report; based on the configured data type, the corresponding data decoding class is dynamically loaded using Java reflection to complete the data decoding; after the data decoding is completed, the fields of the structured data are written to the configured database storage table; for unstructured data, file metadata messages are consumed from the corresponding Kafka topic, and the files are moved to the archive directory according to the archive location settings for this type of file, and the file name and size information are written to the configured database storage table.

4. The intelligent meteorological data processing system according to claim 1, characterized in that: The service configuration subsystem includes: The business configuration client provides a web interface for dynamically configuring business rules. The business configuration service is used to call the configuration service interface to persist configuration data to the configuration database; at the same time, it writes the business rules after configuration changes to a predefined path. Business nodes are used to capture change events of the predefined paths through a listening mechanism, dynamically load new configurations, and adjust business rules in real time.

5. The intelligent meteorological data processing system according to claim 4, characterized in that, The modified business rules are published via ZooKeeper's predefined paths.

6. The intelligent meteorological data processing system according to claim 1, characterized in that, The business monitoring subsystem includes: The data receiving and preprocessing receiver module is used to receive structured and unstructured meteorological data, write the data metadata information to the Kafka log partition, and unpack the packaged surface meteorological data packets at the individual station level. The data decoding parser module is used to write the metadata information corresponding to the weather report into the decoding operation log topic of Kafka after decoding the weather report. The metadata information corresponding to the report is obtained by the log storage subprocess consuming log data streams from the Kafka log partition, processing the metadata information and operation logs of each processing step according to the fixed format of the log. The business analysis analyst module is used to record detailed logs of the entire process of data reception, preprocessing, parsing, and archiving in the system. It is also used to consume log data streams from Kafka log partitions, perform time window aggregation statistics through the streaming computing engine, and obtain business indicators on the hourly and daily time scales. It is also used for data monitoring and statistical analysis of the timeliness of reception and processing.

7. The intelligent meteorological data processing system according to claim 6, characterized in that, The business monitoring subsystem also includes an alarm module, which is used to trigger multi-level alarms based on preset thresholds according to Kafka streaming computation.

8. The intelligent meteorological data processing system according to claim 2, characterized in that, The intelligent meteorological data processing system also includes: A dynamic scheduling cluster is used to assign tasks to a receiving cluster, the dynamic scheduling cluster comprising: The master-slave election module is used to elect active nodes based on the ZooKeeper ephemeral node competition mechanism, while standby nodes continuously monitor the status of active nodes. Among them, active nodes are used to monitor the receiving directory, generate tasks according to round-robin or load balancing strategies, and distribute tasks to the receiving cluster. The failover module is used to trigger the election of a new active node when the active node fails, and to redistribute tasks to surviving nodes when the receiving node fails.

9. A method for intelligent processing meteorological data based on the intelligent meteorological data processing system according to any one of claims 1-8, characterized in that, include: The receiving node formats the received meteorological data into a unified meteorological data message, assigns independent message topics according to the meteorological data type, and writes them into the corresponding Kafka partition to achieve parallel processing. The parsing cluster subscribes to all message topics in Kafka consumer group mode, and achieves parallel parsing and storage of parsing results through partitioning.

10. The intelligent meteorological data processing method according to claim 9, characterized in that, The method further includes: The receiving node's processing program constructs a thread pool for the first-stage preprocessing of meteorological data, a thread pool for decoding structured meteorological data, a thread pool for operating unstructured meteorological data, and a thread pool for decoding log processing, thereby achieving efficient and concurrent processing of meteorological data.

Images