Iron tower data rapid quality control method and system based on message transmission

Abstract

The invention provides an iron tower data rapid quality control method and system based on message transmission. The method comprises the following steps: collecting observation data; constructing an asynchronous data transmission channel based on a message queue according to the observation data; a message-driven architecture is adopted, and each node of a quality control processing module deployed in a server cluster is used as a consumer; continuously monitoring through a quality control processing module, and pulling a message from the original data queue; uniformly distributing the messages in the original data queue to each consumer through a load balancing mechanism of RabbitMQ; performing key field extraction on the delivered message by each node to obtain a plurality of pieces of observation original data; inputting all the observation original data into a quality control algorithm for detection to obtain quality control data corresponding to all the observation original data; generating a quality control result based on the quality control data corresponding to all the observation original data; and packaging the quality control result and the original data into a new data packet, and distributing a corresponding quality control code to the new data packet.

Description

Technical Field

[0001] This invention relates to the field of meteorological observation technology, and in particular to a rapid quality control method and system for tower data based on message transmission. Background Technology

[0002] Currently, traditional urban meteorological observation systems rely on regional automatic stations built by meteorological departments, with low station density (usually 5-10 kilometers apart), and data is uploaded to the central platform periodically via HTTP / FTP.

[0003] Existing technologies suffer from systemic problems such as insufficient spatial coverage, lagging data transmission and quality control, and poor system scalability: traditional dedicated site resources are scarce, making it difficult to achieve dense deployment at the kilometer level, and there are blind spots in monitoring small- and medium-scale phenomena such as local strong winds and urban heat islands; synchronous transmission and batch processing quality control modes result in high data latency and poor timeliness, which cannot meet minute-level business needs; new sites require independent construction of communication and power supply facilities, resulting in long integration cycles and difficulties in expansion. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method and system for rapid quality control of tower data based on message transmission. It utilizes tower stations to achieve dense spatial coverage, constructs an efficient data flow channel through message transmission mode, and then achieves real-time rapid quality control of data through a message-driven rapid quality control module. The three form a closed-loop optimization of the entire "acquisition-transmission-quality control" chain.

[0005] In a first aspect, embodiments of the present invention provide a rapid quality control method for tower data based on message transmission, the method comprising: Observational data is collected using a meteorological sensor kit; An asynchronous data transmission channel based on a message queue is constructed based on the observed data; A message-driven architecture is adopted, with each node of the quality control processing module deployed in the server cluster acting as a consumer; The quality control processing module continuously monitors and pulls messages from the original data queue. The messages in the original data queue are evenly distributed to each consumer through RabbitMQ's load balancing mechanism. Each node extracts key fields from the delivered message to obtain multiple raw observation data. All the original observation data are input into the quality control algorithm for testing to obtain the quality control data corresponding to all the original observation data. Based on the quality control data corresponding to all the original observation data, quality control results are generated. The quality control results and the original data are encapsulated into a new data packet, and a corresponding quality control code is assigned to the new data packet.

[0006] Furthermore, based on the observed data, an asynchronous data transmission channel based on a message queue is constructed, including: Design a message queue topology; wherein the message queue topology includes: a switch, a queue, and a routing policy; Receive the observation data sent by the meteorological sensor suite; The observation data is encapsulated into a message, and the message is sent to the message middleware; The message is sent to the quality control processing module through the message middleware; Based on the switch, the queue, and the routing policy, differentiated processing of hourly and minute-by-minute data is achieved, and the data streams before and after quality control are isolated.

[0007] Furthermore, the switch includes a Topic-type switch, which comprises a raw data switch and a post-quality control data switch; based on the switch, the queue, and the routing strategy, differentiated processing of hourly and minute-level data is implemented, and the data streams before and after quality control are isolated, including: The raw observation data is received through the raw data exchange. Post-quality control enhanced data is published through the post-quality control data exchange. When the raw data exchange is bound to the first queue, the first queue receives and buffers the raw data of the current city tower observation hours. When the raw data exchange is bound to the second queue, the second queue receives and buffers the current city tower observation minute raw data; When the quality control post-data exchange is bound to the third queue, the third queue distributes the quality control post-hourly data; When the quality control post-data exchange is bound to the fourth queue, the fourth queue distributes the quality control post-minute data.

[0008] Furthermore, the raw observation data includes station number, timestamp, and data objects of various meteorological elements; all the raw observation data are input into a quality control algorithm for testing to obtain quality control data corresponding to all the raw observation data. The following process is repeated until each piece of raw observation data has been traversed: The original observation data is subjected to missing data checks, limit value checks, range value checks, internal consistency checks, and time consistency checks to obtain the quality control data. Among them, the missing measurement check is used to detect whether the original observation data is complete, the limit value check is used to detect whether the value of each meteorological element exceeds the predefined physical range, the range value check is used to compare the original observation data with the statistical range of historical data, the internal consistency check is used to check whether the logical relationship between different meteorological elements at the same time is reasonable, and the time consistency check is used to check whether the timestamp is reasonable.

[0009] Secondly, embodiments of the present invention provide a rapid quality control system for tower data based on message transmission, the system comprising: The data acquisition module is used to collect observational data through a meteorological sensor kit; A construction module is used to construct an asynchronous data transmission channel based on a message queue according to the observation data; The listening module is used in a message-driven architecture, where each node of the quality control processing module deployed in the server cluster acts as a consumer; it continuously listens through the quality control processing module and pulls messages from the raw data queue. The distribution module is used to distribute messages in the original data queue evenly to each consumer through RabbitMQ's load balancing mechanism. The extraction module is used by each node to extract key fields from the delivered message to obtain multiple raw observation data. The detection module is used to input all the original observation data into the quality control algorithm for detection, and obtain the quality control data corresponding to all the original observation data. The generation module is used to generate quality control results based on the quality control data corresponding to all the original observation data; The encapsulation module is used to encapsulate the quality control results and the original data into a new data packet, and assign a corresponding quality control code to the new data packet.

[0010] Furthermore, the building module is specifically used for: Design a message queue topology; wherein the message queue topology includes: a switch, a queue, and a routing policy; Receive the observation data sent by the meteorological sensor suite; The observation data is encapsulated into a message, and the message is sent to the message middleware; The message is sent to the quality control processing module through the message middleware; Based on the switch, the queue, and the routing policy, differentiated processing of hourly and minute-by-minute data is achieved, and the data streams before and after quality control are isolated.

[0011] Furthermore, the switch includes a Topic-type switch, which includes a raw data switch and a post-quality control data switch; the construction module is specifically used for: The raw observation data is received through the raw data exchange. Post-quality control enhanced data is published through the post-quality control data exchange. When the raw data exchange is bound to the first queue, the first queue receives and buffers the raw data of the current city tower observation hours. When the raw data exchange is bound to the second queue, the second queue receives and buffers the current city tower observation minute raw data; When the quality control post-data exchange is bound to the third queue, the third queue distributes the quality control post-hourly data; When the quality control post-data exchange is bound to the fourth queue, the fourth queue distributes the quality control post-minute data.

[0012] Furthermore, the raw observation data includes station number, timestamp, and data objects of various meteorological elements; the detection module includes repeatedly performing the following process until each piece of raw observation data has been traversed: The original observation data is subjected to missing data checks, limit value checks, range value checks, internal consistency checks, and time consistency checks to obtain the quality control data. Among them, the missing measurement check is used to detect whether the original observation data is complete, the limit value check is used to detect whether the value of each meteorological element exceeds the predefined physical range, the range value check is used to compare the original observation data with the statistical range of historical data, the internal consistency check is used to check whether the logical relationship between different meteorological elements at the same time is reasonable, and the time consistency check is used to check whether the timestamp is reasonable.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described above.

[0014] Fourthly, embodiments of the present invention provide a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the method described above.

[0015] This invention provides a method and system for rapid quality control of tower data based on message transmission, comprising: collecting observation data through a meteorological sensor suite; constructing an asynchronous data transmission channel based on a message queue according to the observation data; adopting a message-driven architecture, with each node of the quality control processing module deployed in the server cluster acting as a consumer; continuously monitoring and pulling messages from the raw data queue through the quality control processing module; evenly distributing messages in the raw data queue to each consumer through the load balancing mechanism of RabbitMQ; extracting key fields from the delivered messages to obtain multiple raw observation data; inputting all raw observation data into the quality control algorithm for detection to obtain quality control data corresponding to all raw observation data; generating quality control results based on the quality control data corresponding to all raw observation data; encapsulating the quality control results and raw data into a new data packet and assigning a corresponding quality control code to the new data packet; achieving dense spatial coverage using tower stations, constructing an efficient data flow channel through a message transmission mode, and then achieving real-time rapid quality control of data by a message-driven rapid quality control module, forming a closed-loop optimization of the entire "collection-transmission-quality control" chain.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a rapid quality control method for tower data based on message transmission provided in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the queue definition and binding relationship provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the encoding rules provided in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of a rapid quality control system for tower data based on message transmission, provided in Embodiment 2 of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] To address the current lack of horizontal resolution in urban ground meteorological observations, a "meteorology + tower" model was adopted, making full use of existing communication tower resources throughout Beijing to build 2,372 meteorological tower observation stations.

[0022] This application proposes a rapid quality control method and system for tower data based on communication tower resources, with message transmission as the hub, and supporting rapid quality control. It utilizes tower stations to achieve dense spatial coverage, constructs an efficient data flow channel through message transmission mode, and then achieves real-time rapid quality control of data through a message-driven rapid quality control module. The three form a closed-loop optimization of the entire "acquisition-transmission-quality control" chain.

[0023] To address the insufficient horizontal resolution of current urban ground meteorological observations, a "meteorology + tower" model is adopted, reusing existing communication base station tower resources deployed throughout the city. 2372 towers with reasonable locations and stable power supply and 4G / 5G network access were selected, and standardized miniature meteorological sensor kits (temperature, humidity, wind speed, and wind direction) were installed at a height of 10 meters above the ground.

[0024] To meet the needs of rapid quality control of real-time data from the tower, Message Queuing (RMQ) is used as the core component for message transmission, building a highly efficient and stable data transmission channel. A multi-level message queue structure is designed, and a producer-consumer model is implemented to ensure decoupling and smooth data transmission between the acquisition and processing ends. Simultaneously, message persistence and acknowledgment mechanisms are supported to prevent data loss and ensure the reliable transmission of critical data. Furthermore, different routing strategies are set according to data type and source to achieve flexible message distribution.

[0025] Based on an efficient message transmission mechanism, a fast quality control algorithm module was designed to meet the quality control requirements of tower data. The quality control algorithm includes functions such as missing test checks, limit value checks, range value checks, and internal consistency checks. The quality control results are output in a unified quality control code format and are made available to other business systems and users through the interface service provided by the Tianqing platform.

[0026] The overall technical architecture of this application includes three core modules: High-density tower observation and acquisition layer: deploying miniature meteorological sensors using communication towers to build a dense observation network; Message-driven data transmission layer: A data hub is built based on the message middleware (RabbitMQ) to achieve asynchronous, decoupled, and reliable data flow; Real-time quality control processing layer: As a message consumer, it performs multi-level quality control on each piece of observation data in real time and outputs structured quality control results. The three layers form a closed-loop link of "acquisition-transmission-quality control-service", supporting urban meteorological perception capabilities with minute-level updates and second-level responses.

[0027] To facilitate understanding of this embodiment, the embodiments of the present invention will be described in detail below.

[0028] Example 1: Figure 1 The flowchart is a rapid quality control method for tower data based on message transmission provided in Embodiment 1 of the present invention.

[0029] Reference Figure 1 The method includes the following steps: Step S101: Collect observation data using a meteorological sensor kit; Specifically, standardized meteorological observation stations are deployed based on communication towers: 2,372 communication towers with stable power supply and 4G / 5G network access are selected throughout the city. An integrated meteorological sensor kit containing four elements—wind speed, wind direction, temperature, and humidity—is installed on each tower at a height of 10 meters above the ground.

[0030] Step S102: Construct an asynchronous data transmission channel based on message queues according to the observation data; Specifically, to address the high latency and low throughput issues faced by traditional synchronous transmission modes in high-concurrency scenarios, and to effectively handle the system pressure of 2372 tower observation stations simultaneously uploading data, this application adopts RabbitMQ (RMQ) as the core message middleware to construct an efficient, stable, scalable, and persistently reliable asynchronous data transmission channel. This channel is designed for meteorological observation business scenarios, employing a multi-level, topic-based architecture. It achieves complete decoupling between the edge acquisition end and the quality control processing end through a producer-consumer model. Simultaneously, combined with a refined routing strategy and a dual reliability guarantee mechanism (message persistence and manual consumer confirmation), it ensures that tower observation data of different types and frequencies, such as minute-level and hourly-level data, can be transmitted to the corresponding quality control processing module with high concurrency, low latency, and high reliability, providing a solid data flow foundation for subsequent real-time quality control.

[0031] Step S103: Using a message-driven architecture, each node of the quality control processing module deployed in the server cluster is treated as a consumer. Step S104: The quality control processing module continuously monitors and pulls messages from the original data queue; Step S105: Distribute the messages in the original data queue evenly to each consumer using RabbitMQ's load balancing mechanism. Step S106: Each node extracts key fields from the delivered message to obtain multiple raw observation data. Step S107: Input all the original observation data into the quality control algorithm for detection to obtain the quality control data corresponding to all the original observation data; Step S108: Generate quality control results based on the quality control data corresponding to all the original observation data; Step S109: Encapsulate the quality control results and the original data into a new data packet, and assign the corresponding quality control code to the new data packet.

[0032] Furthermore, step S102 includes the following steps: Step S201: Design the message queue topology; wherein, the message queue topology includes: switches, queues, and routing policies; Step S202: Receive observation data sent by the meteorological sensor kit; Step S203: Encapsulate the observation data into a message and send the message to the message middleware; Step S204: Send the message to the quality control processing module through the message middleware; Step S205: Implement differentiated processing of hourly and minute data based on switch, queue and routing policies, and isolate the data flow before and after quality control.

[0033] Specifically, message queue topology design: This channel achieves complete decoupling between the data acquisition end and the processing end through carefully designed switches, queues and routing strategies, and ensures "zero loss" of critical meteorological data through a series of reliability mechanisms.

[0034] Using the classic producer-consumer model: Producers: Data acquisition units deployed at 2372 tower control stations. They are responsible for encapsulating local sensor observation data into messages and sending them to the message middleware. Message hub: RabbitMQ cluster. It is responsible for receiving, buffering, routing, and distributing all messages, serving as the data bus for the entire system. The message middleware RabbitMQ can be replaced with Kafka, RocketMQ, or Pulsar, etc.

[0035] Consumers include the rapid quality control processing module, the data service gateway, and other backend business systems. They acquire and process messages by subscribing to the corresponding queues.

[0036] This architecture allows producers (tower stations) to be unconcerned about who processes the data, and consumers (quality control modules) to be unconcerned about where the data comes from. The two communicate only through message queues, achieving loose coupling and high cohesion.

[0037] Furthermore, the switch includes a Topic-type switch, which includes a raw data switch and a post-quality control data switch; step S205 includes the following steps: Step S301: Receive raw observation data through the raw data exchange; Step S302: Publish the enhanced data after quality control through the post-quality control data exchange; Step S303: When the raw data exchange is bound to the first queue, the first queue receives and buffers the raw data of the current city tower observation hours. Step S304: When the raw data exchange is bound to the second queue, the second queue receives and buffers the raw data of the current city tower observation minutes; Step S305: When the data exchange after quality control is bound to the third queue, the third queue distributes the hourly data after quality control. Step S306: When the data exchange after quality control is bound to the fourth queue, the fourth queue distributes the minute data after quality control.

[0038] Specifically, the message channel design utilizes switches, queues, and routing strategies to differentiate between hourly and minute-level data and isolate data flows before and after quality control. The following message channel structure was designed: For switch and queue planning, we defined two types of switches based on Topic to distinguish different stages of the data flow: A Topic-type exchange is used as the message distribution hub, supporting a flexible subscription mechanism based on wildcard routing keys. The overall topology includes: X_OBS_REG_TT: for receiving raw observation data; X_OBS_REG_TT_PQC: for publishing post-quality control augmentation data. Each exchange is bound to multiple dedicated queues, isolated according to data time granularity (minutes / hours) and processing stage (raw / post-quality control) to avoid mutual interference between different types of data.

[0039] Raw data exchange: X_OBS_REG_TT receives all raw observation data from tower stations that are not under quality control. Bound queue: Queue Q.OBS_REG_TT.CMADAAS.A.3003.0001.R004: Used to receive and buffer the raw data of the current city tower observation hours. Queue Q.OBS_REG_TT.CMADAAS.A.3003.0001.R005: Used to receive and buffer the current minute-by-minute raw data of the city's tower observations.

[0040] Data switch after quality control: X_OBS_REG_TT_PQC receives all observation data with quality control codes after being processed by the fast quality control module. The bound queue Q.OBS_REG_TT.CMADAAS.A.3003.0001.R004_TO_PQC is used to distribute hourly data after quality control. Queue Q.OBS_REG_TT.CMADAAS.A.3003.0001.R005_TO_PQC: Used for distributing minute-by-minute data after quality control. Refer to [link / reference] for queue definition and binding relationships. Figure 2 .

[0041] Data reliability assurance mechanism: To cope with abnormal scenarios such as network jitter and service restarts, the system constructs a dual guarantee from two dimensions: message storage and consumption confirmation. Message Durability Queue persistence: All queues are declared with durable=true to ensure that queue metadata is not lost after the RMQ node restarts; Message persistence: When a producer sends a message, it sets the message attribute delivery_mode=2 (persistence mode) to write the message to disk instead of storing it only in memory.

[0042] Furthermore, the raw observation data includes station number, timestamp, and data objects for each meteorological element; step S107 includes the following steps, repeating the following processing until each piece of raw observation data has been traversed: The raw observation data is subjected to missing data checks, limit value checks, range value checks, internal consistency checks, and time consistency checks to obtain quality control data; Among them, the missing data check is used to detect whether the original observation data is complete, the limit value check is used to detect whether the values ​​of each meteorological element exceed the predefined physical range, the range value check is used to compare the original observation data with the statistical range of historical data, the internal consistency check is used to check whether the logical relationship between different meteorological elements at the same time is reasonable, and the time consistency check is used to check whether the timestamp is reasonable.

[0043] To achieve second-level response and minute-level availability of tower observation data, this application adopts a message-driven architecture. A fast quality control processing module deployed in a server cluster acts as the core consumer, continuously listening for and pulling messages from the raw data queue to achieve a real-time quality control mode of "processing data as soon as it arrives." This mode completely abandons the traditional batch processing method based on database polling, significantly reducing system latency and improving resource utilization efficiency.

[0044] The workflow is as follows: Consume messages > parse data > execute quality control algorithms > generate quality control results > publish new messages after quality control; The closed-loop process ensures that every piece of data is processed in real time, and the message confirmation mechanism guarantees the reliability of the processing.

[0045] Real-time data consumption and analysis Message Subscription: Each node in the quality control processing cluster acts as a consumer, subscribing to the raw data queues Q.OBS_REG_TT.CMADAAS.A.3003.0001.R004 (hourly data) and Q.OBS_REG_TT.CMADAAS.A.3003.0001.R005 (minutely data). RabbitMQ's load balancing mechanism distributes messages in the queues evenly to all consumers within the cluster.

[0046] Message parsing: When a message is delivered to a quality control node, the node first parses the message body. The message body is in standard JSON format, from which the node extracts key fields such as station_id (station number), timestamp (time stamp), and data objects containing various meteorological elements.

[0047] Quality control algorithm flow and specific implementation: For each piece of parsed observation data, the quality control algorithm executes a multi-dimensional set of algorithms in a preset order. These algorithms are progressively enhanced to ensure the comprehensiveness of data quality. In the quality control algorithm, the rule engine can be replaced with a lightweight machine learning model (such as Isolation Forest) for anomaly detection.

[0048] 1) Missing test check: Check the integrity of the data.

[0049] 2) Threshold check: Filters out physically impossible extreme error values. The values ​​of each meteorological element are compared to predefined physical possible ranges. These thresholds are universal and do not depend on any specific station.

[0050] For example: the temperature temp needs to be between [-80.0, 50.0] ℃; the wind speed wind_speed needs to be between [0.0, 75.0] m / s.

[0051] 3) Range Value Check: Identify outliers that exceed the historical statistical characteristics of the station. Compare observed values ​​with the historical statistical range of the station (such as monthly high / low values, standard deviation multiples). These thresholds can be updated periodically from the historical database to accommodate seasonal changes.

[0052] 4) Internal consistency check: Examine whether the logical relationship between different meteorological elements at the same time is reasonable.

[0053] For example, the dew point temperature cannot be higher than the air temperature recorded in the same document.

[0054] 5) Time consistency check: Check whether the timestamp of the data is reasonable to prevent interference from future or outdated data.

[0055] Implementation: Check if the difference between the timestamp in the message and the current server time is within an acceptable window (e.g., 5 minutes before or after). Data exceeding the window will be marked as an error.

[0056] Quality control result generation and packaging: After all quality control checks are completed, the system will generate a unified quality control result and encapsulate it together with the original data into a new data packet.

[0057] Quality control code generation: The system generates a standardized quality control code for each piece of data. The quality control code can be an integer or a bitmask, used to clearly identify the data status. The encoding rules for the quality control code refer to... Figure 3 .

[0058] By leveraging tower stations, the horizontal resolution of observations has been increased to 2.2 kilometers, and message-driven quality control has reduced processing time from hours to within 30 seconds. This marks the first time that the city has achieved "three highs" in meteorological observation capabilities: kilometer-level spatial coverage, minute-level data updates, and real-time quality control. In scenarios with concurrent uploads from 2,372 edge nodes, message queues effectively buffer peak traffic, and persistence mechanisms ensure data reliability, enhancing the overall availability of the system. Real-time quality control data has been successfully integrated into operational systems such as short-term forecasting, urban flooding warnings, and traffic meteorology, supporting 0-1 hour warning accuracy. At the same time, reusing tower resources saves construction costs, achieving a dual leap in technical performance and economic benefits.

[0059] Example 2: Figure 4 This is a schematic diagram of a rapid quality control system for tower data based on message transmission, provided in Embodiment 2 of the present invention.

[0060] Reference Figure 4 The system includes: The data acquisition module is used to collect observational data through a meteorological sensor kit; The building module is used to construct asynchronous data transmission channels based on message queues according to the observation data; The listening module is used in a message-driven architecture, where each node of the quality control processing module deployed in the server cluster acts as a consumer; it continuously listens through the quality control processing module and pulls messages from the raw data queue. The distribution module is used to distribute messages in the original data queue evenly to each consumer through RabbitMQ's load balancing mechanism. The extraction module is used by each node to extract key fields from the delivered messages to obtain multiple raw observation data. The detection module is used to input all the raw observation data into the quality control algorithm for detection, and obtain the quality control data corresponding to all the raw observation data. The generation module is used to generate quality control results based on the quality control data corresponding to all the original observation data; The encapsulation module is used to encapsulate the quality control results and raw data into new data packets and assign corresponding quality control codes to the new data packets.

[0061] Furthermore, the building blocks are specifically used for: Design a message queue topology; the message queue topology includes: switches, queues, and routing policies; Receive observation data sent by the meteorological sensor suite; The observation data is encapsulated into messages and sent to the message middleware; The message is sent to the quality control processing module through the message middleware; The system utilizes switches, queues, and routing strategies to implement differentiated processing of hourly and minute-by-minute data, and isolates the data flow before and after quality control.

[0062] Furthermore, the switches include Topic-type switches, which include raw data switches and post-quality control data switches; the building module is specifically used for: Receive raw observation data through the raw data exchange; Post-quality control enhanced data is published via the post-quality control data exchange; When the raw data exchange is bound to the first queue, the first queue receives and buffers the raw data of the current city tower observation hours; When the raw data exchange is bound to the second queue, the second queue receives and buffers the raw data of the current city tower observation minutes; When the data exchange after quality control is bound to the third queue, the third queue distributes the hourly data after quality control. When the data exchange after quality control is bound to the fourth queue, the fourth queue distributes the minute data after quality control.

[0063] Furthermore, the raw observation data includes station number, timestamp, and data objects for each meteorological element; the detection module includes repeatedly performing the following processing until each piece of raw observation data has been traversed: The raw observation data is subjected to missing data checks, limit value checks, range value checks, internal consistency checks, and time consistency checks to obtain quality control data; Among them, the missing data check is used to detect whether the original observation data is complete, the limit value check is used to detect whether the values ​​of each meteorological element exceed the predefined physical range, the range value check is used to compare the original observation data with the statistical range of historical data, the internal consistency check is used to check whether the logical relationship between different meteorological elements at the same time is reasonable, and the time consistency check is used to check whether the timestamp is reasonable.

[0064] A method for constructing a high-density meteorological observation network based on social infrastructure such as communication towers achieves 2.2-kilometer horizontal resolution coverage; the application of a data transmission architecture centered on message queues in large-scale meteorological observation systems is used to decouple data acquisition and processing and ensure reliable data transmission; a message-driven real-time quality control processing flow is implemented, in which the quality control module acts as a message consumer, performing immediate quality checks on each observation message from the tower station; and the quality control results are integrated into a multi-service system parallel distribution and API service output model through message routing.

[0065] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the message transmission-based rapid quality control method for tower data provided in the above embodiments.

[0066] This invention also provides a computer-readable medium having processor-executable non-volatile program code, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the message transmission-based rapid quality control method for tower data described above.

[0067] The computer program product provided in this embodiment of the invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0070] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0072] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A rapid quality control method for tower data based on message transmission, characterized in that, The method includes: Observational data is collected using a meteorological sensor kit; An asynchronous data transmission channel based on a message queue is constructed based on the observed data; A message-driven architecture is adopted, with each node of the quality control processing module deployed in the server cluster acting as a consumer; The quality control processing module continuously monitors and pulls messages from the original data queue. The messages in the original data queue are evenly distributed to each consumer through RabbitMQ's load balancing mechanism. Each node extracts key fields from the delivered message to obtain multiple raw observation data. All the original observation data are input into the quality control algorithm for testing to obtain the quality control data corresponding to all the original observation data. Based on the quality control data corresponding to all the original observation data, quality control results are generated. The quality control results and the original data are encapsulated into a new data packet, and a corresponding quality control code is assigned to the new data packet.

2. The rapid quality control method for tower data based on message transmission according to claim 1, characterized in that, Constructing an asynchronous data transmission channel based on a message queue using the observed data includes: Design a message queue topology; wherein the message queue topology includes: a switch, a queue, and a routing policy; Receive the observation data sent by the meteorological sensor suite; The observation data is encapsulated into a message, and the message is sent to the message middleware; The message is sent to the quality control processing module through the message middleware; Based on the switch, the queue, and the routing policy, differentiated processing of hourly and minute-by-minute data is achieved, and the data streams before and after quality control are isolated.

3. The rapid quality control method for tower data based on message transmission according to claim 2, characterized in that, The switches include Topic-type switches, which in turn include raw data switches and post-quality control data switches. Based on the switches, the queues, and the routing policies, differentiated processing of hourly and minute-level data is implemented, and the data streams before and after quality control are isolated, including: The raw observation data is received through the raw data exchange. Post-quality control enhanced data is published through the post-quality control data exchange. When the raw data exchange is bound to the first queue, the first queue receives and buffers the raw data of the current city tower observation hours. When the raw data exchange is bound to the second queue, the second queue receives and buffers the current city tower observation minute raw data; When the quality control post-data exchange is bound to the third queue, the third queue distributes the quality control post-hourly data; When the quality control post-data exchange is bound to the fourth queue, the fourth queue distributes the quality control post-minute data.

4. The rapid quality control method for tower data based on message transmission according to claim 1, characterized in that, The raw observation data includes station number, timestamp, and data objects of various meteorological elements. All raw observation data are input into a quality control algorithm for testing, resulting in quality control data corresponding to all raw observation data. The following process is repeated until every piece of raw observation data has been traversed: The original observation data is subjected to missing data checks, limit value checks, range value checks, internal consistency checks, and time consistency checks to obtain the quality control data. Among them, the missing data check is used to detect whether the original observation data is complete, the limit value check is used to detect whether the value of each meteorological element exceeds the predefined physical range, the range value check is used to compare the original observation data with the statistical range of historical data, the internal consistency check is used to check whether the logical relationship between different meteorological elements at the same time is reasonable, and the time consistency check is used to check whether the timestamp is reasonable.

5. A rapid quality control system for tower data based on message transmission, characterized in that, The system includes: The data acquisition module is used to collect observational data through a meteorological sensor kit; A construction module is used to construct an asynchronous data transmission channel based on a message queue according to the observation data; The listening module is used in a message-driven architecture, where each node of the quality control processing module deployed in the server cluster acts as a consumer; it continuously listens through the quality control processing module and pulls messages from the raw data queue. The distribution module is used to distribute messages in the original data queue evenly to each consumer through RabbitMQ's load balancing mechanism. The extraction module is used by each node to extract key fields from the delivered message to obtain multiple raw observation data. The detection module is used to input all the original observation data into the quality control algorithm for detection, and obtain the quality control data corresponding to all the original observation data. The generation module is used to generate quality control results based on the quality control data corresponding to all the original observation data; The encapsulation module is used to encapsulate the quality control results and the original data into a new data packet, and assign a corresponding quality control code to the new data packet.

6. The rapid quality control system for tower data based on message transmission according to claim 5, characterized in that, The building module is specifically used for: Design a message queue topology; wherein the message queue topology includes: a switch, a queue, and a routing policy; Receive the observation data sent by the meteorological sensor suite; The observation data is encapsulated into a message, and the message is sent to the message middleware; The message is sent to the quality control processing module through the message middleware; Based on the switch, the queue, and the routing policy, differentiated processing of hourly and minute-by-minute data is achieved, and the data streams before and after quality control are isolated.

7. The rapid quality control system for tower data based on message transmission according to claim 6, characterized in that, The switch includes Topic-type switches, which include raw data switches and post-quality control data switches; the construction module is specifically used for: The raw observation data is received through the raw data exchange. Post-quality control enhanced data is published through the post-quality control data exchange. When the raw data exchange is bound to the first queue, the first queue receives and buffers the raw data of the current city tower observation hours. When the raw data exchange is bound to the second queue, the second queue receives and buffers the current city tower observation minute raw data; When the quality control post-data exchange is bound to the third queue, the third queue distributes the quality control post-hourly data; When the quality control post-data exchange is bound to the fourth queue, the fourth queue distributes the quality control post-minute data.

8. The rapid quality control system for tower data based on message transmission according to claim 5, characterized in that, The raw observation data includes station number, timestamp, and data objects of various meteorological elements; the detection module includes repeatedly performing the following process until each piece of raw observation data has been traversed: The original observation data is subjected to missing data checks, limit value checks, range value checks, internal consistency checks, and time consistency checks to obtain the quality control data. Among them, the missing data check is used to detect whether the original observation data is complete, the limit value check is used to detect whether the value of each meteorological element exceeds the predefined physical range, the range value check is used to compare the original observation data with the statistical range of historical data, the internal consistency check is used to check whether the logical relationship between different meteorological elements at the same time is reasonable, and the time consistency check is used to check whether the timestamp is reasonable.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 4.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method described in any one of claims 1 to 4.

Images