A heterogeneous protocol meteorological observation data standardization encoding method

Abstract

The application discloses a heterogeneous protocol meteorological observation data standardization coding method, and relates to the technical field of meteorology. The method constructs a unified observation record object between a protocol analysis layer and a template coding layer, adopts the unified observation record object to bear data contents of original data messages of different data transmission protocols and supports coding processes of different standard templates, the unified observation record object reserves field semantics instead of only aligning general fields, guarantees complete observation semantics in a standard coding link, and makes the protocol analysis layer and the template coding layer decoupled as a stable intermediate bearing layer facing observation semantics, avoids point-to-point cross coupling between different data transmission protocols and different standard templates, and makes the protocol analysis layer and the template coding layer capable of being independently registered and extended, so that flexibility and expandability are improved.

Description

Technical Field

[0001] This application relates to the field of meteorological technology, and in particular to a standardized coding method for heterogeneous protocol meteorological observation data. Background Technology

[0002] With the rapid development of smart cities, precision agriculture, smart grids and environmental monitoring, the scale and complexity of meteorological data collection are growing exponentially. Meteorological data collection equipment used in meteorological observation scenarios comes from a wide range of sources, including various meteorological stations, domestic industrial-grade sensors and a large number of low-cost IoT terminals. Different manufacturers and even different models of meteorological data collection equipment from the same manufacturer use different data transmission protocols. Common data transmission protocol types include: (1) Fixed frame length binary protocol: uses bitmap index to identify valid fields, has a fixed frame structure, and high transmission efficiency. (2) Variable length character delimiter protocol: uses CSV or similar delimiter format, with fields separated by specific delimiters, which is highly readable. (3) Identifier-value-quality code protocol: uses a combination of encoding, value and quality control information to express observation elements. (4) Industry standard protocol: such as register mapping-based data acquisition protocol, which maps different observation elements through addresses.

[0003] Meteorological data acquisition equipment using different data transmission protocols outputs observational data in various formats, making direct fusion and analysis of data from different equipment impossible. Therefore, it is necessary to convert the observational data output by meteorological acquisition equipment into a specific standard encoding format to achieve data interconnection and cross-system sharing. Currently, commonly used standard encoding formats in the field of meteorological observation include BUFR (Binary Universal Data Representation Format) and NetCDF (Network Universal Data Format) promoted by the World Meteorological Organization (WMO). Other standard encoding formats include JSON Schema, etc.

[0004] Currently, format conversion of observation data output from meteorological data acquisition equipment typically employs a point-to-point conversion method. This involves developing dedicated conversion logic for each data transmission protocol and standard encoding format, and then using this dedicated logic to convert the format of the observation data output by meteorological data acquisition equipment using the corresponding data transmission protocol. However, when there are a large number of data transmission protocols and standard encoding formats, the cross-conversion relationships between them become complex, resulting in high development and maintenance costs. Furthermore, when expanding to new data transmission protocols or standard encoding formats, not only is it necessary to redevelop the dedicated conversion logic, but it is also necessary to recompile or modify the core logic. This tightly coupled architecture has poor flexibility and scalability. Summary of the Invention

[0005] To address the aforementioned problems and technical requirements, this application proposes a standardized coding method for heterogeneous protocol meteorological observation data. The technical solution of this application is as follows:

[0006] A method for standardizing and coding meteorological observation data using heterogeneous protocols, comprising: A unified observation record object is constructed and data is initialized between the protocol parsing layer and the template encoding layer. The unified observation record object includes a device identification segment, an observation timestamp segment, and multiple meteorological semantic segments, each of which corresponds to a meteorological observation element. Acquire at least one frame of raw data message sent by the meteorological acquisition equipment, and use the protocol parser that matches each frame of raw data message among the multiple protocol parsers deployed in the protocol parsing layer to parse the corresponding raw data message, and extract the equipment identifier of the meteorological acquisition equipment, the observation timestamp of the raw data message, and the observation data of multiple meteorological observation elements. Using the unified write interface between the protocol parsing layer and the unified observation record object, the parsed device identifier is written into the device identifier segment of the unified observation record object, the parsed observation timestamp is mapped to the unified time semantics and then written into the observation timestamp segment of the unified observation record object, and the observation data of each meteorological observation element is mapped to a predetermined semantic structure and then written into the corresponding meteorological semantic segment of the unified observation record object. The target template encoder in the template coding layer reads the data content in the unified observation record object through the unified reading interface between the template coding layer and the unified observation record object, fills it into the corresponding coding field of the target template used by the target template encoder, and outputs the coding result of at least one frame of raw data message that conforms to the target template.

[0007] The further technical solution is that each meteorological semantic segment includes a data element segment and carries a data existence identifier. The data existence identifier is explicitly stored or deduced according to the field setting rules. The data existence identifier is initialized to a miss state. Mapping the observation data of each meteorological observation element obtained from parsing to a predetermined semantic structure and writing it into the corresponding meteorological semantic segment of the unified observation record object includes: mapping the observation data of each meteorological observation element obtained from parsing to a predetermined semantic structure and writing it into the data element segment of the meteorological semantic segment corresponding to the current meteorological observation element, and modifying the data existence identifier of the meteorological semantic segment corresponding to the current meteorological observation element to the hit status. When reading the content of the unified observation record object through the unified reading interface between the template coding layer and the unified observation record object using the target template encoder in the template coding layer, the data content in the device identification segment, observation timestamp segment, and various meteorological semantic segments with data existence indicators as hits is read.

[0008] The further technical solution is that the data element segment in each meteorological semantic segment includes a numerical attribute field and its associated quality control attribute field and status attribute field. The data element segment of the meteorological semantic segment corresponding to the current meteorological observation element, after mapping the parsed observation data of each meteorological observation element to a predetermined semantic structure, includes: Map the observed values ​​of the current meteorological observation elements to the standard coded dimensions to obtain normalized observed values, and write the normalized observed values ​​and numerical mapping rules into the numerical attribute fields of the meteorological semantic segment corresponding to the current meteorological observation elements. After semantically mapping the quality control information of the current meteorological observation element's observation values ​​to a standard quality control code, write it into the quality control attribute field of the meteorological semantic segment corresponding to the current meteorological observation element. After mapping the data status semantics of the current meteorological observation element to standard status flags, write them into the status attribute field of the meteorological semantic segment corresponding to the current meteorological observation element. The standard status flags include valid status, invalid status, missing data, and status to be ignored.

[0009] A further technical solution involves using the target template encoder in the template coding layer to read the data content from the unified observation record object via the unified reading interface between the template coding layer and the unified observation record object, including: From the unified observation record objects, select the equipment identification segment, observation timestamp segment, and meteorological semantic segments with the data existence indicator set to "hit state". Expand and read the data content of each selected meteorological semantic segment. Fill the data content of the selected equipment identification segment, observation timestamp segment, and meteorological semantic segment into the corresponding coding fields according to the order of each coding field in the target template. For the data content of meteorological semantic segments with the standard status flag set to "invalid state", "missing data", and "to be ignored state", handle them according to the missing value rule.

[0010] The further technical solution is that the data element segment in the meteorological semantic segment corresponding to the variable-length structure meteorological observation element also includes an array counting attribute field, and the observation data of the variable-length structure meteorological observation element includes multiple observation values ​​under the corresponding observation dimension. When mapping the observation data of the variable-length structure meteorological observation elements obtained by parsing to a predetermined semantic structure and writing it into the data element segment of the meteorological semantic segment corresponding to the current variable-length structure meteorological observation element, multiple observation values ​​in the observation data are written into the numerical attribute field in the form of an array, and the array length is written into the array count attribute field. When reading the data content in the meteorological semantic segment corresponding to the variable-length structure meteorological observation element using the target template encoder in the template coding layer, the array length in the array count attribute field is read, and the corresponding number of observation values ​​are read according to the array length.

[0011] A further technical solution is that the data element segment in the meteorological semantic segment corresponding to the variable-length structure meteorological observation element also includes an auxiliary information field. When writing the parsed observation data of the variable-length structure meteorological observation element into the data element segment, the dimension coordinates corresponding to the array index of each observation value and the auxiliary information in the observation data are written into the auxiliary information field.

[0012] The further technical solution is that the meteorological observation elements corresponding to the meteorological semantic segments in the unified observation and recording objects include at least one of air pressure, temperature and humidity, wind, precipitation, ground temperature, grass temperature, visibility, and snow cover; among them, precipitation and ground temperature are meteorological observation elements with variable length structures. The precipitation observation data includes minute precipitation at multiple different collection times in the time dimension, and the dimension coordinates corresponding to the array index of each observation value are the collection times. The auxiliary information extracted from the precipitation observation data includes at least one of the following: precipitation measurement method, extreme values ​​of minute precipitation at different collection times and their corresponding collection times, and cumulative precipitation. The geothermal observation data includes geothermal values ​​at multiple different depths in a spatial dimension, with the array index of each observation value corresponding to the depth of the layer. The auxiliary information extracted from the geothermal observation data includes the extreme values ​​of temperature at different depths and their corresponding depths.

[0013] A further technical solution involves determining which protocol parsers among the multiple protocol parsers deployed in the protocol parsing layer match the original data packet for each frame, including: The original data message is scanned for message structure and features are extracted. The extracted message structure features include at least one of the following: message preamble, message end marker, number of fields, delimiter form, time field format, bitmap features, and keyword position. Iterate through all registered and deployed protocol parsers in the protocol parsing layer in turn. Different protocol parsers correspond to different data transmission protocols and have their own message structure rules. A candidate parser set is formed based on the message structure rules of multiple protocol parsers and sorted according to the matching score with the message structure features. The protocol parser with the highest matching degree with the original data message is determined through parsing verification.

[0014] Further technical solutions include: the standardized coding method for heterogeneous protocol meteorological observation data also includes: A new protocol parser is added to the protocol parsing layer. The new protocol parser maps the parsing results to the unified observation record object through a unified write interface. A new template encoder is added to the template encoding layer. The new template encoder reads the data content from the unified observation record object and performs template encoding through a unified read interface.

[0015] Further technical solutions include: the standardized coding method for heterogeneous protocol meteorological observation data also includes: When expanding access to meteorological data acquisition equipment that adopts a new data transmission protocol, a protocol parser corresponding to the new data transmission protocol is registered and deployed in the protocol parsing layer; when expanding support for new standard templates, a template encoder corresponding to the new standard template is registered and deployed in the template encoding layer.

[0016] The beneficial technical effects of this application are: This application discloses a standardized coding method for meteorological observation data using heterogeneous protocols. This method adds a unified observation record object between the protocol parsing layer and the template coding layer. The unified observation record object carries the data content of the original data packets of different data transmission protocols and supports the coding process of different standard templates. The unified observation record object retains domain semantics rather than just performing general field alignment, ensuring the integrity of observation semantics in the standard coding link. As a stable intermediate carrier layer oriented towards observation semantics, the unified observation record object decouples the protocol parsing layer and the template coding layer, avoiding point-to-point cross-coupling between different data transmission protocols and different standard templates. This allows both the protocol parsing layer and the template coding layer to be independently extended through registration, improving flexibility and scalability.

[0017] The unified observation record object records the hit status of the meteorological observation elements corresponding to its meteorological semantic segment through the data existence identifier. This allows the template encoder to skip the unhit meteorological semantic segments directly based on the data existence identifier and selectively traverse the hit meteorological semantic segments during traversal encoding. This reduces the overhead of invalid processing, reduces the amount of computation and improves efficiency, making it suitable for real-time encoding scenarios.

[0018] This method completes operations such as unit normalization, time unification, missing measurement mapping, and QC inheritance during the parsing phase, ensuring the semantic integrity of observations in the standard coding link. It can also effectively handle variable structures such as minute-by-minute rainfall and multi-layer ground temperature, and selectively encode the corresponding fields, making it suitable for embedded resource-constrained scenarios.

[0019] The unified observation and recording object does not rely on databases, middleware, or other heavy components, making the system deployment difficult and the operation less complex. It is suitable for resource-constrained scenarios. Furthermore, the method adopts a real-time processing link for the creation, initialization, data writing, and data encoding of the unified observation and recording object for single-frame raw data packets. It is suitable for real-time encoding on the terminal side as well as lightweight standardized output scenarios such as edge gateways. Attached Figure Description

[0020] Figure 1 This is an application architecture diagram of one embodiment of this application.

[0021] Figure 2 This is a flowchart of a method according to an embodiment of this application. Detailed Implementation

[0022] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0023] This application discloses a standardized coding method for heterogeneous protocol meteorological observation data, the application architecture of which is as follows: Figure 1 As shown, a layered and scalable architecture is adopted, including a protocol parsing layer, a template encoding layer, and a unified observation and recording object built between the two layers: The protocol parsing layer interfaces with multiple meteorological data acquisition devices and acquires the raw data packets collected by these devices. Since at least two meteorological data acquisition devices use different data transmission protocols, the protocol parsing layer receives raw data packets with multiple different data transmission protocols. The protocol parsing layer deploys multiple protocol parsers, each supporting different data transmission protocols. Each protocol parser is used to parse packets corresponding to its specific data transmission protocol. In one embodiment, the protocol parsers deployed support data transmission protocols including fixed-frame-length binary protocols, variable-length character-separated protocols, identifier-value-quality code protocols, industry standard protocols, and custom protocols, such as... Figure 1 The parentheses in the different protocol parsers indicate the data transmission protocols they support.

[0024] The template encoding layer deploys several template encoders, each encoding data based on a different standard template. The standard template referred to in this application is a data template using a standard data format. In one embodiment, the standard templates used by the template encoders deployed in the template encoding layer include at least one of the following: BUFR format standard template, NetCDF format standard template, JSONSchema format standard template, and custom standard template.

[0025] A unified observation and recording object is positioned between the protocol parsing layer and the template encoding layer. This unified observation and recording object is implemented using methods such as memory objects, structured blackboards, and structured buffers. The protocol parsing layer and the unified observation and recording object are connected via a unified write interface. Different protocol parsers deployed in the protocol parser layer all write data to the unified observation and recording object through this unified write interface. The template encoding layer and the unified observation and recording object are connected via a unified read interface. Different template encoders deployed in the template encoding layer read data from the unified observation and recording object through this unified read interface.

[0026] based on Figure 1 The application architecture and the standardized coding method for meteorological observation data of this heterogeneous protocol include the following steps. Please refer to [reference needed]. Figure 2 The flowchart shown: Step 210: Construct a unified observation record object between the protocol parsing layer and the template encoding layer and initialize the data. This unified observation record object is used to carry various types of data in the original data message according to the meteorological observation semantics.

[0027] like Figure 1 As shown, the created unified observation record object includes a device identifier segment, an observation timestamp segment, and multiple meteorological semantic segments. Each meteorological semantic segment corresponds to a meteorological observation element and is used to carry the observation data of the corresponding meteorological observation element. The meteorological observation elements corresponding to the meteorological semantic segments in the unified observation record object include at least one of the following: air pressure, temperature and humidity, wind, precipitation, ground temperature, grass temperature, visibility, and snow cover. Figure 1 Taking an example that includes all the meteorological observation elements mentioned above.

[0028] Different meteorological data acquisition devices have their own acquisition content, and each frame of raw data sent by the meteorological data acquisition device may only contain observation data of some meteorological observation elements. Therefore, some semantic meteorological segments in the unified observation record object may not have corresponding valid data. In order to quickly indicate which semantic meteorological segments in the unified observation record object have valid data, in one embodiment, each meteorological semantic segment includes a data element segment and carries a data existence identifier. The data element segment is used to carry the observation data of the corresponding meteorological observation element, and the data existence identifier is used to indicate whether the meteorological semantic segment contains valid observation data. The data existence identifier is explicitly stored or derived according to field setting rules, such as... Figure 1 Taking explicit storage of data existence identifiers as an example, the data existence identifier is either in a "missed" or "hit" state. A "missed" state indicates that the original data message does not contain the meteorological observation element corresponding to the meteorological semantic segment, while a "hit" state indicates that the original data message contains the meteorological observation element corresponding to the meteorological semantic segment. By detecting the data existence identifier, it is possible to quickly determine whether the data element segment of its meteorological semantic segment contains observation data, without further detailed reading and judgment of the data element segment. The data element segment of each meteorological semantic segment is initialized to empty, and the data existence identifier is initialized to a "missed" state.

[0029] In addition to observed values, the observation data for each meteorological observation element also includes other data, such as quality control information (e.g., QC codes) and data status. Therefore, the data element segment in each meteorological semantic segment includes a numerical attribute field and its associated quality control attribute field and status attribute field. The numerical attribute field carries the observed values ​​of the corresponding meteorological observation element, the quality control attribute field carries the quality control information of the observed values ​​of the corresponding meteorological observation element, and the status attribute field carries the data status of the observed values ​​of the corresponding meteorological observation element.

[0030] The forms of observed values ​​differ across meteorological observation elements. Broadly speaking, meteorological observation elements can be categorized into two types: fixed-length meteorological observation elements and variable-length structure meteorological observation elements. Fixed-length meteorological observation elements have a fixed number of observed values, while variable-length structure meteorological observation elements include multiple observed values ​​at different coordinates within the corresponding observation dimension, and the number of observed values ​​is variable. Among the meteorological observation elements listed above, precipitation and ground temperature belong to variable-length structure meteorological observation elements, while the other meteorological observation elements belong to fixed-length meteorological observation elements. Precipitation observation data includes minute-level precipitation at multiple different collection times in the time dimension. The coordinate of each minute-level precipitation is its collection time, and the number of minute-level precipitation values ​​in the precipitation observation data is variable. Ground temperature observation data includes ground temperature values ​​at multiple different depths in the spatial dimension. The coordinate of each ground temperature value is its depth, and the number of ground temperature values ​​in the ground temperature observation data is variable.

[0031] To adapt to the data characteristics of meteorological observation elements with variable length structures, the data element segment in the meteorological semantic segment corresponding to the meteorological observation element with variable length structure also includes an array count attribute field. The array count attribute field is used to record the number of observation values ​​under the corresponding observation dimension included in the observation data of the meteorological observation element with variable length structure.

[0032] In addition, the data element segment in the meteorological semantic segment corresponding to the variable-length structure meteorological observation element also includes an auxiliary information field. The auxiliary information field is used to record the dimensional coordinate information of different observation values ​​in the observation data of the variable-length structure meteorological observation element, as well as other auxiliary information.

[0033] The unified observation record object created in this step is designed for processing single-frame raw data packets. All unified observation record objects reuse the same semantic skeleton, rather than being arbitrarily assembled at runtime. In addition, the unified observation record object is created and initialized before processing each frame of raw data packet to avoid historical residual data affecting the processing of the current frame.

[0034] Step 220: Obtain at least one frame of raw data message sent by the meteorological acquisition equipment. Use the protocol parser that matches each frame of raw data message among the multiple protocol parsers deployed in the protocol parsing layer to parse the corresponding raw data message and extract the equipment identifier of the meteorological station to which the meteorological acquisition equipment belongs, the observation timestamp of the raw data message, and the observation data of multiple meteorological observation elements.

[0035] Different protocol parsers deployed in the protocol parsing layer support different data transmission protocols. To ensure correct parsing of the original data packets, a protocol parser matching the data transmission protocol of the original data packets needs to be used. One approach is to predetermine the data transmission protocol used by the meteorological data acquisition devices connected to each port number and configure the corresponding protocol parser for each port number, then use the protocol parser corresponding to the port number of the original data packet for parsing. However, in real-world applications, the number of connected meteorological data acquisition devices is large and the application environment is complex, making it difficult to fix the ports and pre-configure them. Therefore, in another embodiment, an automatic protocol identification and adaptation mechanism is adopted, including: The original data packet undergoes a message structure scan and feature extraction. Extracted message structure features include at least one of the following: message preamble, message end marker, number of fields, delimiter format, time field format, bitmap features, and keyword position. Then, all registered and deployed protocol parsers in the protocol parsing layer are sequentially traversed. Different protocol parsers correspond to different data transmission protocols and have their own message structure rules. The message structure rules for each protocol parser are fixed and can be predetermined. A candidate parser set is formed based on the message structure rules of multiple protocol parsers and sorted according to their matching score with the message structure features. The higher the consistency between the original data packet's message structure features and the protocol parser's message structure rules, the higher the matching degree. Finally, the protocol parser with the highest matching degree to the original data packet, as determined through parsing verification, is selected to parse the original data packet.

[0036] When evaluating the consistency between the message structure characteristics of the original data packet and the message structure rules of the protocol parser, consistency scoring rules for each message structure characteristic can be pre-set and the overall consistency can be determined by weighted summation. When the overall consistency scores of multiple protocol parsers and the original data packet are close, further judgment can be made by combining the station type, message length, or service configuration.

[0037] Then, using the protocol parser that matches the original data message, the original data message is parsed according to the message structure rules adopted by the protocol parser to remove the message header and footer and ignore the check field, and extract the device identifier, observation timestamp and observation data of multiple meteorological observation elements.

[0038] Step 230: Using the unified write interface between the protocol parsing layer and the unified observation record object, the parsed data of the original data packet is written into the unified observation record object, including: The parsed device identifier is written into the device identifier segment of the unified observation record object. The device identifier here includes the identifier of the meteorological acquisition equipment itself and / or the station identifier of the meteorological station where the meteorological acquisition equipment is located.

[0039] After the parsed observation timestamps are mapped to the unified time semantics, they are written into the observation timestamp segment of the unified observation record object.

[0040] After mapping the observation data of each meteorological observation element obtained from the parsing to a predetermined semantic structure, the data is written into the corresponding meteorological semantic segment of the unified observation record object, including: After mapping the observation data of each meteorological observation element obtained from the parsing to a predetermined semantic structure, the data is written into the data element segment of the meteorological semantic segment corresponding to the current meteorological observation element. The data existence flag of the meteorological semantic segment corresponding to the current meteorological observation element is then modified to a "hit" state, while the data existence flag of the meteorological semantic segments corresponding to meteorological observation elements that did not hit remains in a "missed" state. Specifically, when writing the data element segment: (1) Map the observed values ​​of the current meteorological observation elements to the standard coding units to obtain normalized observed values, and write the normalized observed values ​​and numerical mapping rules into the numerical attribute fields of the meteorological semantic segments corresponding to the current meteorological observation elements. Among them, the numerical mapping rules are the conversion rules used when mapping the measurement units, multipliers or integer precision of the original observed values ​​to the standard coding units.

[0041] For multiple observation values ​​in a meteorological observation element with a variable length structure, the multiple observation values ​​in the observation data will be written into the numerical attribute field in the form of an array.

[0042] (2) After semantically mapping the quality control information of the current meteorological observation element to the standard quality control code, write it into the quality control attribute field of the meteorological semantic segment corresponding to the current meteorological observation element. The extracted quality control information includes QC code, abnormality identifier or verification status.

[0043] (3) After mapping the data status semantics of the current meteorological observation element to the standard status flag bit, write it into the status attribute field of the meteorological semantic segment corresponding to the current meteorological observation element. The standard status flag bit includes valid status, invalid status, missing data, and status to be ignored.

[0044] (4) For meteorological observation elements with variable length structure, after writing multiple observation values ​​in the observation data of the meteorological observation elements with variable length structure into the numerical attribute field in the form of an array, the array length, that is, the number of observation values, will also be written into the array count attribute field.

[0045] In addition, the dimension coordinates corresponding to the array index of each observation value and auxiliary information from the observation data will be written into the auxiliary information field. For precipitation and ground temperature, two meteorological observation elements with variable elongation structures: The dimensional coordinates corresponding to the array indices of each minute of precipitation in the precipitation observation data are the acquisition time. This means the correlation between the array indices of each minute of precipitation and the acquisition time is written into the auxiliary information field. Additionally, auxiliary information from the precipitation observation data is extracted, including at least one of the following: precipitation measurement method, extreme values ​​of minute precipitation at different acquisition times and their corresponding acquisition times, and cumulative precipitation.

[0046] In the geothermal observation data, each observation value, i.e., the array index of each geothermal value, corresponds to a layer depth in dimensional coordinates. This means the relationship between the array index of each geothermal value and the corresponding layer depth is written into the auxiliary information field. The auxiliary information extracted from the geothermal observation data includes the extreme values ​​of temperature at different layer depths and their corresponding layer depths.

[0047] Step 240: The target template encoder in the template coding layer reads the data content from the unified observation record object through the unified reading interface between the template coding layer and the unified observation record object, fills it into the corresponding coding field of the target template used by the target template encoder, and outputs the coding result that conforms to the target template. In practical applications, step 220 above usually acquires multiple frames of raw data packets from a set of raw data packets sent by multiple meteorological acquisition devices. In this case, the coding result output here is the merged coding result of the batch-acquired raw data packets.

[0048] As mentioned above, some semantic meteorological segments in the unified observation record object may not contain valid data, and each semantic meteorological segment contains a data existence identifier. Therefore, when reading the data content in the unified observation record object, it is not necessary to traverse each meteorological semantic segment sequentially. Instead, it is only necessary to take the data content in each meteorological semantic segment where the device identifier segment, observation timestamp segment, and data existence identifier are in a hit state. That is, it is necessary to directly skip the meteorological semantic segments where the level flag attribute segment is in a hit state, reduce invalid data traversal, and thus improve data reading efficiency. This includes: selecting each meteorological semantic segment where the device identifier segment, observation timestamp segment, and data existence identifier are in a hit state from the unified observation record object, and expanding and reading the data content of each selected meteorological semantic segment. Then, following the order of the encoded fields in the target template, the data content of the selected device identifier segment, observation timestamp segment, and meteorological semantic segment are filled into the corresponding encoded fields. The data content of the meteorological semantic segment includes the correlation between the observed value in the numerical attribute field, the standard quality control code in the quality control attribute field, and the standard status flag bit in the status attribute field, thereby maintaining the consistent transmission of the correlation between observed values, quality control information, and data status. For the data content of the meteorological semantic segment with an invalid standard status flag bit, missing data, or a state to be ignored, it is handled according to the missing value rule. The processing order of each encoded field in the target template can be customized. For example, for the BUFR format standard template, each encoded field is processed sequentially according to the descriptor order.

[0049] For meteorological observation elements with variable-length structures, since the number of observations in these elements is not fixed, when reading the data content of the meteorological semantic segment corresponding to the variable-length structure meteorological observation element using the target template encoder in the template coding layer, the array length in the array count attribute field is read first, and then the corresponding number of observations is read according to the array length. For the BUFR format standard template, when reading the corresponding number of observations according to the array length, that is, setting the short repetition factor or delayed repetition factor according to the array length, the dimension coordinates corresponding to the array index of each observation value and the auxiliary information in the observation data are further read from the auxiliary information field to determine the dimension coordinates and other auxiliary information of each observation value.

[0050] The specific target template used is determined based on the specific business scenario. Since different template encoders in the template encoding layer can access the unified observation record object through a unified reading interface, in practical applications, at least two different target template encoders in the template encoding layer can read the content of each semantic segment in the same unified observation record object through the unified reading interface between the template encoding layer and the unified observation record object, and fill them into the corresponding encoding fields of the target templates used by the target template encoders respectively. This outputs multiple encoding results of at least one frame of original data packets that conform to different target templates, that is, multiplexing the output of multiple standard formats through a unified observation record object.

[0051] Because the protocol parsing layer and template encoding layer are decoupled through a unified observation and recording object, modifications to either layer do not require changes to the entire data acquisition chain. When expanding to include meteorological data acquisition equipment using a new data transmission protocol, it is only necessary to register and deploy the protocol parser corresponding to the new data transmission protocol in the protocol parsing layer. The newly added parser also maps the parsing results to the unified observation and recording object through a unified write interface. Similarly, when expanding to support new standard templates, it is only necessary to register and deploy the template encoder corresponding to the new standard template in the template encoding layer. The newly added template encoder also reads data from the unified observation and recording object and performs template encoding through a unified read interface. Both the protocol parser and template encoder can be extended independently without requiring a full cross-modification, offering strong flexibility and scalability.

[0052] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A heterogeneous protocol meteorological observation data standardization encoding method, characterized in that, The standardized coding method for heterogeneous protocol meteorological observation data includes: A unified observation record object is constructed and data is initialized between the protocol parsing layer and the template encoding layer. The unified observation record object includes a device identification segment, an observation timestamp segment, and multiple meteorological semantic segments, each of which corresponds to a meteorological observation element. Acquire at least one frame of raw data message sent by the meteorological acquisition equipment, and use the protocol parser that matches each frame of raw data message among the multiple protocol parsers deployed in the protocol parsing layer to parse the corresponding raw data message, and extract the equipment identifier of the meteorological acquisition equipment, the observation timestamp of the raw data message, and the observation data of multiple meteorological observation elements. Using the unified write interface between the protocol parsing layer and the unified observation record object, the parsed device identifier is written into the device identifier segment of the unified observation record object, the parsed observation timestamp is mapped to the unified time semantics and then written into the observation timestamp segment of the unified observation record object, and the observation data of each meteorological observation element is mapped to a predetermined semantic structure and then written into the corresponding meteorological semantic segment of the unified observation record object. The target template encoder in the template coding layer reads the data content in the unified observation record object through the unified reading interface between the template coding layer and the unified observation record object, fills it into the corresponding coding field of the target template used by the target template encoder, and outputs the coding result that conforms to the target template.

2. The heterogeneous protocol weather observation data standardization encoding method of claim 1, wherein, Each meteorological semantic segment includes a data element segment and carries a data existence identifier, which is initialized to a "missed" state. Mapping the observation data of each meteorological observation element obtained from parsing to a predetermined semantic structure and writing it into the corresponding meteorological semantic segment of the unified observation record object includes: mapping the observation data of each meteorological observation element obtained from parsing to a predetermined semantic structure and writing it into the data element segment of the meteorological semantic segment corresponding to the current meteorological observation element, and modifying the data existence identifier of the meteorological semantic segment corresponding to the current meteorological observation element to the hit status. When reading the content of the unified observation record object through the unified reading interface between the template coding layer and the unified observation record object using the target template encoder in the template coding layer, the data content in the device identification segment, observation timestamp segment, and various meteorological semantic segments with data existence indicators as hits is read.

3. The heterogeneous protocol meteorological observation data standardization coding method according to claim 2, characterized in that, Each meteorological semantic segment's data element segment includes numerical attribute fields and their associated quality control attribute fields and status attribute fields. The data element segment written into the meteorological semantic segment corresponding to the current meteorological observation element after mapping the parsed observation data of each meteorological observation element to a predetermined semantic structure includes: Map the observed values ​​of the current meteorological observation elements to the standard coded dimensions to obtain normalized observed values, and write the normalized observed values ​​and numerical mapping rules into the numerical attribute fields of the meteorological semantic segment corresponding to the current meteorological observation elements. After semantically mapping the quality control information of the current meteorological observation element's observation values ​​to a standard quality control code, write it into the quality control attribute field of the meteorological semantic segment corresponding to the current meteorological observation element. After mapping the data status semantics of the current meteorological observation element to standard status flags, write them into the status attribute field of the meteorological semantic segment corresponding to the current meteorological observation element. The standard status flags include valid status, invalid status, missing data, and status to be ignored.

4. The heterogeneous protocol meteorological observation data standardization coding method according to claim 3, characterized in that, The data content in the unified observation record object is read using the target template encoder in the template coding layer via the unified read interface between the template coding layer and the unified observation record object, including: From the unified observation record objects, select the equipment identification segment, observation timestamp segment, and meteorological semantic segments with the data existence indicator set to "hit state". Expand and read the data content of each selected meteorological semantic segment. Fill the data content of the selected equipment identification segment, observation timestamp segment, and meteorological semantic segment into the corresponding coding fields according to the order of each coding field in the target template. For the data content of meteorological semantic segments with the standard status flag set to "invalid state", "missing data", and "to be ignored state", handle them according to the missing value rule.

5. The heterogeneous protocol meteorological observation data standardization coding method according to claim 3, characterized in that, The data element segment in the meteorological semantic segment corresponding to the variable-length structure meteorological observation element also includes the array counting attribute field. The observation data of the variable-length structure meteorological observation element includes multiple observation values ​​under the corresponding observation dimension. When mapping the observation data of the variable-length structure meteorological observation elements obtained by parsing to a predetermined semantic structure and writing it into the data element segment of the meteorological semantic segment corresponding to the current variable-length structure meteorological observation element, multiple observation values ​​in the observation data are written into the numerical attribute field in the form of an array, and the array length is written into the array count attribute field. When reading the data content in the meteorological semantic segment corresponding to the variable-length structure meteorological observation element using the target template encoder in the template coding layer, the array length in the array count attribute field is read, and the corresponding number of observation values ​​are read according to the array length.

6. The heterogeneous protocol meteorological observation data standardization coding method according to claim 5, characterized in that, The data element segment in the meteorological semantic segment corresponding to the variable-length structure meteorological observation element also includes an auxiliary information field. When writing the parsed observation data of the variable-length structure meteorological observation element into the data element segment, the dimension coordinates corresponding to the array index of each observation value and the auxiliary information in the observation data are written into the auxiliary information field.

7. The heterogeneous protocol meteorological observation data standardization coding method according to claim 6, characterized in that, The meteorological observation elements corresponding to the meteorological semantic segments in the unified observation and recording objects include at least one of the following: air pressure, temperature and humidity, wind, precipitation, ground temperature, grass temperature, visibility, and snow cover; among them, precipitation and ground temperature are meteorological observation elements with variable length structures. The precipitation observation data includes minute precipitation at multiple different collection times in the time dimension, and the dimension coordinates corresponding to the array index of each observation value are the collection times. The auxiliary information extracted from the precipitation observation data includes at least one of the following: precipitation measurement method, extreme values ​​of minute precipitation at different collection times and their corresponding collection times, and cumulative precipitation. The geothermal observation data includes geothermal values ​​at multiple different depths in a spatial dimension, with the array index of each observation value corresponding to the depth of the layer. The auxiliary information extracted from the geothermal observation data includes the extreme values ​​of temperature at different depths and their corresponding depths.

8. The heterogeneous protocol meteorological observation data standardization coding method according to claim 1, characterized in that, The protocol parsers that match the original data packets in each frame among the multiple protocol parsers deployed at the protocol parsing layer include: The original data message is scanned for message structure and features are extracted. The extracted message structure features include at least one of the following: message preamble, message end marker, number of fields, delimiter form, time field format, bitmap features, and keyword position. Iterate through all registered and deployed protocol parsers in the protocol parsing layer in turn. Different protocol parsers correspond to different data transmission protocols and have their own message structure rules. A candidate parser set is formed based on the message structure rules of multiple protocol parsers and sorted according to the matching score with the message structure features. The protocol parser with the highest matching degree with the original data message is determined through parsing verification.

9. The heterogeneous protocol meteorological observation data standardization coding method according to claim 1, characterized in that, The heterogeneous protocol meteorological observation data standardization coding method also includes: Using at least two different target template encoders in the template coding layer, the content of each semantic segment in the same unified observation record object is read through the unified reading interface between the template coding layer and the unified observation record object. The content is then filled into the corresponding coding field of the target template used by each target template encoder, and the output of at least one frame of raw data message with multiple coding results conforming to different target templates is generated.

10. The heterogeneous protocol meteorological observation data standardization coding method according to claim 1, characterized in that, The heterogeneous protocol meteorological observation data standardization coding method also includes: A new protocol parser is added to the protocol parsing layer. The new protocol parser maps the parsing results to the unified observation record object through a unified write interface. A new template encoder is added to the template encoding layer. The new template encoder reads the data content from the unified observation record object and performs template encoding through a unified read interface.

Images