GTS data exception analysis method and system

Abstract

The invention discloses a GTS data exception analysis method and system, and the method comprises the steps: receiving a real-time GTS message, cutting the GTS message according to a starting identifier and an ending identifier of an announcement to obtain a complete message, and carrying out the reading and analysis of similar data files; reading the complete announcement in a whole code cutting mode, dividing the complete announcement into four layers, and decoding the four layers to obtain a layer code; performing abnormal condition judgment on the first-level code, performing abnormal condition judgment and analysis on the second-level code, performing abnormal condition judgment on the third-level code and the fourth-level code, and processing the third-level code and the fourth-level code; and integrating the end of the year in the message with the file generation time, obtaining reasonable time through comparison and adjustment, decoding the fourth-level code, and recording abnormal details to a log file.

Description

Technical Field

[0001] The present invention relates to the field of anomaly analysis, and in particular to a method and system for analyzing GTS data anomalies. Background Art

[0002] For a long time, the acquisition of on-site marine observation data has been regarded as a very valuable information resource. Due to the complex marine observation environment, high costs and great difficulties in deployment and maintenance operations, compared with land-based observation data, the acquisition of marine observation data is more difficult and the data is sparser. Especially the on-site marine data with near real-time, all-weather, global scope, and various types of observation elements is even harder to come by. In recent years, in order to strengthen the marine observation ability, many countries and international organizations around the world have carried out a number of marine observation programs globally and effectively organized and utilized these data resources through the Global Telecommunication System (GTS), providing strong data support and services for marine and meteorological forecasting centers globally.

[0003] In recent years, the number of countries, regions, and organizations participating in the GTS for global international exchanges has been increasing continuously, the types of observations have been continuously expanding, and the volume of transmitted data has been continuously increasing. Since GTS observation reports are real-time observation data uploaded by multiple countries, regions, and organizations globally, there are differences in the observation equipment, instrument models, sampling resolutions, operation standards, submission formats, etc. used by each data collection agency. Therefore, the data quality of different reporting centers is uneven, and there are often abnormal situations that do not conform to the reporting specifications in the actually received messages. Since abnormal data is the main reason for causing the decoding system to crash or fall into an infinite loop, and it will cause the normal part in subsequent encoding to be unable to be processed normally.

[0004] Therefore, there is an urgent need for a new method for analyzing GTS data anomalies to automatically identify abnormal encodings, and through effective judgment, classify different types of abnormal situations, select corresponding decoding strategies for real-time processing, so as to minimize data misanalysis and subsequent message reading and decoding errors caused by abnormal data, and even program crashes. While filtering abnormal encodings, retain valid observation data to the greatest extent, and improve the operation efficiency and robustness of the decoding program. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for analyzing GTS data anomalies.

[0006] To achieve the above object, the present invention is implemented according to the following technical solutions: The present invention includes the following steps: Receive real-time GTS messages, integrate the original data in hours, cut the GTS messages according to the announced start identifier and end identifier to obtain complete messages, judge the message types of the complete messages according to the report identifier, and classify and store them according to the data type to obtain homogeneous data files, and read and parse the homogeneous data files; Read the complete announcement in the way of integer code cutting, divide the complete announcement into four levels and perform decoding processing to obtain level codes; among them, the first level includes the starting reporting line and the corresponding abbreviated title, and the information end flag; the second level includes the content of each announcement, also known as each observation report, and an observation report is distinguished by the report identifier as the start mark and the announcement content end identifier as the end mark; the third level includes each paragraph of each observation report, and each paragraph consists of several fourth levels, and each group reports an observation element; the fourth level includes each group in each paragraph; the level codes include the first level code, the second level code, the third level code and the fourth level code; Judge the abnormal conditions for the first-level code, judge and parse the abnormal conditions for the second-level code, judge the abnormal conditions for the third-level code and the fourth-level code, and process the third-level code and the fourth-level code; Integrate the end of the year in the message and the file generation time, obtain a reasonable time through comparison and adjustment, decode the fourth-level code, and record the abnormal details in the log file.

[0007] Further, the method for reading and parsing the homogeneous data files includes: The method of first cutting and reading the announcement in integer code and then performing hierarchical decoding, that is, identifying a complete announcement according to the starting reporting line identifier and the information end identifier of the announcement, and then performing hierarchical segmentation on different parts of the announcement and performing decoding processing on different levels hierarchically.

[0008] Further, the method for judging the abnormal conditions of the first-level code includes: Record the starting reporting line and the abbreviated title, and perform integrity and validity judgments; for the complete information that lacks the information transmission serial number, geographical / data type / time indication code, location identifier and date-time group that must be reported, do not decode the paragraphs of the entire announcement, look for the starting reporting line identifier of the next announcement downward, and perform repeated judgments until a complete and valid announcement is found; When the starting reporting line and the abbreviated title are complete, look for the corresponding information end flag; for the abnormal announcement lacking the information end flag, retain the observation data information to the greatest extent, decode sequentially downward from the starting reporting line until the starting reporting line identifier of the next announcement is encountered as the end flag; On the premise that neither of the first two abnormal situations is satisfied, judge the uniqueness condition of the first-level content; when there are multiple abbreviated titles appearing continuously, an abnormal error is thrown, the announcement is not parsed, and continue to search backward for the next announcement.

[0009] Furthermore, the method for judging and parsing the abnormal conditions of the second-level coding includes: Read the second-level coding, use the report identifier as the start marker, and the end marker of the announcement content as the end marker; In the case where the end marker of the announcement content is normally read at the end of the reporting group, process it according to the normal decoding process; In the case where the previous situation is not satisfied, continue to read downward and judge whether the next group of coding is the end marker of the announcement content. If it is, use this marker as the end; In the case where neither of the previous two situations is satisfied, continue to read downward and judge whether the next group of coding is the report identifier of the next announcement. If it is, stop decoding this announcement and enter the decoding loop of the next announcement; In the case where all the above situations are not satisfied, find the end marker of the information of the entire announcement and stop decoding.

[0010] Furthermore, the method for judging the abnormal conditions of the third-level coding and the fourth-level coding includes: Use the context logical relationship to judge the abnormality of the third level and the fourth level. When the content that must be reported is missing in the previous paragraph, but the content of the next paragraph is directly reported in the same line, and when the paragraph indicator code of the next paragraph does not conflict with the content missing in the previous paragraph, when parsing the group of the previous paragraph, first add a character judgment on whether the placeholder part of the group identification code overlaps with the paragraph identification code of the subsequent paragraph. If there is an overlapping situation, first judge according to the logical relationship between the paragraph identification code of the next paragraph and its element coding. If it conforms to the logical rules of the next paragraph, it can be judged as an abnormal coding situation, jump out of the decoding process of the previous paragraph, and proceed to the decoding of the next paragraph; When there are multiple consecutive missing paragraphs that must be reported, add the paragraph identification codes of multiple subsequent paragraphs during the overlapping judgment; When the necessary paragraph identification flag or group identification flag is missing, first judge the conditions according to the logical relationship between the upper and lower group messages; secondly, for data groups without an obvious context logical relationship, further judge according to the announcement content end symbol; When there is a situation where the ship call sign or station identification group, the announcement content end symbol, and the content of the next announcement are wrongly reported in the same group not in accordance with the specification, perform data segmentation according to the end identifier, and extract the content of the third-level coding and the fourth-level coding.

[0011] Furthermore, the method for processing the third-level coding and the fourth-level coding includes: In the actually received message, for the cases of non - standard - format line - breaking or misaligned reporting, or when the content of the same group of codes is misaligned and reported on different lines without exceeding the fixed width of the line, and these abnormal situations are not regular, generally manifested as inconsistent digit changes; among them, misaligned coding or the lack of standard delimiters between codes or the addition of garbled characters during communication causes inconsistent digit changes; For a group of codes in the message without obvious paragraph - ending or line - breaking marks, after reading, first judge whether the string occupancy width of the codes conforms to the standard occupancy width. For the non - conforming codes, make judgments and select decoding strategies according to the following two types of conditions: First, for the code group with a length greater than the standard occupancy width, first perform data segmentation. Read the corresponding number of characters according to the standard placeholder width of the next group of reporting in the reporting specification and record them in a temporary variable. Judge whether the group identification flag is equal to the next group identification flag; when the above - mentioned situation is met, perform decoding processing according to the decoding process of the next group of data and record it in the next group of data. At the same time, mask the already decoded string and perform the same repeated judgment on the remaining string; conversely, if effective data segmentation cannot be performed, mark this group of codes as abnormal and mask it. After masking, still read the data downward and perform repeated judgment; Second, for the code group with a length shorter than the standard width, judge whether it belongs to the situation of misaligned reporting across lines or the abnormal situation of insufficient coding digits; make a judgment based on the total occupancy width of this group of codes and the next group of codes and the first character of the codes. If the sum of the total lengths of the two groups of codes is equal to the total occupancy width of the codes in the standard situation and the group identification flag of the first group is also the same as that in the standard situation, it can be judged as the situation of misaligned reporting across lines; at this time, set a separate flag to mark the head and tail of the data across lines and perform splicing processing, and decode the complete record obtained after splicing; if the situation of misaligned reporting across lines is not met, it is the situation of insufficient coding digits, then it is determined as invalid coding, skip the code and point to the next position to be traversed; Judge the abnormal situation of confusion in the coding content of the observed elements in two types: The first type is to judge the situation of confusion between the codes in the same paragraph. Take the group identification flag as the principle of decoding judgment and make a combined judgment in combination with the logical relationship of the previous group of codes and the next group of codes; The second type is to judge the situation of confusion between the codes in different paragraphs. Since there may be the same group identification flag between different groups in different paragraphs, when the decoding of a certain group of data in a certain paragraph fails and the information of this group is not captured, set the traversal flag for the already traversed paragraphs and groups in time.

[0012] Furthermore, the method for obtaining a reasonable time by comparison and adjustment includes: Extract the parsed time group in the message to obtain the observation date, hour, minute, and the last digit of the year, obtain the system generation time of the data file, and extract the complete year, month, and day; Construct a confidence interval based on the file generation time and the maximum business backtracking period. Calculate the possible years based on the observation date, hour, minute, and the last digit of the year in the message, and filter the possible years through the confidence interval to obtain the confidence interval years; Concatenate the last digit of the year in the message with the first three digits of the file generation time. When the concatenated year is less than or equal to the file generation year, use the concatenated result as the first candidate time; when the concatenated year is greater than the file generation year, subtract 10 years from the concatenated year as the second candidate time; if the concatenated observation time is greater than the file generation time, subtract 1 from the month and compare again until it is within the confidence interval, and output the third candidate time; Use the first candidate time, the second candidate time, and the third candidate time as the candidate set. If the current message and the previous message are from the same sending agency and the time is not continuous, preferentially select the candidate set year close to the time of the previous message as the candidate year; If the message contains a month and the month does not belong to the nearest 3 months of the file generation time, the candidate year is used as the adjusted year; For high-frequency sending agencies, if the time jump is greater than 24 hours, mark it as abnormal and review the candidate year, and output a reasonable time.

[0013] Furthermore, the method for decoding the fourth-level encoding includes: For the encoding with variable number of groups and variable placeholder widths, when reading the corresponding paragraph identifier and the internal group identification identifier, first, when there is line break reporting, integrate the data according to the line break reporting conditions, set up a separate identifier to mark and splice the line break data, and use the element-based placeholder width, the continuity and consistency of the front and rear group encodings as the criteria during splicing; Second, according to the logical relationship of the front and rear group encodings, use the group identification identifier as a marker to circularly extract each set of complete matching records and store them in the corresponding data space; set up the paragraph identification identifier of the rear paragraph as the stop prompt flag for circular extraction during the circular extraction process; Third, for each set of complete matching records stored in the data space, find the group identification identifier, select the group decoding strategy and value conversion algorithm according to the group identification identifier, calculate the value and store the data of the same set of matching records in different data members of the same data structure.

[0014] On the second aspect, a GTS data anomaly parsing system includes: Receiving and Classifying Storage Module: It is used to receive real-time GTS messages, integrate the original data in hours, cut the GTS messages according to the announced start identifier and end identifier to obtain complete messages, judge the message types of the complete messages according to the report identifier, classify and store them according to the data type to obtain homogeneous data files, and read and parse the homogeneous data files; Cutting and Hierarchical Encoding Module: It is used to read the complete announcement in the way of integer code cutting, divide the complete announcement into four levels and perform decoding processing to obtain hierarchical encoding; among them, the first level includes the start reporting line and the corresponding abbreviated title, and the information end flag; the second level includes the content of each announcement, also known as each observation report, and an observation report is distinguished by the report identifier as the start mark and the announcement content end identifier as the end mark; the third level includes each paragraph of each observation report, and each paragraph is composed of several fourth levels, and each group reports an observation element; the fourth level includes each group in each paragraph; the hierarchical encoding includes the first-level encoding, the second-level encoding, the third-level encoding and the fourth-level encoding; Abnormality Judgment and Processing Module: It is used to judge the abnormal conditions of the first-level encoding, judge and analyze the abnormal conditions of the second-level encoding, judge the abnormal conditions of the third-level encoding and the fourth-level encoding, and process the third-level encoding and the fourth-level encoding; Adjusting and Decoding Record Module: Integrate the end of the year in the message and the file generation time, obtain a reasonable time through comparison and adjustment, decode the fourth-level encoding, and record the abnormal details in the log file.

[0015] The beneficial effects of the present invention are: The present invention is a method and system for parsing GTS data abnormalities. Compared with the prior art, the present invention has the following technical effects: The present invention has developed a set of methods and systems for automatically parsing GTS ocean observation data and processing abnormal data that have been operationalized and run in real time. According to the pre-set data parsing rules, it can automatically identify abnormal encodings, and through effective judgment, classify different types of abnormal situations, and select corresponding decoding strategies for real-time processing to minimize data misparsing and subsequent message reading and decoding errors and even program crashes caused by abnormal data. While filtering abnormal encodings, it maximally retains valid observation data, improving the operation efficiency and robustness of the decoding program. Description of the Drawings

[0016] Figure 1 It is the step flow chart of a method for parsing GTS data abnormalities of the present invention; Figure 2 It is the abnormal data log file output by decoding in the embodiment of the present specification; Figure 3 This is the decoded output result file in the embodiments of this specification. Specific implementation manners

[0017] The present invention will be further described below through specific embodiments. The illustrative embodiments and descriptions of this invention are used to explain the present invention, but not to limit the present invention.

[0018] A method and system for parsing GTS data anomalies of the present invention include the following steps: As Figure 1 shown, in this embodiment, the following steps are included: Receive real-time GTS messages, integrate the original data in hours, cut the GTS messages according to the announced start identifier and end identifier to obtain complete messages, judge the message types of the complete messages according to the report identifier, classify and store the same type of data files according to the data type, and read and parse the same type of data files; In actual evaluation, one announcement only corresponds to a unique start reporting line and an abbreviated title. There can be multiple announcement contents, and each announcement content can contain multiple paragraphs; GTS data reporting is carried out in a unified format specified by WMO for global exchange; a complete GTS observation report consists of four parts: a start reporting line, an abbreviated title, the content of the announcement, and an information end flag of the announcement; The content of the announcement is the core part. One announcement can contain multiple paragraphs, and each paragraph is distinguished by a paragraph identification identifier. The paragraph numbers start from 0 or 1; some of these paragraphs are mandatory, usually the first one or two paragraphs, and the basic information such as station number, data collection time, longitude and latitude position, etc. and the elements information that must be observed are reported; other paragraphs are selectively reported and are selectively reported according to the data actually collected by different reporting agencies on site; Read the complete announcement in an integer code cutting manner, divide the complete announcement into four levels and perform decoding processing to obtain level codes; among them, the first level includes the start reporting line and the corresponding abbreviated title, information end flag; the second level includes the content of each announcement, also called each observation report. An observation report is distinguished by a report identifier as the start mark and an announcement content end identifier as the end mark; the third level includes each paragraph of each observation report, and each paragraph consists of several fourth levels, and each group reports an observation element; the fourth level includes each group in each paragraph; the level codes include first-level codes, second-level codes, third-level codes and fourth-level codes; Perform an abnormal condition judgment on the first-level coding, perform an abnormal condition judgment and parsing on the second-level coding, perform an abnormal condition judgment on the third-level coding and the fourth-level coding, and process the third-level coding and the fourth-level coding; Integrate the end of the year in the message with the file generation time, obtain a reasonable time through comparison and adjustment, decode the fourth-level coding, and record the abnormal details in the log file.

[0019] In this embodiment, the method for reading and parsing the same type of data file includes: First, perform a whole-code cutting to read the announcement and then perform hierarchical decoding, that is, identify a complete announcement according to the starting reporting line identifier and the information end identifier of the announcement, then perform hierarchical segmentation on different parts of the announcement, and perform hierarchical decoding processing on different levels.

[0020] In this embodiment, the method for judging the abnormal conditions of the first-level coding includes: Record the starting reporting line and the abbreviated title, and perform integrity and validity judgments; for the complete information lacking the mandatory information transmission sequence number, geographical / data type / time indicator code, location identifier, and date and time group, do not decode the paragraphs of the whole announcement, look for the starting reporting line identifier of the next announcement downward, and perform repeated judgments until a complete and valid announcement is found; When the starting reporting line and the abbreviated title are complete, look for the corresponding information end flag; for the abnormal announcement lacking the information end flag, retain the observation data information to the greatest extent, decode sequentially downward from the starting reporting line until the starting reporting line identifier of the next announcement is encountered as the end flag; On the premise that the first two abnormal situations are not met, judge the uniqueness condition of the first-level content; throw an exception for the situation where the abbreviated title appears continuously multiple times, do not parse the announcement, and continue to look for the next announcement backward; In the actual evaluation, in the thermohaline current message, the reporting content after the first group of abbreviated titles is garbled, there is no announcement identifier, and there is no announcement content end flag "=", and then the abbreviated title is reported again, which does not conform to the principle that the abbreviated title of an announcement issued by a reporting agency should be unique; throw an exception for the situation where the abbreviated title appears continuously multiple times, do not parse this announcement, and continue to look for the next announcement backward.

[0021] In this embodiment, the method for judging and parsing the abnormal conditions of the second-level coding includes: Read the second-level coding, read the report identifier as the start mark, and read the announcement content end flag as the end mark; In the case where the end flag of the announcement content is normally read at the end of the reporting group, it is processed according to the normal decoding process; In the case where the previous situation is not satisfied, continue to read downward and determine whether the next group of codes is the end flag of the announcement content. If so, end with this flag; In the case where the previous two situations are not satisfied, continue to read downward and determine whether the next group of codes is the report identifier of the next announcement. If so, stop decoding this announcement and enter the decoding loop of the next announcement; In the case where all the above situations are not satisfied, find the end flag of the information of the entire announcement and stop decoding.

[0022] In this embodiment, the method for judging the abnormal conditions of the third-level code and the fourth-level code includes: Use the context logical relationship to judge the abnormality of the third level and the fourth level. When the content that must be reported is missing in the previous paragraph, but the content of the next paragraph is directly reported in the same line, and when the paragraph indicator code of the next paragraph does not conflict with the content missing in the previous paragraph, when parsing the group of the previous paragraph, first add a character judgment on whether the placeholder part of the group identification code overlaps with the paragraph identification code of the subsequent paragraph. If an overlapping situation occurs, first judge according to the logical relationship between the paragraph identification code of the next paragraph and its element code. If it conforms to the logical rules of the next paragraph, it can be judged that it is an abnormal coding situation, jump out of the decoding process of the previous paragraph, and proceed to decode the next paragraph; When there are multiple consecutive missing paragraphs that must be reported, add the paragraph identification codes of multiple subsequent paragraphs during the overlapping judgment; When the necessary paragraph identification flag or group identification flag is missing. For example, in the thermohaline current report, the ship call sign or station identification group is a digital code. To distinguish it from the previous observation data, theoretically, there should be a 99999 identifier for marking, and there is an announcement content end character '=' for marking at the back. However, in the actual message, there are situations where the 99999 identifier is missing and the ship call sign or station identification group is directly reported, and there are also situations where the data group following 99999 does not have an '=' end identifier. Since the ship call sign or station identification group is usually a digital code and the number of digits is not fixed, there will be a special situation of confusion with the observation data coding; First, make a conditional judgment according to the logical relationship between the upper and lower group messages; Secondly, for data groups without an obvious context logical relationship, further judge according to the announcement content end character; When there is a situation where the ship call sign or station identification group, the announcement content end character, and the next announcement content are incorrectly reported in the same group not in accordance with the specification, perform data segmentation according to the end identifier, and extract the content of the third-level code and the fourth-level code; In the actual evaluation, when distinguishing between the ship call sign and the temperature, salinity and depth group data, the joint relationship between the three groups of temperature, salinity and depth data and the logical order of the beginning of the observation group are used to make a circular judgment. The data group with a circular relationship should not be judged as the ship call sign; secondly, for the data group without obvious contextual logical relationship, further judgment is made based on the end mark of the announcement content; because there is a situation where the ship call sign or station identification group, the end mark of the announcement content and the next announcement content are not reported in the same group according to the specifications, it will also cause decoding anomalies. At this time, it is necessary to further segment the data according to the '=' mark, and then extract the corresponding third-level code and fourth-level code content.

[0023] In this embodiment, the method for processing the third level coding and the fourth level coding includes: In the messages actually received, there are cases where the line breaks or misplaced reporting are not in accordance with the standard format, or the content of the same group of codes is misplaced in different lines without exceeding the fixed line width, and these abnormal situations are irregular, generally manifested as non-fixed changes in the number of digits; among them, the misplaced coding or the lack of standard separators between codes or the addition of garbled codes during the communication process cause the non-fixed changes in the number of digits; for example, in the temperature and salinity flow section of the buoy report, for the salinity data at a specific depth, starting with 4 and the string length is 5, the salinity data will be misplaced, with the first 2 digits being reported at the end of the first line and the last 3 digits at the beginning of the second line. When the code of the last 3 digits is 444 or 555, it is the same as the segment indicator code 444 or 555 of the 5th or 6th paragraph. Since there is no obvious paragraph end mark or line break mark in the message, it is necessary to avoid the decoding program from misjudging the salinity code as the segment indicator code of the next paragraph; In the case that there is no obvious paragraph end mark or line break mark in the message, for a group of codes, after reading, first determine whether the string placeholder width meets the standard placeholder width. For codes that do not meet the standard, determine and select a decoding strategy based on the following two types of conditions: First, for the coding group whose length is greater than the standard placeholder width, firstly split the data, read the characters of the corresponding number of digits according to the standard placeholder width of the next group in the reporting specification and record them in the temporary variable, and judge whether the group identification mark is equal to the next group identification mark; when the above situation is met, decode and process according to the decoding process of the next group of data and record them in the next group of data, and at the same time mask the decoded character string and perform the same repeated judgment on the remaining character strings; otherwise, effective data segmentation cannot be performed, and this group of codes is marked as abnormal and masked, and the data is still read downward after masking and repeated judgment is performed; Second, for the coding group with a length shorter than the standard width, determine whether it belongs to the case of out-of-line misalignment reporting or the abnormal case of insufficient coding digits; make a judgment based on the total occupancy width of the current group's coding and the next group's coding and the first character of the coding. If the sum of the total lengths of the two groups of coding is equal to the total occupancy width of the coding under the specification and the group identification flag of the first group is also the same as that under the specification, it can be judged as the case of out-of-line misalignment reporting; at this time, set a separate flag to mark and splice the head and tail of the out-of-line data, and decode the complete record obtained after splicing; if it does not meet the case of out-of-line reporting, it is the case of insufficient coding digits, then it is recognized as invalid coding, skip the code and point to the next position to be traversed; Judge the abnormal situation of the confusion of the coding content of the observed elements in two types: The first type is to judge the situation of confusion between the codings in the same paragraph. Take the group identification flag as the principle of decoding judgment, and make a joint judgment in combination with the logical relationship of the previous group of coding and the next group of coding; for example, in the engineering technical parameter section of the buoy report, each quality control parameter is not a content that must be reported and is selectively reported; among them, the first two digits of the time correction group record the date. If the first two digits are 30 or 31, it conflicts with the next group starting with 3 and recording cable information, and the specified lengths of the two groups of reporting are the same, so misidentification is likely to occur during the decoding process; another example is that in the temperature-salinity-current message, the flow direction and velocity group d0d0c0c0c0 in the 66 ocean current section is reported as 25005, which is the same as the beginning of the water depth group 2z0z0z0z020012. Therefore, the group identification flag alone cannot be used as the principle of decoding judgment; The second type is to judge the situation of confusion between the codings in different paragraphs. Since there may be the same group identification flag between different groups in different paragraphs, when a certain group of data in a certain paragraph fails to be decoded and the information of this group is not captured, set the traversal flag for the paragraphs and groups that have been traversed in time.

[0024] In this embodiment, the method for obtaining a reasonable time by comparison and adjustment includes: Extract the time group parsed from the message, obtain the observation date, hour, minute and the last digit of the year, obtain the system generation time of the data file, and extract the complete year, month and day; Construct a credible interval according to the file generation time and the maximum business backtracking period, calculate the possible years according to the observation date, hour, minute and the last digit of the year in the message, and obtain the credible interval years by filtering the possible years through the credible interval; Concatenate the last digit of the year in the message with the first three digits of the file generation time. When the concatenated year is less than or equal to the file generation year, use the concatenated result as the first candidate time; when the concatenated year is greater than the file generation year, subtract 10 years from the concatenated year as the second candidate time; if the concatenated observation time is greater than the file generation time, subtract 1 from the month and compare again until it is within the credible interval, and output the third candidate time. Use the first candidate time, the second candidate time, and the third candidate time as the candidate set. If the current message and the previous message are from the same reporting agency and the time is not continuous, preferentially select the candidate year in the candidate set that is close to the time of the previous message as the candidate year. If the message contains a month and the month does not belong to the last 3 months of the file generation time, the candidate year is used as the adjusted year. For high-frequency reporting agencies, if the time jump is greater than 24 hours, mark it as abnormal and review the candidate year, and output a reasonable time.

[0025] In this embodiment, the method for decoding the fourth-level encoding includes: For encodings with variable number of groups and variable placeholder widths, when the corresponding paragraph identifier and the internal group identification identifier are read, first, when there is line break reporting, perform data integration according to the line break reporting conditions, set up a separate identifier to flag and concatenate the line break data, and use the element reference placeholder width, the continuity and consistency of the front and rear group encodings as the criteria during concatenation. Second, according to the logical relationship of the front and rear group encodings, use the group identification identifier as a marker to cyclically extract each complete set of matching records and store them in the corresponding data space; set up the paragraph identification identifier of the rear paragraph as the stop prompt flag for cyclic extraction during the cyclic extraction process. Third, for each complete set of matching records stored in the data space, find the group identification identifier, select the group decoding strategy and value conversion algorithm according to the group identification identifier, calculate the value, and store the data of the same set of matching records in different data members of the same data structure. In actual evaluation, when entering each announcement content, perform segmented decoding according to the paragraph identifier. When entering each paragraph, perform grouped decoding according to the group identification identifier of each group of encodings. For the temperature, salinity, and depth group data in the temperature-salinity current report, due to differences in the physical elements, number of profile layers, and total depth actually collected by different reporting agencies, the actual number of groups reported and the total string placeholder length are not fixed and are determined according to the on-site collection situation. When k2 is 1, it indicates that there is salinity data. The correct reporting order of the temperature-salinity-depth group should be 234, 234, 234. However, the actual reporting order may be a random combination of 23, 2, 234, 24, with irregular reporting. The actual number of groups and the total length of string placeholders are not fixed. When k2 = 0, the situation of 234, 234, 234 may also occur. Especially when the profile is deep, the data record is long, and the same group of data may be split during the recording process, it is more likely to have data parsing errors.

[0026] In a second aspect, a GTS data abnormal parsing system includes: A receiving and classifying storage module: used to receive real-time GTS messages, integrate the original data in hours, cut the GTS message according to the announced start identifier and end identifier to obtain a complete message, judge the message type of the complete message according to the report identifier, classify and store the same type of data files by data type, and read and parse the same type of data files; A cutting, layering and encoding module: used to read the complete announcement by using the integer code cutting method, divide the complete announcement into four levels and perform decoding processing to obtain level codes. The first level includes the starting reporting line and the corresponding abbreviated title, and the information end flag; the second level includes the content of each announcement, also known as each observation report. An observation report is distinguished by the report identifier as the start mark and the announcement content end identifier as the end mark; the third level includes each paragraph of each observation report, and each paragraph consists of several fourth levels, and each group reports an observation element; the fourth level includes each group in each paragraph; the level codes include the first level code, the second level code, the third level code and the fourth level code; An abnormal judgment and processing module: used to judge abnormal conditions for the first level code, judge and parse abnormal conditions for the second level code, judge abnormal conditions for the third level code and the fourth level code, and process the third level code and the fourth level code; An adjustment and decoding record module: integrate the end of the year in the message and the file generation time, obtain a reasonable time through comparison and adjustment, decode the fourth level code, and record the abnormal details in a log file.

[0027] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for parsing abnormal GTS data, characterized in that, It includes the following steps: Receive real-time GTS messages, integrate the original data in hours, cut the GTS messages according to the announced start identifier and end identifier to obtain complete messages, judge the message types of the complete messages according to the report identifier, classify and store them according to the data type to obtain homogeneous data files, and read and parse the homogeneous data files; Read the complete announcement in the way of integer code cutting, divide the complete announcement into four levels and perform decoding processing to obtain level codes; among them, the first level includes the start reporting line and the corresponding abbreviated title, and the information end flag; the second level includes the content of each announcement, also known as each observation report, and an observation report is distinguished by the report identifier as the start mark and the announcement content end identifier as the end mark; The third level includes each paragraph of each observation report, each paragraph consists of several fourth levels, and each group reports an observation element; the fourth level includes each group in each paragraph; the level codes include the first-level code, the second-level code, the third-level code and the fourth-level code; Judge the abnormal conditions of the first-level code, judge and analyze the abnormal conditions of the second-level code, judge the abnormal conditions of the third-level code and the fourth-level code, and process the third-level code and the fourth-level code; Integrate the end of the year in the message and the file generation time, obtain a reasonable time through comparison and adjustment, decode the fourth-level code, and record the abnormal details in the log file.

2. The method for parsing abnormal GTS data according to claim 1, wherein The method for reading and parsing the homogeneous data files includes: The method of first cutting and reading the announcement in integer code and then performing hierarchical decoding, that is, identifying a complete announcement according to the start reporting line identifier of the announcement and the information end identifier, then performing hierarchical segmentation on different parts of the announcement, and performing decoding processing on different levels hierarchically.

3. The method for parsing abnormal GTS data according to claim 1, wherein The method for judging the abnormal conditions of the first-level code includes: Record the start reporting line and the abbreviated title, and perform integrity and validity judgments; for the complete information of the information transmission serial number, geographical / data type / time indication code, position identifier, and date and time group that must be reported but are missing, do not decode the paragraphs of the entire announcement, look for the start reporting line identifier of the next announcement downward, and perform repeated judgments until a complete and valid announcement is found; When the start reporting line and the abbreviated title are complete, look for the corresponding information end flag; for the abnormal announcement missing the information end flag, retain the observation data information to the greatest extent, decode sequentially downward from the start reporting line until the start reporting line identifier of the next announcement is encountered as the end flag; On the premise that the first two abnormal situations are not met, judge the uniqueness condition of the first-level content; throw an exception for the situation where the abbreviated title appears continuously multiple times, do not parse the announcement, and continue to look for the next announcement backward.

4. The method for parsing abnormal GTS data according to claim 1, characterized in that, The method for judging and analyzing the abnormal conditions of the second-level code includes: Read the second-level code, read the report identifier as the start mark, and read the announcement content end flag as the end mark; In the case where the end flag of the announcement content is normally read at the end of the reporting group, it is processed according to the normal decoding process; In the case where the previous situation is not satisfied, continue to read downward and determine whether the next group of codes is the end flag of the announcement content. If so, it ends with this flag; In the case where the previous two situations are not satisfied, continue to read downward and determine whether the next group of codes is the report identifier of the next announcement. If so, stop decoding this announcement and enter the decoding loop of the next announcement; In the case where all the above situations are not satisfied, find the end flag of the entire announcement information and stop decoding.

5. The method for parsing GTS data anomalies according to claim 1, wherein The method for judging the abnormal conditions of the third-level coding and the fourth-level coding includes: Using the context logical relationship to perform abnormal judgment on the third level and the fourth level. When the content that must be reported is missing in the previous paragraph, but the content of the next paragraph is directly reported in the same line, and when the paragraph indicator code of the next paragraph does not conflict with the content missing in the previous paragraph, when parsing the group of the previous paragraph, first add a character judgment on whether the placeholder part of the group identification code overlaps with the paragraph identification code of the subsequent paragraph. If an overlapping situation occurs, first judge according to the logical relationship between the paragraph identification code of the next paragraph and its element coding. If it conforms to the logical rules of the next paragraph, it can be judged as an abnormal coding situation, jump out of the decoding process of the previous paragraph, and perform decoding of the next paragraph; When there are multiple consecutive missing paragraphs that must be reported, add the paragraph identification codes of multiple subsequent paragraphs during the overlapping judgment; When the necessary paragraph identification flag or group identification flag is missing, first judge the conditions according to the logical relationship between the upper and lower group messages; Secondly, for data groups without obvious context logical relationships, further judge according to the announcement content end symbol; When there is a situation where the ship call sign or station identification group, the announcement content end symbol, and the content of the next announcement are incorrectly reported in the same group in violation of the specification, perform data segmentation according to the end identifier, and extract the content of the third-level coding and the fourth-level coding.

6. The method for parsing abnormal GTS data according to claim 1, wherein The method for processing the third-level coding and the fourth-level coding includes: In the actually received message, for the situation of non-standard format line break reporting or misaligned reporting, or the content of the same group of codes is misaligned and reported in different lines without exceeding the fixed width of the line, and these abnormal situations are not regular, generally manifested as inconsistent digit changes; Among them, misaligned coding or missing standard delimiters between codes or adding garbled characters during communication causes inconsistent digit changes; For a group of codes without obvious paragraph end flag or line break flag in the message, after reading, first judge whether the string placeholder width conforms to the standard placeholder width. For the non-conforming codes, judge and select the decoding strategy according to the following two types of conditions:

1. For an encoding group with a length greater than the standard placeholder width, first perform data segmentation. Read the corresponding number of characters according to the standard placeholder width of the next encoding in the encoding specification and record them in a temporary variable. Then, determine whether the group identification flag is equal to the next group identification flag. When the above conditions are met, perform decoding processing according to the decoding process of the next group of data and record it in the next group of data. At the same time, mask the decoded string and perform the same repeated judgment on the remaining string. Otherwise, effective data segmentation cannot be performed. Mark this group of encoding as abnormal and mask it. After masking, still read the data downward and perform repeated judgment; 2. For an encoding group with a length shorter than the standard width, determine whether it belongs to the case of out-of-line misaligned encoding or the abnormal case of insufficient encoding digits. Make a judgment based on the total placeholder width of this group of encoding and the next group of encoding and the first character of the encoding. If the sum of the total lengths of the two groups of encoding is equal to the total placeholder width of the encoding under the specification and the group identification flag of the first group is also the same as that under the specification, it can be judged as the case of out-of-line misaligned encoding. At this time, set a separate flag to mark the head and tail of the out-of-line data and perform splicing processing, and decode the complete record obtained after splicing. If the condition of out-of-line encoding is not met, it is the case of insufficient encoding digits, and it is considered an invalid encoding. Skip the code and point to the next position to be traversed; There are two types of abnormal situations for judging the confusion of the encoding content of the observed elements: The first type is to judge the confusion situation between the encodings in the same paragraph. Take the group identification flag as the principle of decoding judgment and make a combined judgment in combination with the logical relationship of the previous group of encoding and the next group of encoding; The second type is to judge the confusion situation between the encodings in different paragraphs. Since there may be the same group identification flag between different groups in different paragraphs, when the decoding of a certain group of data in a certain paragraph fails and the information of this group is not captured, set the traversal flag for the traversed paragraphs and groups in time.

7. The method for parsing GTS data anomalies according to claim 1, characterized in that The method for obtaining a reasonable time by comparison and adjustment includes: Extract the time group parsed from the message to obtain the observation date, hour, minute, and the last digit of the year. Obtain the system generation time of the data file and extract the complete year, month, and day; Construct a credible interval according to the file generation time and the maximum business backtracking period. Calculate the possible years according to the observation date, hour, minute, and the last digit of the year in the message, and filter the possible years through the credible interval to obtain the credible interval years; Concatenate the last digit of the year in the message with the first three digits of the file generation time. When the concatenated year is less than or equal to the file generation year, use the concatenated result as the first candidate time. When the concatenated year is greater than the file generation year, subtract 10 years from the concatenated year as the second candidate time. If the concatenated observation time is greater than the file generation time, subtract 1 from the month and compare again until it is within the credible interval, and output the third candidate time; Use the first candidate time, the second candidate time, and the third candidate time as the candidate set. If the current message and the previous message are from the same sending agency and the time is not continuous, preferentially select the candidate set year close to the time of the previous message as the candidate year; If the message contains a month and the month does not belong to the most recent 3 months of the file generation time, the candidate year is used as the adjusted year; For high-frequency message-sending institutions, if the time jump is greater than 24 hours, it is marked as abnormal and the candidate year is reviewed to output a reasonable time.

8. The method for parsing abnormal GTS data according to claim 1, wherein The method for decoding the fourth-level encoding includes: For encodings with variable number of groups and variable placeholder widths, when the corresponding paragraph identifier and the internal group identification identifier are read, first, when there is line break reporting, data integration is performed according to the line break reporting conditions, and a separate identifier is set to flag and splice the line break data. When splicing, the element-based placeholder width, the continuity and consistency of the front and rear group encodings are used as standards; Second, according to the logical relationship of the front and rear group encodings, the group identification identifier is used as a marker to cyclically extract each complete set of matching records and store them in the corresponding data space; during the cyclic extraction process, the paragraph identification identifier of the rear paragraph is set as the stop prompt flag for cyclic extraction; Third, for each complete set of matching records stored in the data space, the group identification identifier is searched for, and the group decoding strategy and value conversion algorithm are selected according to the group identification identifier to calculate the value and store the data of the same set of matching records in different data members of the same data structure.

9. A GTS data anomaly parsing system for performing the method according to any one of claims 1-8, characterized in that, It includes: Receiving and classifying storage module: used to receive real-time GTS messages, integrate the original data in hours, cut the GTS message according to the announced start identifier and end identifier to obtain a complete message, judge the message type of the complete message according to the report identifier, and classify and store it according to the data type to obtain a homogeneous data file, and read and parse the homogeneous data file; Cutting and hierarchical encoding module: used to read the complete announcement in an integer code cutting manner, divide the complete announcement into four levels and perform decoding processing to obtain hierarchical encoding; among them, the first level includes the starting reporting line and the corresponding abbreviated title, and the information end flag; the second level includes the content of each announcement, also called each observation report. An observation report is distinguished by the report identifier as the start marker and the announcement content end identifier as the end marker; The third level includes each paragraph of each observation report. Each paragraph consists of several fourth levels, and each group encodes an observation element; the fourth level includes each group in each paragraph; the hierarchical encoding includes the first-level encoding, the second-level encoding, the third-level encoding, and the fourth-level encoding; Abnormality judgment and processing module: used to judge the abnormal conditions of the first-level encoding, judge and analyze the abnormal conditions of the second-level encoding, judge the abnormal conditions of the third-level encoding and the fourth-level encoding, and process the third-level encoding and the fourth-level encoding; Adjusting and decoding record module: integrate the end of the year in the message and the file generation time, obtain a reasonable time through comparison and adjustment, decode the fourth-level encoding, and record the abnormal details in the log file.

Images