A method and system for parsing GTS data anomalies

Abstract

This invention discloses a GTS data anomaly analysis method and system, comprising: receiving real-time GTS messages; segmenting the GTS messages according to the start and end identifiers of the announcement to obtain complete messages; reading and parsing the data files of the same type; reading the complete announcement using a whole-code segmentation method; dividing the complete announcement into four levels and decoding it to obtain hierarchical codes; performing anomaly condition judgment on the first-level code; performing anomaly condition judgment and parsing on the second-level code; performing anomaly condition judgment on the third-level code and the fourth-level code; processing the third-level code and the fourth-level code; integrating the end of the year in the message with the file generation time, obtaining a reasonable time through comparison and adjustment; decoding the fourth-level code; and recording the anomaly details to a log file.

Description

Technical Field

[0001] This invention relates to the field of anomaly analysis, and in particular to a method and system for analyzing GTS data anomalies. Background Technology

[0002] For a long time, the acquisition of marine field observation data has been regarded as an invaluable information resource. Due to the complex marine observation environment and the high costs and difficulties of deployment and maintenance, marine observation data is more difficult to acquire and more sparse compared to land-based observation data. In particular, near-real-time, all-weather, global-scale marine field data with diverse observation elements is extremely rare. In recent years, to enhance marine observation capabilities, many countries and international organizations have launched numerous marine observation programs worldwide, effectively organizing and utilizing these data resources through the Global Telegraph and Telephone System (GTS), providing strong data support and services to marine and meteorological forecasting centers globally.

[0003] In recent years, the number of countries, regions, and organizations joining the Global System for Telecommunications (GTS) for global international data exchange has been increasing, the types of observations have been expanding, and the amount of transmitted data has been continuously growing. Because GTS observation reports consist of real-time observation data uploaded by multiple countries, regions, and organizations worldwide, the observation equipment, instrument models, sampling resolutions, operating standards, and submission formats used by different data acquisition agencies vary. Therefore, the data quality varies between different reporting centers, and abnormal situations that do not conform to the reporting specifications frequently appear in the received messages. Abnormal data is a major cause of decoding system crashes or infinite loops, and it also prevents the normal parts of subsequent encoding from being processed correctly.

[0004] Therefore, a new GTS data anomaly parsing method is urgently needed to automatically identify anomalous codes, effectively classify different types of anomalies, and select appropriate decoding strategies for real-time processing. This aims to minimize data parsing errors and subsequent message reading and decoding errors, or even program crashes, caused by anomalous data. While filtering anomalous codes, it should also retain as much valid observation data as possible, improving the efficiency and robustness of the decoding program. Summary of the Invention

[0005] The purpose of this invention is to provide a method for parsing GTS data anomalies.

[0006] To achieve the above objectives, the present invention is implemented according to the following technical solution:

[0007] This invention includes the following steps:

[0008] Receive real-time GTS messages, integrate raw data on an hourly basis, segment the GTS messages according to the start and end identifiers of the announcement to obtain complete messages, determine the message type of the complete messages according to the report identifier, and classify and store them according to data type to obtain similar data files, and read and parse the similar data files.

[0009] The complete announcement is read using a whole-code segmentation method, divided into four levels and decoded to obtain hierarchical encoding. The first level includes the starting line and its corresponding abbreviated title and information end marker. The second level contains the content of each announcement, also known as each observation report, distinguished by a report identifier as the start mark and an announcement content end marker as the end mark. The third level contains each paragraph of each observation report, with each paragraph consisting of several fourth-level groups, each group reporting one observation element. The fourth level contains each group within each paragraph. The hierarchical encoding includes first-level encoding, second-level encoding, third-level encoding, and fourth-level encoding.

[0010] Anomaly condition judgment is performed on the first level code, anomaly condition judgment and parsing are performed on the second level code, anomaly condition judgment is performed on the third level code and the fourth level code, and the third level code and the fourth level code are processed.

[0011] The end of the year in the message is combined with the file generation time. A reasonable time is obtained by comparison and adjustment. The fourth level of encoding is then decoded, and the exception details are recorded to the log file.

[0012] Furthermore, the method for reading and parsing the aforementioned similar data files includes:

[0013] The method involves first cutting and reading the entire announcement and then performing layered decoding. This means identifying a complete announcement based on the start line identifier and end information identifier, then dividing the different parts of the announcement into layers, and performing decoding processing on each layer.

[0014] Furthermore, the method for judging abnormal conditions in the first-level encoding includes:

[0015] Record the starting line and abbreviation title, and perform completeness and validity checks; if the required information transmission sequence number, geographic / data type / time indicator code, location identifier, and date and time group are missing, do not decode the paragraphs of the entire announcement, search downwards for the starting line identifier of the next announcement, and perform repeated checks until a complete and valid announcement is found;

[0016] If the starting line and abbreviation title are complete, find the corresponding information end marker; for abnormal announcements that lack information end markers, preserve the observation data information to the maximum extent, decode sequentially from the starting line until the starting line identifier of the next announcement is encountered as the end marker;

[0017] If neither of the first two abnormal scenarios is met, the uniqueness condition of the first-level content is judged; if multiple abbreviated titles appear consecutively, an error is thrown, the announcement is not parsed, and the search continues for the next announcement.

[0018] Furthermore, the method for judging and parsing abnormal conditions in the second-level encoding includes:

[0019] Read the second-level code, using the report identifier as the start marker and the announcement content end marker as the end marker;

[0020] If the end marker of the announcement content is read normally at the end of the editing group, proceed with the normal decoding process;

[0021] If the first condition is not met, continue reading downwards and determine whether the next set of codes is the end marker of the announcement content. If it is, then the announcement ends with that marker.

[0022] If neither of the first two conditions is met, continue reading downwards and determine whether the next set of codes is the report identifier for the next announcement. If it is, stop decoding the current announcement and enter the decoding loop for the next announcement. If none of the above conditions are met, look for the end marker of the entire announcement and stop decoding.

[0023] Furthermore, the method for judging the abnormal conditions of the third-level encoding and the fourth-level encoding includes:

[0024] Using contextual logic, anomaly detection is performed at the third and fourth levels. When the previous paragraph lacks required content, but the next paragraph's content is directly reported on the same line, and the next paragraph's segment indicator code does not conflict with the missing content in the previous paragraph, when parsing the previous paragraph's group, a character check is first added to the placeholder part of the group identifier code to determine if it overlaps with the segment identifier code of the following paragraph. If overlap occurs, the logical relationship between the next paragraph's segment identifier code and its element code is first used for judgment. If it conforms to the next paragraph's logical rules, it can be determined as an abnormal encoding case, and the decoding process of the previous paragraph is skipped to proceed to the next paragraph's decoding. When multiple required paragraphs are consecutively missing, the segment identifier codes of the following multiple paragraphs are added simultaneously during overlap detection.

[0025] When the necessary paragraph or group identification markers are missing, conditional judgment is made first based on the logical relationship between the upper and lower groups of messages; secondly, for data groups without obvious contextual logical relationships, further judgment is made based on the announcement content end marker; when there is a situation where the ship call sign or station identification group, the announcement content end marker, and the next announcement content are incorrectly reported in the same group, the data is further segmented according to the end marker, and the content of the third-level code and the fourth-level code is extracted.

[0026] Furthermore, the method for processing the third-level encoding and the fourth-level encoding includes:

[0027] In actual received messages, there are cases where the text is not formatted according to the standard format or is misaligned, or where the same set of encoded content is misaligned and encoded in different lines without exceeding the fixed line width. These abnormalities are not regular and are generally characterized by inconsistent changes in the number of bits. Among them, misaligned encoding, the lack of standard separators between encodings, or the addition of garbled characters during communication can cause inconsistent changes in the number of bits.

[0028] For messages without obvious paragraph end markers or line break markers, after reading a set of encodings, first determine whether its string width conforms to the standard width. For encodings that do not conform, the following two types of conditions are used for judgment and decoding strategy selection:

[0029] First, for coded groups whose length exceeds the standard placeholder width, the data is segmented. The corresponding number of characters is read according to the standard placeholder width of the next group in the coding specification and recorded in a temporary variable. The group identification identifier is then checked against the next group's identifier. If this condition is met, the data is decoded according to the decoding process of the next group and recorded in the next group. Simultaneously, the already decoded string is masked, and the remaining string is checked for duplicates. Otherwise, if effective data segmentation is not possible, this group of codes is marked as abnormal and masked. After masking, the data is still read downwards, and duplicate checks are performed.

[0030] Second, for code groups shorter than the standard width, determine whether it is a case of cross-line misalignment or an abnormal case of insufficient code bits. Based on the total width of the current code group and the next code group, and the first character of the code, if the sum of the lengths of the two code groups equals the total width of the code under the standard condition, and the group identifier of the first group is also the same as under the standard condition, then it can be determined as a cross-line misalignment. In this case, a separate identifier is set to mark the beginning and end of the cross-line data and perform concatenation processing, and the complete record obtained after concatenation is decoded. If it does not meet the cross-line reporting condition, then it is a case of insufficient code bits, and is therefore considered an invalid code, with the code skipping to the next position to be traversed.

[0031] There are two types of abnormal situations where the coded content of the observed elements is confused: The first type is to judge the situation where the codes of the same paragraph are confused. The group identification mark is used as the principle for decoding judgment, and the logical relationship between the previous group of codes and the next group of codes is combined for joint judgment.

[0032] The second type is to determine the situation where the encoding of different paragraphs is confused. Since there may be the same group identification mark between different groups in different paragraphs, when a group of data in a certain paragraph fails to be decoded, and when the information of that group is not captured, the traversal mark is set in time for the paragraphs and groups that have been traversed.

[0033] Furthermore, the method for obtaining a reasonable time through comparison and adjustment includes:

[0034] Extract the parsed time group from the message to obtain the observation day, hour, minute and last digit of the year, obtain the system generation time of the data file, and extract the complete year, month and day;

[0035] A reliable interval is constructed based on the file generation time and the maximum business backtracking cycle. Possible years are calculated based on the observation day, hour, minute and last digit of the year of the message. The reliable interval year is obtained by filtering the possible years through the reliable interval.

[0036] The last digit of the year in the message is concatenated with the first three digits of the file generation time. If the concatenated year is less than or equal to the file generation year, the concatenated result is taken as the first candidate time. If the concatenated year is greater than the file generation year, 10 years are subtracted from the concatenated year to take the second candidate time. If the concatenated observation time is greater than the file generation time, the month is subtracted by 1 and the comparison is repeated until it is within the confidence interval, at which point the third candidate time is output.

[0037] The first, second, and third candidate times are used as candidate sets. If the current message and the previous message come from the same sending organization, and if the times are not consecutive, the candidate year that is closer to the time of the previous message is selected as the candidate year.

[0038] If the message contains a month, and the month is not within the most recent 3 months of the file generation time, then the candidate year is used as the adjustment year;

[0039] For high-frequency reporting agencies, if the time jump is greater than 24 hours, it is marked as abnormal and the candidate years are reviewed to output a reasonable time.

[0040] Furthermore, the method for decoding the fourth-level encoding includes:

[0041] For encoding variable group number and variable placeholder width, when the corresponding paragraph identifier and internal group identification identifier are read, firstly, if there is a line break in the data reporting, the data is integrated according to the line break reporting conditions, and a separate identifier is set to mark and splice the line break data. When splicing, the element base placeholder width, the continuity and consistency of the preceding and following group codes are used as the standard.

[0042] Secondly, based on the logical relationship between the preceding and following group codes, each complete set of matching records is extracted cyclically using the group identification identifier as a marker and stored in the corresponding data space; during the cyclic extraction process, the segment identification identifier of the following segment is set as a stop prompt for the cyclic extraction.

[0043] Third, for each complete set of matching records stored in the data space, find the group identification identifier, select the group decoding strategy and value conversion algorithm according to the group identification identifier, calculate the value, and store the same set of matching records in different data members of the same data structure.

[0044] Secondly, a GTS data anomaly analysis system includes:

[0045] The receiving and classification storage module is used to receive real-time GTS messages, integrate the raw data in hours, cut the GTS messages into complete messages according to the start and end identifiers of the announcement, determine the message type of the complete message according to the report identifier, classify and store the data according to the data type to obtain similar data files, and read and parse the similar data files.

[0046] The layered encoding module is used to read the complete announcement using a whole-code segmentation method, divide the complete announcement into four layers, and decode them to obtain the layered encoding. The first layer includes the starting line and its corresponding abbreviated title and information end marker. The second layer contains the content of each announcement, also known as each observation report, with each observation report distinguished by a report identifier as the start marker and an announcement content end marker as the end marker. The third layer contains each paragraph of each observation report, with each paragraph consisting of several fourth-level segments, each group reporting one observation element. The fourth layer contains each group within each paragraph. The layered encoding includes first-level encoding, second-level encoding, third-level encoding, and fourth-level encoding.

[0047] Anomaly detection and processing module: used to perform anomaly condition judgment on the first level code, perform anomaly condition judgment and parsing on the second level code, perform anomaly condition judgment on the third level code and the fourth level code, and process the third level code and the fourth level code;

[0048] Adjust the decoding and recording module: integrate the end of the year in the message with the file generation time, obtain a reasonable time by comparison and adjustment, decode the fourth level encoding, and record the exception details to the log file.

[0049] The beneficial effects of this invention are:

[0050] This invention provides a method and system for parsing GTS data anomalies. Compared with existing technologies, this invention has the following technical advantages:

[0051] This invention develops a method and system for automatic parsing and anomaly processing of GTS marine observation data, which is already operational and running in real time. Based on pre-set data parsing rules, it automatically identifies anomaly codes, classifies different types of anomalies, and selects appropriate decoding strategies for real-time processing. This minimizes data parsing errors and subsequent message reading and decoding errors, and even program crashes, caused by anomaly data. While filtering out anomaly codes, it maximizes the retention of valid observation data, improving the efficiency and robustness of the decoding program. Attached Figure Description

[0052] Figure 1 This is a flowchart of the steps of a GTS data anomaly parsing method according to the present invention;

[0053] Figure 2 This refers to the abnormal data log file decoded and output in the embodiments of this specification;

[0054] Figure 3 This is the decoding output file in the embodiments of this specification. Detailed Implementation

[0055] The present invention will be further described below through specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0056] The present invention provides a GTS data anomaly analysis method and system, comprising the following steps:

[0057] like Figure 1 As shown, this embodiment includes the following steps:

[0058] Receive real-time GTS messages, integrate raw data on an hourly basis, segment the GTS messages according to the start and end identifiers of the announcement to obtain complete messages, determine the message type of the complete messages according to the report identifier, and classify and store them according to data type to obtain similar data files, and read and parse the similar data files.

[0059] In actual assessment, a single announcement corresponds to only one starting line and abbreviated title, but there can be multiple announcement contents, and each announcement content can contain multiple paragraphs;

[0060] GTS data reporting is exchanged globally in accordance with the format uniformly stipulated by WMO; a complete GTS observation report consists of four parts: the starting reporting line, the abbreviation title, the content of the announcement, and the information end marker of the announcement.

[0061] The content of the announcement is the core part. An announcement may contain multiple paragraphs, each of which is distinguished by a paragraph identifier. The paragraph numbering starts from 0 or 1. Some paragraphs are mandatory, usually the first one or two paragraphs, which include basic information such as station number, data collection time, latitude and longitude location, and information on the elements that must be observed. Other paragraphs are optional and are selectively compiled based on the actual data collected on-site by different reporting agencies.

[0062] The complete announcement is read using a whole-code segmentation method, divided into four levels and decoded to obtain hierarchical encoding. The first level includes the starting line and its corresponding abbreviated title and information end marker. The second level contains the content of each announcement, also known as each observation report, distinguished by a report identifier as the start mark and an announcement content end marker as the end mark. The third level contains each paragraph of each observation report, with each paragraph consisting of several fourth-level groups, each group reporting one observation element. The fourth level contains each group within each paragraph. The hierarchical encoding includes first-level encoding, second-level encoding, third-level encoding, and fourth-level encoding.

[0063] Anomaly condition judgment is performed on the first level code, anomaly condition judgment and parsing are performed on the second level code, anomaly condition judgment is performed on the third level code and the fourth level code, and the third level code and the fourth level code are processed.

[0064] The end of the year in the message is combined with the file generation time. A reasonable time is obtained by comparison and adjustment. The fourth level of encoding is then decoded, and the exception details are recorded to the log file.

[0065] In this embodiment, the method for reading and parsing the aforementioned similar data files includes:

[0066] The method involves first cutting and reading the entire announcement and then performing layered decoding. This means identifying a complete announcement based on the start line identifier and end information identifier, then dividing the different parts of the announcement into layers, and performing decoding processing on each layer.

[0067] In this embodiment, the method for judging abnormal conditions in the first-level encoding includes:

[0068] Record the starting line and abbreviation title, and perform completeness and validity checks; if the required information transmission sequence number, geographic / data type / time indicator code, location identifier, and date and time group are missing, do not decode the paragraphs of the entire announcement, search downwards for the starting line identifier of the next announcement, and perform repeated checks until a complete and valid announcement is found;

[0069] If the starting line and abbreviation title are complete, find the corresponding information end marker; for abnormal announcements that lack information end markers, preserve the observation data information to the maximum extent, decode sequentially from the starting line until the starting line identifier of the next announcement is encountered as the end marker;

[0070] If neither of the first two abnormal situations is met, the uniqueness condition of the first-level content is judged; if multiple abbreviated titles appear consecutively, an error is thrown, the announcement is not parsed, and the search continues for the next announcement.

[0071] In the actual assessment, the content following the first abbreviation title in the thermo-salinity flow report was garbled, lacked an announcement identifier, and lacked the "=" sign indicating the end of the announcement content. Subsequently, the abbreviation title was reported again, which does not comply with the principle that the abbreviation title of an announcement issued by a reporting agency should be unique. When multiple abbreviation titles appear consecutively, an error is thrown, the current announcement is not parsed, and the search continues for the next announcement.

[0072] In this embodiment, the method for judging and parsing abnormal conditions in the second-level encoding includes:

[0073] Read the second-level code, using the report identifier as the start marker and the announcement content end marker as the end marker;

[0074] If the end marker of the announcement content is read normally at the end of the editing group, proceed with the normal decoding process;

[0075] If the first condition is not met, continue reading downwards and determine whether the next set of codes is the end marker of the announcement content. If it is, then the announcement ends with that marker.

[0076] If neither of the first two conditions is met, continue reading downwards and determine whether the next set of codes is the report identifier for the next announcement. If it is, stop decoding the current announcement and enter the decoding loop for the next announcement. If none of the above conditions are met, look for the end marker of the entire announcement and stop decoding.

[0077] In this embodiment, the method for judging the abnormal conditions of the third-level encoding and the fourth-level encoding includes:

[0078] Using contextual logic, anomaly detection is performed at the third and fourth levels. When the previous paragraph lacks required content, but the next paragraph's content is directly reported on the same line, and the next paragraph's segment indicator code does not conflict with the missing content in the previous paragraph, when parsing the previous paragraph's group, a character check is first added to the placeholder part of the group identifier code to determine if it overlaps with the segment identifier code of the following paragraph. If overlap occurs, the logical relationship between the next paragraph's segment identifier code and its element code is first used for judgment. If it conforms to the next paragraph's logical rules, it can be determined as an abnormal encoding case, and the decoding process of the previous paragraph is skipped to proceed to the next paragraph's decoding. When multiple required paragraphs are consecutively missing, the segment identifier codes of the following multiple paragraphs are added simultaneously during overlap detection.

[0079] When necessary paragraph or group identification markers are missing, for example, in temperature, salinity, and current reports, ship call signs or station identification groups are numerically coded. In theory, to distinguish them from preceding observation data, they should be marked with a 99999 identifier and followed by an '=" sign to end the announcement content. However, in actual reports, there are cases where the 99999 identifier is missing and the ship call sign or station identification group is directly reported, or where the data group following 99999 does not have an '=" sign to end the announcement content. Since ship call signs or station identification groups are usually numerically coded and the number of digits is not fixed, special situations may arise that can be confused with the code of the observation data.

[0080] First, conditional judgments are made based on the logical relationship between the upper and lower groups of messages; second, for data groups without obvious contextual logical relationships, further judgments are made based on the end marker of the announcement content; when there is a situation where the ship call sign or station identification group, the end marker of the announcement content, and the next announcement content are incorrectly compiled into the same group, the data is further segmented according to the end marker, and the content of the third-level code and the fourth-level code is extracted.

[0081] In actual assessments, when distinguishing between ship call signs and temperature, salinity, and depth (TSD) data, the relationship between the three data sets is used, and the logical order at the beginning of the observation groups is used for cyclical judgment. Data groups with cyclical relationships should not be identified as ship call signs. Secondly, for data groups without obvious contextual logical relationships, further judgment is made based on the end marker of the announcement content. Because there may be cases where ship call signs or station identification groups, the end marker of the announcement content, and the next announcement content are incorrectly reported in the same group, decoding anomalies will also occur. In this case, it is necessary to further segment the data according to the '=" marker and then extract the corresponding third-level and fourth-level codes.

[0082] In this embodiment, the method for processing the third-level encoding and the fourth-level encoding includes:

[0083] In actual received messages, there are cases where the text is not formatted according to the standard format, or where the text is misaligned. For example, the same set of encoded content may be misaligned and displayed on different lines without exceeding the fixed line width. These anomalies are not regular and generally manifest as inconsistent bit length variations. Misaligned encoding, the lack of standard separators between codes, or the introduction of garbled characters during communication can cause inconsistent bit length variations. For instance, in the salinity data of a buoy report, for salinity data at a specific depth starting with 4 and with a string length of 5, the salinity data may be misaligned, with the first two bits displayed at the end of the first line and the last three bits at the beginning of the second line. When the last three bits are encoded as 444 or 555, they are the same as the segment indicator code 444 or 555 for the 5th or 6th segment. Since there are no obvious paragraph end marks or line break marks in the message, it is necessary to prevent the decoding program from misinterpreting the salinity code as the segment indicator code for the next segment.

[0084] For messages without obvious paragraph end markers or line break markers, after reading a set of encodings, first determine whether its string width conforms to the standard width. For encodings that do not conform, the following two types of conditions are used for judgment and decoding strategy selection:

[0085] First, for coded groups whose length exceeds the standard placeholder width, the data is segmented. The corresponding number of characters is read according to the standard placeholder width of the next group in the coding specification and recorded in a temporary variable. The group identification identifier is then checked against the next group's identifier. If this condition is met, the data is decoded according to the decoding process of the next group and recorded in the next group. Simultaneously, the already decoded string is masked, and the remaining string is checked for duplicates. Otherwise, if effective data segmentation is not possible, this group of codes is marked as abnormal and masked. After masking, the data is still read downwards, and duplicate checks are performed.

[0086] Second, for code groups shorter than the standard width, determine whether it is a case of cross-line misalignment or an abnormal case of insufficient code bits. Based on the total width of the current code group and the next code group, and the first character of the code, if the sum of the lengths of the two code groups equals the total width of the code under the standard condition, and the group identifier of the first group is also the same as under the standard condition, then it can be determined as a cross-line misalignment. In this case, a separate identifier is set to mark the beginning and end of the cross-line data and perform concatenation processing, and the complete record obtained after concatenation is decoded. If it does not meet the cross-line reporting condition, then it is a case of insufficient code bits, and is therefore considered an invalid code, with the code skipping to the next position to be traversed.

[0087] There are two types of abnormal situations where the code content of observation elements is confused: The first type is the situation where the codes in the same segment are confused. The group identification mark is used as the principle for decoding judgment, and the logical relationship between the previous group of codes and the next group of codes is combined for joint judgment. For example, in the engineering and technical parameters section of the buoy report, each quality control parameter is not required to be reported and is selectively reported. The first two digits of the time correction group record the date. If the first two digits are 30 or 31, it will conflict with the group that starts with 3 and records cable information. Moreover, the prescribed length of the two groups is the same, which is easy to misidentify during the decoding process. Another example is that in the temperature, salinity and current report, the current direction and velocity group d0d0c0c0c0 in the 66 ocean current segment is reported as 25005, which is the same as the beginning of the water depth group 2z0z0z0z020012. Therefore, the group identification mark cannot be used as the principle for decoding judgment alone.

[0088] The second type is to determine the situation where the encoding of different paragraphs is confused. Since there may be the same group identification mark between different groups in different paragraphs, when a group of data in a certain paragraph fails to be decoded, and when the information of that group is not captured, the traversal mark is set in time for the paragraphs and groups that have been traversed.

[0089] In this embodiment, the method for obtaining a reasonable time through comparison and adjustment includes:

[0090] Extract the parsed time group from the message to obtain the observation day, hour, minute and last digit of the year, obtain the system generation time of the data file, and extract the complete year, month and day;

[0091] A reliable interval is constructed based on the file generation time and the maximum business backtracking cycle. Possible years are calculated based on the observation day, hour, minute and last digit of the year of the message. The reliable interval year is obtained by filtering the possible years through the reliable interval.

[0092] The last digit of the year in the message is concatenated with the first three digits of the file generation time. If the concatenated year is less than or equal to the file generation year, the concatenated result is taken as the first candidate time. If the concatenated year is greater than the file generation year, 10 years are subtracted from the concatenated year to take the second candidate time. If the concatenated observation time is greater than the file generation time, the month is subtracted by 1 and the comparison is repeated until it is within the confidence interval, at which point the third candidate time is output.

[0093] The first, second, and third candidate times are used as candidate sets. If the current message and the previous message come from the same sending organization, and if the times are not consecutive, the candidate year that is closer to the time of the previous message is selected as the candidate year.

[0094] If the message contains a month, and the month is not within the most recent 3 months of the file generation time, then the candidate year is used as the adjustment year;

[0095] For high-frequency reporting agencies, if the time jump is greater than 24 hours, it is marked as abnormal and the candidate years are reviewed to output a reasonable time.

[0096] In this embodiment, the method for decoding the fourth-level encoding includes:

[0097] For encoding variable group number and variable placeholder width, when the corresponding paragraph identifier and internal group identification identifier are read, firstly, if there is a line break in the data reporting, the data is integrated according to the line break reporting conditions, and a separate identifier is set to mark and splice the line break data. When splicing, the element base placeholder width, the continuity and consistency of the preceding and following group codes are used as the standard.

[0098] Secondly, based on the logical relationship between the preceding and following group codes, each complete set of matching records is extracted cyclically using the group identification identifier as a marker and stored in the corresponding data space; during the cyclic extraction process, the segment identification identifier of the following segment is set as a stop prompt for the cyclic extraction.

[0099] Third, for each complete set of matching records stored in the data space, find the group identification identifier, select the group decoding strategy and value conversion algorithm according to the group identification identifier, calculate the value, and store the same set of matching records in different data members of the same data structure;

[0100] In actual evaluation, when entering each announcement content, it is decoded segment by segment according to paragraph identifiers, and when entering each paragraph, it is decoded group by group identifiers of each group of codes.

[0101] For the temperature, salinity, and depth group data in the temperature, salinity, and depth reports, the actual number of groups and the total length of the strings are not fixed and are determined based on the on-site data collection conditions, because different reporting agencies collect different physical elements, number of profile layers, and total depth.

[0102] When k2 is 1, it indicates that salinity data is available. The correct reporting order for the temperature, salinity, and depth groups should be 234, 234, 234. However, the actual reporting order may be a random combination of 23, 2, 234, 24, resulting in non-standard reporting and inconsistent actual group numbers and total string length. Similarly, when k2 = 0, the order 234, 234, 234 may also occur. This is especially problematic when the profile is deep, the data records are long, and the same data set may be split during the recording process, making data parsing errors more likely.

[0103] Secondly, a GTS data anomaly analysis system includes:

[0104] The receiving and classification storage module is used to receive real-time GTS messages, integrate the raw data in hours, cut the GTS messages into complete messages according to the start and end identifiers of the announcement, determine the message type of the complete message according to the report identifier, classify and store the data according to the data type to obtain similar data files, and read and parse the similar data files.

[0105] The layered encoding module is used to read the complete announcement using a whole-code segmentation method, divide the complete announcement into four layers, and decode them to obtain the layered encoding. The first layer includes the starting line and its corresponding abbreviated title and information end marker. The second layer contains the content of each announcement, also known as each observation report, with each observation report distinguished by a report identifier as the start marker and an announcement content end marker as the end marker. The third layer contains each paragraph of each observation report, with each paragraph consisting of several fourth-level segments, each group reporting one observation element. The fourth layer contains each group within each paragraph. The layered encoding includes first-level encoding, second-level encoding, third-level encoding, and fourth-level encoding.

[0106] Anomaly detection and processing module: used to perform anomaly condition judgment on the first level code, perform anomaly condition judgment and parsing on the second level code, perform anomaly condition judgment on the third level code and the fourth level code, and process the third level code and the fourth level code;

[0107] Adjust the decoding and recording module: integrate the end of the year in the message with the file generation time, obtain a reasonable time by comparison and adjustment, decode the fourth level encoding, and record the exception details to the log file.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for parsing GTS data anomalies, characterized in that, Includes the following steps: Receive real-time GTS messages, integrate raw data on an hourly basis, segment the GTS messages according to the start and end identifiers of the announcement to obtain complete messages, determine the message type of the complete messages according to the report identifier, and classify and store them according to data type to obtain similar data files, and read and parse the similar data files. The complete announcement is read using a whole-code segmentation method. The complete announcement is divided into four levels and parsed to obtain the hierarchical code. The first level includes the starting line and the corresponding abbreviated title and information end mark. The second level includes the content of each announcement, also known as each observation report. An observation report is distinguished by a report identifier as the start mark and an announcement content end mark as the end mark. The third level contains each paragraph of each observation report, and each paragraph consists of several fourth levels, with each group reporting one observation element; the fourth level contains each group within each paragraph; the hierarchical coding includes first-level coding, second-level coding, third-level coding, and fourth-level coding; Anomaly condition judgment is performed on the first level code, anomaly condition judgment and parsing are performed on the second level code, anomaly condition judgment is performed on the third level code and the fourth level code, and the third level code and the fourth level code are processed. The last digit of the year in the message is integrated with the file generation time. A reasonable time is obtained through comparison and adjustment. The fourth-level encoding is then decoded, and the exception details are recorded to the log file. The method for processing the third-level encoding and the fourth-level encoding includes: For messages without obvious paragraph end markers or line break markers, after reading a set of encodings, first determine whether its string width conforms to the standard width. For encodings that do not conform, the following two types of conditions are used for judgment and decoding strategy selection: First, for coded groups whose length exceeds the standard placeholder width, the data is segmented. The corresponding number of characters are read according to the standard placeholder width of the next group in the coding specification and recorded in a temporary variable. It is then determined whether the group identification identifier is equal to the identification identifier of the next group. If the above conditions are met, the decoding process of the next group of data is followed and the data is recorded in the next group of data. At the same time, the already decoded strings are masked and the remaining strings are checked for duplicates. Conversely, if effective data segmentation is not possible, this group of codes is marked as abnormal and masked. After masking, data is still read down and duplicate judgment is performed. Second, for code groups shorter than the standard width, determine whether it is a case of cross-line misalignment or an abnormal case of insufficient code bits. The determination is made based on the total width of the current code group and the next code group, as well as the first character of the code. If the sum of the lengths of the two code groups equals the total width of the code under the standard condition, and the group identifier of the first group is also the same as under the standard condition, then it is determined to be a case of cross-line misalignment. In this case, a separate identifier is set to mark the beginning and end of the cross-line data and perform concatenation processing, and the complete record obtained after concatenation is decoded. If the cross-line reporting condition is not met, then it is a case of insufficient code bits, and it is considered an invalid code, with the skip code pointing to the next position to be traversed.

2. The GTS data anomaly analysis method according to claim 1, characterized in that, The method for reading and parsing the aforementioned data files includes: The method involves first cutting and reading the entire announcement and then performing layered decoding. This means identifying a complete announcement based on the start line identifier and end information identifier, then dividing the different parts of the announcement into layers, and performing decoding processing on each layer.

3. The GTS data anomaly analysis method according to claim 1, characterized in that, The methods for judging and parsing abnormal conditions in the second-level encoding include: Read the second-level code, using the report identifier as the start marker and the announcement content end marker as the end marker; If the end marker of the announcement content is read normally at the end of the editing group, proceed with the normal decoding process; If the first condition is not met, continue reading downwards and determine whether the next set of codes is the end marker of the announcement content. If it is, then the announcement ends with that marker. If neither of the first two conditions is met, continue reading downwards and determine whether the next set of codes is the report identifier for the next announcement. If it is, stop decoding the current announcement and enter the decoding loop for the next announcement. If none of the above conditions are met, look for the end marker of the entire announcement and stop decoding.

4. The GTS data anomaly analysis method according to claim 1, characterized in that, The method for judging abnormal conditions of the third-level encoding and the fourth-level encoding includes: Using contextual logic, anomaly detection is performed at the third and fourth levels. When the previous paragraph lacks required content, but the next paragraph's content is directly reported on the same line, and the next paragraph's segment indicator code does not conflict with the missing content in the previous paragraph, when parsing the previous paragraph's group, a character check is first added to the placeholder part of the group identifier code to determine if it overlaps with the segment identifier code of the following paragraph. If overlap occurs, the logical relationship between the next paragraph's segment identifier code and its element code is first used for judgment. If it conforms to the next paragraph's logical rules, it is determined to be an abnormal encoding case, and the decoding process of the previous paragraph is skipped, and the decoding of the next paragraph is performed. When multiple required paragraphs are consecutively missing, the segment identifier codes of the following multiple paragraphs are added simultaneously during overlap detection. When the necessary paragraph or group identification markers are missing, conditional judgment is made first based on the logical relationship between the upper and lower groups of messages; secondly, for data groups without obvious contextual logical relationships, further judgment is made based on the announcement content end marker; when there is a situation where the ship call sign or station identification group, the announcement content end marker, and the next announcement content are incorrectly reported in the same group, the data is further segmented according to the end marker, and the content of the third-level code and the fourth-level code is extracted.

5. The GTS data anomaly analysis method according to claim 1, characterized in that, The method for decoding the fourth-level encoding includes: For encoding variable group number and variable placeholder width, when the corresponding paragraph identifier and internal group identification identifier are read, firstly, if there is a line break in the data reporting, the data is integrated according to the line break reporting conditions, and a separate identifier is set to mark and splice the line break data. When splicing, the element base placeholder width, the continuity and consistency of the preceding and following group codes are used as the standard. Secondly, based on the logical relationship between the preceding and following group codes, each complete set of matching records is extracted cyclically using the group identification identifier as a marker and stored in the corresponding data space; during the cyclic extraction process, the segment identification identifier of the following segment is set as a stop prompt for the cyclic extraction. Third, for each complete set of matching records stored in the data space, find the group identification identifier, select the group decoding strategy and value conversion algorithm according to the group identification identifier, calculate the value, and store the same set of matching records in different data members of the same data structure.

6. The GTS data anomaly parsing method according to claim 1, characterized in that, Extract the parsed time group from the message to obtain the observation day, hour, minute and last digit of the year, obtain the system generation time of the data file, and extract the complete year, month and day; A reliable interval is constructed based on the file generation time and the maximum business backtracking cycle. Possible years are calculated based on the observation day, hour, minute and last digit of the year of the message. The reliable interval year is obtained by filtering the possible years through the reliable interval. The last digit of the year in the message is concatenated with the first three digits of the file generation time. If the concatenated year is less than or equal to the file generation year, the concatenated result is taken as the first candidate time. If the concatenated year is greater than the year the file was generated, then subtract 10 years from the concatenated year and use it as the second candidate time. If the spliced ​​observation time is greater than the file generation time, the month is reduced by 1 and the comparison is repeated until it is within the confidence interval, at which point the third candidate time is output. The first, second, and third candidate times are used as candidate sets. If the current message and the previous message come from the same sending organization, and if the times are not consecutive, the candidate year that is closer to the time of the previous message is selected as the candidate year. If the message contains a month, and the month is not within the most recent 3 months of the file generation time, then the candidate year is used as the adjustment year; For high-frequency reporting agencies, if the time jump is greater than 24 hours, it is marked as abnormal and the candidate years are reviewed to output a reasonable time.

7. A GTS data anomaly analysis system, used to perform the method according to any one of claims 1-6, characterized in that, include: The receiving and classification storage module is used to receive real-time GTS messages, integrate the raw data in hours, cut the GTS messages into complete messages according to the start and end identifiers of the announcement, determine the message type of the complete message according to the report identifier, classify and store the data according to the data type to obtain similar data files, and read and parse the similar data files. The layered encoding module is used to read the complete announcement using a whole-code segmentation method, divide the complete announcement into four layers and decode them to obtain the layered encoding. The first layer includes the starting line of the report and the corresponding abbreviated title and information end marker. The second layer includes the content of each announcement, also known as each observation report. An observation report is distinguished by a report identifier as the start marker and an announcement content end marker as the end marker. The third level contains each paragraph of each observation report, and each paragraph consists of several fourth levels, with each group reporting one observation element; the fourth level contains each group within each paragraph; the hierarchical coding includes first-level coding, second-level coding, third-level coding, and fourth-level coding; Anomaly detection and processing module: used to perform anomaly condition judgment on the first level code, perform anomaly condition judgment and parsing on the second level code, perform anomaly condition judgment on the third level code and the fourth level code, and process the third level code and the fourth level code; Adjust the decoding record module: integrate the last digit of the year in the message with the file generation time, obtain a reasonable time by comparison and adjustment, decode the fourth level encoding, and record the exception details to the log file.

Images