Data quality evaluation and anomaly monitoring method based on trusted data space

Abstract

The invention discloses a data quality evaluation and anomaly monitoring method based on a trusted data space, and the method comprises the following specific steps: S1, obtaining original data of an AIS system, and carrying out the analysis and classification of the original data, and obtaining dynamic data and static data; s2, constructing a quality evaluation module, wherein the quality evaluation module comprises an abnormal value detection unit, a missing value filling unit and an error value correction unit; s3, dynamic data are cleaned through an abnormal value detection unit, a missing value filling unit and an error value correction unit; static data are cleaned through an abnormal value detection unit and an error value correction unit; and S4, generating a visual monitoring chart according to the data counted in the step S3, and generating a monitoring report according to the template. According to the invention, the quality and reliability of AIS data are improved, real-time monitoring and statistical analysis are carried out on abnormal data, and powerful support is provided for marine traffic management, ship operation safety and efficiency improvement.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a method for data quality assessment and anomaly monitoring based on a trusted data space. Background Art

[0002] The Automatic Identification System for Ships, abbreviated as AIS system, refers to a new type of navigation aid system applied to maritime safety and communication between ships and the shore, and between ships. It is usually composed of a VHF communicator, a GPS locator, and a communication controller connected to an on-board display and sensors, etc., and can automatically exchange important information such as ship position, speed, course, ship name, call sign, etc. The AIS installed on the ship sends out this information while receiving the information of other ships within the VHF coverage range at the same time, thus realizing automatic response. In addition, as an open data transmission system, it can be connected to terminal devices such as radar, ARPA, ECDIS, VTS, and INTERNET to form a maritime traffic management and monitoring network, which is an effective means to obtain traffic information without using radar detection and can effectively reduce ship collision accidents.

[0003] The AIS system at sea has problems of data loss and anomalies during the data transmission process under specific circumstances. Therefore, it is necessary to design a method for data quality assessment and anomaly monitoring, aiming to comprehensively evaluate the integrity, accuracy, consistency, and timeliness of AIS data. Through advanced technologies such as missing value replacement rule algorithms and anomaly identification rule algorithms, the quality and reliability of AIS data are improved, and real-time monitoring and statistical analysis of abnormal data are carried out to provide strong support for maritime traffic management, ship operation safety, and efficiency improvement. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for data quality assessment and anomaly monitoring based on a trusted data space.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] A method for data quality assessment and anomaly monitoring based on a trusted data space includes the following specific steps:

[0007] S1. Obtain the original data of the AIS system, parse and classify the original data to obtain dynamic data and static data;

[0008] S2. Construct a quality assessment module, and the quality assessment module includes an outlier detection unit, a missing value filling unit, and an error value correction unit;

[0009] S3. Clean the dynamic data through the outlier detection unit, missing value filling unit, and error value correction unit respectively, detect and identify the outliers, missing values, and error values in the data, and repair them after statistics; clean the static data through the outlier detection unit and error value correction unit respectively, detect and identify the outliers and error values in the data, and repair them after statistics.

[0010] S4. Calculate the outlier rate and missing rate of the dynamic data based on the data statistics in S3, calculate the outlier rate of the static data, record the fields where each outlier, missing value, and error value is located, generate a visual monitoring chart for the statistical results of the outlier rate, missing rate, and the fields where they are located, and generate a monitoring report according to the template.

[0011] Further, the specific steps of step S1 are as follows:

[0012] Obtain the original data of the AIS system, conduct a comprehensive analysis and conversion of the original data. First, extract the message content from the obtained original data, match and classify the message content according to dynamic or static types, then, according to the parsing rules, parse and translate each field of the message content one by one, and then convert it into a structured format. Finally, obtain the plaintext data including dynamic AIS messages and static AIS messages.

[0013] Classify the plaintext data to obtain dynamic data and static data.

[0014] Further, the specific steps of step S1 also include:

[0015] The parsing of the message content is specifically to parse out each field of the message content one by one according to its start symbol, length, and data type, and extract the meaning and value of each field.

[0016] Verify the integrity of the data through the CRC check algorithm.

[0017] Convert the analyzed and translated data into a structured format, store it at the specified path, and generate an information table for the corresponding ship according to the information table template and the structured format.

[0018] Further, the dynamic data includes ship position, course, speed, and track, and the static information includes ship name, MMSI, IMO, call sign, ship length, ship width, and ship depth.

[0019] Further, the specific steps of step S2 are as follows:

[0020] The outlier detection unit identifies the data beyond the range in the dynamic data or static data and defines it as an outlier, and statistics the mean, median, total amount of data, and total amount of outliers in the dynamic data of the current ship.

[0021] The missing value filling unit identifies the missing data values in the dynamic data, defines them as missing values, fills the mean, median or preset default value into the positions where the dynamic data is missing, and counts the total amount of missing data;

[0022] The error value correction unit identifies the duplicate or conflicting data values in the dynamic data or static data, defines them as error values, modifies or deletes the corresponding error values, and counts the total amount of error values.

[0023] Furthermore, the outlier detection unit, the missing value filling unit and the error value correction unit are all equipped with machine learning algorithms and statistical algorithms. According to the machine learning algorithms, the outlier, missing value or error value in the dynamic data or static data is identified and learned. According to the statistical algorithms, the total amount of detected data and the corresponding total detection amount are counted.

[0024] Furthermore, the specific steps of step S4 are as follows:

[0025] S41. According to the detection data respectively counted by the outlier detection unit, the missing value filling unit and the error value correction unit in S3;

[0026] S42. Classify the detection data according to the dynamic data and the static data, calculate the proportion of the total amount of detected data and the total amount of data whose type belongs to the dynamic data to obtain the outlier rate and the missing rate, and calculate the proportion of the total amount of detected data and the total amount of data belonging to the static data to obtain the outlier rate;

[0027] S43. Record the types, total amounts and their corresponding fields of each outlier, missing value and error value. Combine the outlier rate and the missing rate calculated in S42, and fill each part of the data into the monitoring chart template according to the monitoring chart template to obtain a visual monitoring chart;

[0028] S44. Generate a monitoring report according to the report template, combining the data in S42, the data in S43 and the visual monitoring chart.

[0029] After adopting the above technical solution, compared with the background technology, the present invention has the following advantages:

[0030] After obtaining the original data of the AIS system, the present invention analyzes and classifies it to obtain dynamic data and static data. Subsequently, a quality evaluation module is constructed, and the quality evaluation module is used to detect and evaluate the dynamic data and the static data, intelligently detect outliers, fill in the missing missing values and correct the incorrect data, so that the data has integrity and consistency, improves the quality and reliability of the obtained AIS system data, and monitors and analyzes in real time during the acquisition process to avoid data anomalies, missing and errors caused by specific situations, and can perform statistics and generate visual charts and reports, which is convenient for the regulatory authorities to use and provides strong support for maritime traffic management, ship operation safety and efficiency. Brief Description of the Drawings

[0031] Figure 1 This is a flowchart of the method of the present invention. Detailed Embodiment

[0032] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] It should be noted that in the present invention, terms such as "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are all based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the devices or elements of the present invention must have a specific orientation. Therefore, it should not be construed as a limitation to the present invention.

[0034] Embodiment

[0035] Referring to Figure 1 as shown, the present invention discloses a method for data quality evaluation and anomaly monitoring based on a trusted data space, including the following specific steps:

[0036] S1. Obtain the original data of the AIS system, parse and classify the original data to obtain dynamic data and static data.

[0037] S2. Construct a quality evaluation module, which includes an outlier detection unit, a missing value filling unit, and an error value correction unit.

[0038] S3. Clean the dynamic data through the outlier detection unit, the missing value filling unit, and the error value correction unit respectively, detect and identify the outliers, missing values, and error values in the data, count them and repair them; clean the static data through the outlier detection unit and the error value correction unit respectively, detect and identify the outliers and error values in the data, count them and repair them.

[0039] The detection of dynamic data is mainly responsible for cleaning the message fields of dynamic data. Dynamic messages refer to the dynamic data sent by AIS devices in real time, including information such as the position, heading, and speed of ships. The main purpose of this cleaning component is to remove outliers, fill in missing values, correct error values, etc. to ensure the accuracy and integrity of the data. For example, if the speed of a certain ship suddenly appears abnormal, this component will identify and correct it.

[0040] The detection of data also adopts consistency detection technology and data correction technology. The consistency detection technology is to detect the consistency of data through comparison and matching technologies, such as the uniqueness of ship names and whether the ship length is within a reasonable range. The data correction technology is to correct and fill in the data that does not meet the requirements, for example, using default values or filling in through calculation strategies.

[0041] S4. According to the data counted in S3, calculate the abnormality rate and missing rate of dynamic data, calculate the abnormality rate of static data, record the fields where each abnormal value, missing value and error value are located, generate a visual monitoring chart for the statistical results of the abnormality rate, missing rate and the fields where they are located, and generate a monitoring report according to the template.

[0042] The quality assessment function for AIS system data in this embodiment refers to comprehensively assessing and analyzing the parsed AIS data, including aspects such as data integrity, accuracy, consistency and timeliness. This function can be configured for each field of AIS, set monitoring rules for missing value replacement and abnormality identification, and perform rule replacement according to the detection rules, so as to improve the quality and reliability of the data.

[0043] Among them, the parsing algorithm is to develop a parsing algorithm according to the format and specifications of AIS system data to extract the meaning and value of each field. The verification technology is to use the CRC verification algorithm to verify the data to ensure the integrity and accuracy of the data. The data conversion technology is to convert the format of the parsed data for subsequent data analysis and processing.

[0044] The specific steps of step S1 are as follows:

[0045] Obtain the original data of the AIS system, and perform comprehensive parsing and conversion on the original data. First, extract the message content from the obtained original data, match and classify the message content according to dynamic or static types, then parse and translate each field of the message content according to the parsing rules, and then convert it into a structured format. Finally, obtain the plaintext data including dynamic AIS messages and static AIS messages;

[0046] Classify the plaintext data to obtain dynamic data and static data.

[0047] The specific steps of step S1 also include:

[0048] The parsing of the message content is specifically to parse out each field of the message content one by one according to its start symbol, length and data type, and extract the meaning and value of each field;

[0049] Verify the integrity of the data through the CRC verification algorithm;

[0050] Convert the analyzed and translated data into a structured format and store it at the specified path. Generate an information sheet for the corresponding ship according to the information sheet template and the structured format.

[0051] The dynamic data includes ship position, course, speed, and track, and the static information includes ship name, MMSI, IMO, call sign, ship length, ship width, and ship depth.

[0052] The specific steps of step S2 are as follows:

[0053] The outlier detection unit identifies data outside the range in the dynamic data or static data and defines it as an outlier, and counts the mean, median, total data volume, and total outlier volume in the dynamic data of the current ship. The outlier detection technology in this embodiment: Through statistical algorithms and machine learning algorithms, detect data that does not conform to the conventional range, such as outliers where the acceleration suddenly becomes negative or the speed is much higher than the normal range.

[0054] The missing value filling unit identifies the missing data values in the dynamic data and defines them as missing values, fills the mean, median, or preset default value into the positions where the dynamic data is missing, and counts the total missing data volume; Through statistical algorithms and machine learning algorithms, detect the data of the missing fields, such as the problems of missing destination and ETA.

[0055] The error value correction unit identifies duplicate or conflicting data values in the dynamic data or static data and defines them as error values, modifies or deletes the corresponding error values, and counts the total data volume of the error values.

[0056] The outlier detection unit, the missing value filling unit, and the error value correction unit are all equipped with machine learning algorithms and statistical algorithms. According to the machine learning algorithms, identify and learn the outliers, missing values, or error values in the dynamic data or static data, and according to the statistical algorithms, count the total volume of the detected data and the corresponding detection total volume.

[0057] The AIS anomaly rate function mainly monitors and analyzes the data of the AIS system in real time, discovers abnormal situations and conducts statistics. Use anomaly detection algorithms to monitor the data and discover abnormal situations. Display the abnormal situations through visualization technology for users to understand more intuitively. At the same time, conduct statistical analysis, calculate the anomaly rate and analyze the anomaly pattern to form an anomaly table. Finally, generate a report or statement according to the statistical analysis results for users to refer to and use. This function can improve the safety and efficiency of ship operation and help users better understand and master the abnormal patterns of ship behavior.

[0058] The AIS missing rate function performs statistical analysis on AIS data, calculates the missing rates of each field, and visually displays the missing situation. This helps users understand the integrity and quality of the data, discover data problems, and provide a reference for subsequent data processing and analysis. This function includes data preprocessing, missing rate statistics, visual display, anomaly detection, and report generation. Through the AIS missing rate function, users can quickly understand the integrity and quality of AIS data, providing an accurate and reliable data basis for data processing and analysis.

[0059] The specific steps of step S4 are as follows:

[0060] S41. According to the detection data respectively counted by the outlier detection unit, missing value filling unit, and error value correction unit in S3;

[0061] S42. Classify the detection data into dynamic data and static data, calculate the proportion of the total amount of detected dynamic data and the total amount of data to obtain the anomaly rate and missing rate, and calculate the proportion of the total amount of detected static data and the total amount of data to obtain the anomaly rate;

[0062] S43. Record the types, total amounts, and the fields where each outlier, missing value, and error value are located. Combine the anomaly rate and missing rate calculated in S42, and fill each part of the data into the monitoring chart template according to the monitoring chart template to obtain a visual monitoring chart;

[0063] S44. Generate a monitoring report according to the report template, combining the data in S42, the data in S43, and the visual monitoring chart.

[0064] After obtaining the original data of the AIS system in this embodiment, it is analyzed and classified to obtain dynamic data and static data. Subsequently, a quality assessment module is constructed, and the quality assessment module is used to detect and evaluate the dynamic data and static data, intelligently identify outliers, fill in missing values, and correct incorrect data, making the data complete and consistent, improving the quality and reliability of the obtained AIS system data, and performing real-time monitoring and analysis during the acquisition process to avoid data anomalies, missing, and errors caused by specific situations, and being able to perform statistics and generate visual charts and reports for the convenience of regulatory authorities to use, providing strong support for maritime traffic management, ship operation safety, and efficiency.

[0065] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A data quality assessment and anomaly monitoring method based on a trusted data space, characterized in that: The specific steps include: S1. Obtain the original data of the AIS system, parse and classify the original data, and obtain dynamic data and static data; S2. Construct a quality assessment module, which includes an outlier detection unit, a missing value filling unit, and an error value correction unit; S3, respectively clean the dynamic data through the outlier detection unit, the missing value filling unit and the error value correction unit, detect and identify the outliers, missing values ​​and error values ​​in the data, count and repair them; respectively clean the static data through the outlier detection unit and the error value correction unit, detect and identify the outliers and error values ​​in the data, count and repair them; S4. Based on the statistical data of S3, calculate the abnormality rate and missing rate of dynamic data, calculate the abnormality rate of static data, record the fields where each abnormal value, missing value and error value is located, generate a visual monitoring chart with the statistical results of the abnormality rate, missing rate and the field where they are located, and generate a monitoring report based on the template.

2. A data quality assessment and anomaly monitoring method based on a trusted data space as claimed in claim 1, characterized in that: The specific steps of step S1 are as follows: Obtain the original data of the AIS system, and perform comprehensive analysis and conversion on the original data. First, extract the message content from the obtained original data, match and classify the message content according to dynamic or static types, and then parse and translate the message content field by field according to the analysis rules, and then convert the structured format to finally obtain the plaintext data containing dynamic AIS messages and static AIS messages; Classify the plaintext data into dynamic data and static data.

3. A data quality assessment and anomaly monitoring method based on a trusted data space as claimed in claim 2, characterized in that: The specific steps of step S1 also include: The parsing of the message content is to parse out each field of the message content one by one according to its start character, length and data type, and extract the meaning and value of each field; Verify the integrity of the data through the CRC verification algorithm; The analyzed and translated data is converted into a structured format and stored in a specified path, and an information table of the corresponding ship is generated according to the information table template and the structured format.

4. The method for data quality assessment and anomaly monitoring based on a trusted data space as claimed in claim 3, characterized in that: The dynamic data includes the ship's position, heading, speed and track, and the static information includes the ship's name, MMSI, IMO, call sign, length, width and depth.

5. The data quality assessment and anomaly monitoring method based on a trusted data space as claimed in claim 4, characterized in that: The specific steps of step S2 are as follows: The outlier detection unit identifies out-of-range data in dynamic data or static data and defines it as an outlier, and counts the mean, median, total data and total outlier in the dynamic data of the current ship; The missing value filling unit identifies missing data values ​​in the dynamic data and defines them as missing values, fills the missing positions of the dynamic data with mean values, medians or preset default values, and counts the total amount of missing data; The error value correction unit identifies repeated or conflicting data values ​​in dynamic data or static data, defines them as error values, modifies or deletes corresponding error values, and counts the total amount of data of the error values.

6. The method for data quality assessment and anomaly monitoring based on a trusted data space according to claim 5, characterized in that: The outlier detection unit, missing value filling unit and error value correction unit are all equipped with a machine learning algorithm and a statistical algorithm. Outliers, missing values ​​or error values ​​in dynamic data or static data are identified and learned according to the machine learning algorithm, and the total amount of detection data and the corresponding detection total amount are counted according to the statistical algorithm.

7. The data quality assessment and anomaly monitoring method based on a trusted data space as claimed in claim 5, characterized in that: The specific steps of step S4 are as follows: S41, according to the detection data respectively counted by the abnormal value detection unit, the missing value filling unit and the error value correction unit in S3; S42, classifying the detection data according to dynamic data and static data, calculating the total detection amount and the proportion of the total data of the dynamic data type, obtaining the abnormality rate and the missing rate, calculating the total detection amount and the proportion of the total data of the static data type, obtaining the abnormality rate; S43, record the type, total amount and field of abnormal values, missing values ​​and error values, combine the abnormal rate and missing rate calculated in S42, fill each part of the data into the monitoring chart template according to the monitoring chart template, and obtain a visual monitoring chart; S44. Generate a monitoring report based on the report template and combining the data from S42, the data from S43 and the visual monitoring chart.

Images