A draft data anomaly detection method for ship cargo capacity prediction

Abstract

This invention belongs to the field of shipping big data analysis technology, specifically relating to a method for detecting anomalies in draft data for ship cargo volume prediction. The method includes: collecting data related to ship cargo volume and draft, and preprocessing the data; establishing a Cartesian coordinate system based on IMO navigation safety requirements and the collected data, with ship draft as the x-axis and ship cargo volume as the y-axis, and generating quadrilateral normal data boundaries within the coordinate system; data corresponding to ship cargo volume and draft outside the quadrilateral are considered anomaly data; otherwise, they are considered normal data. The method provided by this invention follows the physical correlation between ship draft and cargo volume, ensuring that ship draft and cargo volume data maintain logical accuracy and positive correlation according to basic physical logic. The method provided by this invention can accurately and quickly identify anomalies in different ship cargo volume and draft data within the dataset.

Description

Technical Field

[0001] This invention belongs to the field of shipping big data analysis technology, specifically relating to a method for detecting anomalies in draft data for ship cargo volume prediction. Background Technology

[0002] Ship cargo load and draft data are core input parameters for machine learning prediction of ship cargo load, and their data quality directly affects the model's prediction accuracy and generalization ability. However, anomalies in cargo load and draft data are common due to sensor errors (such as draft measurement drift and load cell calibration deviation), data entry errors (such as unit confusion and spatiotemporal misalignment), or improper operations, severely compromising data consistency. Although various data anomaly detection methods exist (such as statistical model anomaly detection methods, unsupervised machine learning algorithm anomaly detection methods, and manual screening) for assessing the cargo load-draft relationship, these methods still have significant limitations in handling domain-specific issues related to ship cargo load and draft data, severely restricting data quality optimization and improvement of model prediction accuracy.

[0003] (1) The core problem of anomaly detection in statistical models (such as IQR and Z test): First, only a threshold is set for a single dimension (such as analyzing draft or cargo volume alone), but the relationship between cargo volume and draft needs to be verified by multiple dimensions. Second, a fixed threshold is set based on statistical assumptions (normal distribution), but the cargo volume-draft data of ships presents a multimodal or skewed distribution due to differences in ship type, cargo type, etc., which leads to frequent misjudgments in the detection of cargo volume-draft data of ships, and it is impossible to verify whether the data conforms to the physical laws of ships.

[0004] (2) Core problem of anomaly detection in unsupervised machine learning algorithms (such as Isolation Forest): Although Isolation Forest can handle multidimensional data and isolate outliers by randomly dividing the feature space, ship data naturally presents a multimodal distribution due to differences in voyage tasks (full load, empty load, half load), which leads to the misjudgment of heavy cargo loading under normal working conditions as anomalies.

[0005] (3) In addition, although manual screening can partially identify explicit errors in the data, it is inefficient and difficult to meet the real-time requirements of large-scale data streams. Summary of the Invention

[0006] This invention addresses the problems of existing draft data anomaly detection methods, which only verify the single-dimensional data distribution and ignore the physical correlation between the two, leading to frequent misjudgments, and the low efficiency of manual inspection, which is difficult to handle large-scale data streams. It proposes a draft data anomaly detection method for ship cargo volume prediction, thus solving the above problems.

[0007] The technical solution claimed by this invention is as follows:

[0008] A method for detecting draft data anomalies in ship cargo load prediction includes the following steps:

[0009] S1: Collect data related to the ship's cargo load and draft, and preprocess the data; the data includes dynamic data and static parameters;

[0010] S2: Based on IMO navigation safety requirements and the draft and cargo data collected in S1, a rectangular coordinate system is established with the ship's draft as the x-axis and the ship's cargo as the y-axis, and a quadrilateral normal data boundary is generated in the coordinate system; the four points of the quadrilateral are determined according to the maximum and minimum permissible draft when the ship is unloaded and the maximum and minimum permissible draft when the ship is loaded with the maximum cargo.

[0011] S3: Data corresponding to ship cargo load and draft outside the quadrilateral formed in S2 are abnormal data; otherwise, they are normal data.

[0012] Preferably, the dynamic data includes: the ship's real-time draft and the ship's actual cargo load; the static data includes: the ship's length between perpendiculars, the ship's identification code, the design draft, and the deadweight tonnage.

[0013] Preferably, the four points of the square are represented by B, C, D and E. Then the four points of the quadrilateral are: point B is the minimum permissible draft corresponding to the maximum cargo capacity of the ship: 90%-95% of the design draft; point C is the maximum permissible draft corresponding to the maximum cargo capacity of the ship: 105%-110% of the design draft; point E is the maximum permissible draft when unloaded; and point D is the point in the coordinate system corresponding to the minimum permissible draft when unloaded.

[0014] Preferably, the dynamic data is collected in real time through the Automatic Identification System (AIS) and draft sensors; the static parameters are obtained through the ship's technical files and ship inspection certificates.

[0015] Preferably, the preprocessing refers to standardizing the data format, specifically by unifying units, removing spatiotemporally missing data, and forming a structured dataset. The data in the structured dataset includes the following fields: ship identification number (MMSI), collection timestamp, length between perpendiculars (LBP), design draft (draught), deadweight tonnage (DWT), real-time midship draft (start_draught), and actual cargo load (realcargo). The spatiotemporally missing data refers to data without a timestamp or ship identification number.

[0016] Preferably, according to IMO navigation safety requirements, the basic requirements for draft of vessels sailing empty are as follows: For vessels with a length of 150 meters or less, when sailing empty, the minimum forward draft must be greater than or equal to 0.025 times their length, and the minimum midship draft must be greater than or equal to 0.02 times their length + 2; For vessels with a length greater than 150 meters, when sailing empty, the minimum forward draft must be greater than or equal to 0.12 times their length + 2, and the minimum midship draft must be greater than or equal to 0.015-0.025 times their length + 2; the minimum permissible draft when empty in S2 corresponds to the minimum midship draft; the length refers to the hull length.

[0017] Preferably, the maximum allowable draft in S2 under no-load conditions is the minimum allowable draft under no-load conditions plus 20% of the design draft.

[0018] Preferably, the maximum deadweight tonnage in S2 is obtained directly through matching the ship identification number.

[0019] Preferably, the specific steps of S3 are as follows:

[0020] S31: Extract the current ship's dynamic parameters and dynamic data for each piece of data obtained from preprocessing;

[0021] S32: Calculate the minimum permissible draft when unloaded based on the length between the ship's perpendiculars, and determine the coordinates of the four vertices of the quadrilateral based on the minimum permissible draft and the design draft;

[0022] S33: Determine whether the current data point of the ship is within the quadrilateral based on dynamic data;

[0023] S34: If the data point corresponding to the dynamic data is within the quadrilateral, it is considered a normal data point; otherwise, the corresponding data point is considered abnormal.

[0024] Beneficial effects

[0025] This invention provides a method for detecting anomalies in draft data for ship cargo volume prediction. Based on IMO navigation safety requirements and data collected in step S1, a Cartesian coordinate system is established with ship draft as the x-axis and ship cargo volume as the y-axis. A quadrilateral normal data boundary is generated within this coordinate system. Following the physical correlation between ship draft and cargo volume, and adhering to basic physical logic, the normal data boundary is defined by combining ship static parameters and IMO navigation safety requirements. This ensures that ship draft and cargo volume data remain logically accurate and positively correlated, solving the problem of frequent misjudgments in existing draft data anomaly detection methods that only verify single-dimensional data distribution while ignoring the physical correlation between the two. The method provided by this invention can accurately and quickly identify anomalies in different ship cargo volume and draft data within a dataset using dynamically collected data, overcoming the problem of low efficiency and inability to handle large-scale data streams with manual inspection in existing draft data anomaly detection methods.

[0026] This invention integrates ship structural characteristics and navigation safety regulations to construct a logical verification method for cargo load and draft data. This method solves the problem of abnormal cargo load and draft data caused by sensor errors, data entry errors, or violations of regulations. It ensures that the ship's cargo load and draft data conform to the basic logic of Archimedes' principle, optimizes the quality of the dataset, makes the data better reflect the relationship between ship cargo load and draft, improves the prediction accuracy of machine learning models, and provides a highly reliable training dataset for machine learning models such as cargo load prediction. Attached Figure Description

[0027] Figure 1 This is a flowchart of a method for detecting anomalies in draft data for ship cargo volume prediction in an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of the theoretical relationship between ship cargo capacity and draft, and a schematic diagram of boundary division in an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram illustrating the processing of abnormal global ship cargo load and draft data in an embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram showing the abnormal data processing of ship cargo load and draft in an embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram illustrating the ship's cargo load and draft anomaly identification effect in an embodiment of the present invention. Specific implementation methods

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions will be further described clearly and completely below with reference to the accompanying drawings.

[0033] This invention provides a method for detecting anomalies in draft data for ship cargo load prediction, such as... Figure 1 As shown, it includes the following steps:

[0034] S1: Collect data related to the ship's cargo load and draft, and preprocess the data; the data includes dynamic data and static parameters; in a specific embodiment of the present invention, the static parameters are obtained through authoritative channels such as ship technical files and ship inspection certificates, including the ship identification code (MMSI), length between perpendiculars (LBP), design draft (draught), and deadweight tonnage (DWT), with the data accuracy required to retain one decimal place; the dynamic data are collected in real time through equipment such as the Automatic Identification System (AIS) and draft sensors, including real-time midship draft (start_draught) and actual cargo load (realcargo), ensuring spatiotemporal consistency (i.e., the draft and cargo load data at the same timestamp correspond one-to-one, avoiding data misalignment across voyage segments); the above data covers the core data related to ship cargo load prediction, ensuring the accuracy of subsequent data detection.

[0035] The data preprocessing standardizes the format of the collected data, such as unifying units (converting draft to meters and cargo capacity to tons) and removing spatiotemporally missing data (data without timestamps or ship identification numbers MMSI), forming a structured dataset with fields including MMSI, collection timestamp, LBP, draught, DWT, start_draught, and realcargo.

[0036] S2: Based on IMO navigation safety requirements and the draft and cargo volume data collected in step S1, a rectangular coordinate system is established with the ship's draft as the x-axis and the ship's cargo volume as the y-axis. A quadrilateral normal data boundary is generated within this coordinate system. The four points of the quadrilateral are determined based on the maximum and minimum permissible draft when unloaded, and the maximum and minimum permissible draft when the ship has the maximum cargo volume. The four points of the square are: the minimum permissible draft corresponding to the ship's maximum cargo volume, the maximum permissible draft corresponding to the ship's maximum cargo volume, the maximum permissible draft when unloaded, and the minimum permissible draft when unloaded, corresponding to these values ​​in the coordinate system. The minimum permissible draft corresponding to the ship's maximum cargo volume is 90%-95% of the design draft; the maximum permissible draft corresponding to the ship's maximum cargo volume is 105%-110% of the design draft. In a specific embodiment of this invention, the maximum deadweight tonnage of each ship is recorded in the ship's file and directly obtained through matching the ship identification code (MMSI).

[0037] In a specific embodiment of the present invention, based on IMO navigation safety requirements and ship physical characteristics, the BCDE quadrilateral normal data boundary is generated through the following formulas and steps, with specific parameters calculated as follows:

[0038] 1. Calculation of minimum draft parameters

[0039] The minimum bow draft (dFmin) and minimum midship draft (dMmin) are calculated in two scenarios based on the ship's length between perpendiculars (LBP):

[0040] When LBP≤150m: dFmin≥0.025×LBP (m), dMmin≥0.02×LBP + 2 (m);

[0041] When LBP>150m: dFmin≥0.012×LBP + 2 (m), dMmin≥0.02×LBP + 2 (m);

[0042] For example, for a bulk carrier with LBP=180m (>150m), then dFmin≥0.012×180 + 2=4.16m, dMmin≥0.02×180 + 2=5.6m.

[0043] 2. Determining the coordinates of the vertices of quadrilateral BCDE

[0044] Based on the ship's static parameters and the above calculation results, the coordinates (draft, cargo capacity) of the four vertices of the quadrilateral are determined as follows:

[0045] Point B: (0.9 × draft, DWT), representing the maximum cargo capacity at 90% design draft (lower limit of full load).

[0046] Point C: (1.1×draught, DWT), representing the maximum cargo capacity at 110% of the design draft (the upper limit of full load, taking into account reasonable draft fluctuations).

[0047] Point D (dMmin, 0) represents the minimum permissible draft when unloaded (cargo load is 0).

[0048] Point E: (dMmin + 0.2×draught, 0), represents the maximum permissible draft when unloaded (cargo load is 0, and the ballast water is considered for reasonable adjustment).

[0049] For example: A container ship has a design draft of draught = 12m, DWT = 50000t, and dMmin = 5.6m. Then, the quadrilateral BCDE formed by points B (10.8m, 50000t), C (13.2m, 50000t), D (5.6m, 0t), and E (5.6 + 2.4 = 8.0m, 0t) represents the ship's normal data boundaries. Figure 2 ).

[0050] S3: Data for ship cargo load and draft outside the quadrilateral formed in S2 are considered abnormal; otherwise, they are considered normal. The specific steps include:

[0051] S31: Extract the dynamic parameters and dynamic data of the current ship for each piece of data obtained through preprocessing;

[0052] S32: Calculate the minimum allowable draft when the ship is空载 and determine the four vertex coordinates of the quadrilateral based on the minimum allowable draft and the designed draft;

[0053] S33: Determine whether the data point corresponding to the current ship in the dynamic data is within the quadrilateral;

[0054] S34: If the data point corresponding to the dynamic data is within the quadrilateral, it is judged that the data point is normal; otherwise, the corresponding data point is abnormal.

[0055] In a specific embodiment of the present invention, the following verification logic is executed for each piece of preprocessed data record (including fields such as MMSI, start_draught, realcargo, etc.): (1) Extract the static parameters (LBP, draught, DWT) and dynamic data (start_draught, realcargo) of the current ship; (2) Calculate dMmin based on LBP and determine the four vertex coordinates of the BCDE quadrilateral; (3) Judge whether the data point (start_draught, realcargo) satisfies "being within (including the boundary) the BCDE quadrilateral":

[0056] Draft dimension: When start_draught ≥ the abscissa of point D (dMmin) and start_draught ≤ the abscissa of point E (dMmin + 0.2draught), realcargo should = 0 (空载 condition);

[0057] Draft dimension: When start_draught > the abscissa of point E (dMmin + 0.2draught) and start_draught ≤ the abscissa of point B (0.9draught), realcargo should be within the interval (0, DWT], and show a positive correlation trend with the increase of the draft;

[0058] Draft dimension: When start_draught > the abscissa of point B (0.9draught) and start_draught ≤ the abscissa of point C (1.1draught), realcargo should = DWT (fully loaded condition);

[0059] Draft dimension: When start_draught < the abscissa of point C (1.1draught) or start_draught > the abscissa of point D (dMmin), it is directly determined as abnormal;

[0060] (4) Output verification results: If the above conditions are met, it is marked as "normal data"; otherwise, it is marked as "abnormal data" and the abnormality type is recorded (such as "excessive draft when empty", "insufficient draft when fully loaded", "negative correlation between cargo load and draft").

[0061] The above verification logic can be used for batch data processing: For historical datasets (such as one year's voyage data for a certain ship), group by ship MMSI, execute the above verification logic sequentially, and output the normal / abnormal data marking results in batches, such as... Figure 5 The comparison between the "All Data Scatter Plot" and the "Scatter Plot Based on Whether It Is Normal" shows that outliers deviating from the BCDE quadrilateral can be intuitively filtered out (for example, in the ship data with MMSI=477439400, the point with a draft of 12m but a cargo capacity of 200,000t is judged as abnormal, while the point with a draft of 16m and a cargo capacity of 150,000t is judged as normal). In the specific embodiment of the present invention, the BCDE quadrilateral is essentially the draft boundary range determined for each ship based on its empty and full load and draft combined with the design draft. It is essentially a numerical range, so abnormal data for each ship can be determined based on this rule.

[0062] In a specific embodiment of the present invention, the effectiveness of the above method is verified as follows:

[0063] The verification results are visually displayed using a scatter plot: Using "real-time draft (x-axis) - actual cargo load (y-axis)" as the coordinates, all data points are plotted against the boundary of the BCDE quadrilateral. Normal data points are concentrated inside the quadrilateral, while abnormal data points are distributed outside the quadrilateral (e.g., ...). Figure 4 As shown in the image, blue dots represent normal data, and red dots represent abnormal data.

[0064] In a specific embodiment of the present invention, the draft fluctuation coefficients: the coefficients (0.9, 1.1) at point B (0.9×draught) and point C (1.1×draught) can be adjusted according to the type of vessel. For example, for container ships, due to the uniformity of cargo loading, the coefficients can be adjusted to 0.95 and 1.05, while for bulk carriers, due to the large differences in loading, the coefficients remain at 0.9 and 1.1.

[0065] Unloaded draft limit coefficient: 0.2 in point E (dMmin+0.2×draught) can be adjusted according to the navigation area. It is adjusted to 0.15 for coastal vessels and 0.25 for ocean-going vessels to adapt to the wind and wave environment requirements of different sea areas.

[0066] In a specific embodiment of the present invention, records marked as "abnormal data" need to be processed according to the type of abnormality: sensor drift-related abnormalities require equipment calibration, data entry errors require data correction, and violations of operation require verification of voyage logs; discrete points in normal datasets (such as draft-cargo fluctuations under half-load conditions) do not need to be removed because they conform to the ship loading logic and can be directly used for machine learning model training.

[0067] Through the above specific implementation methods, the problems of single-dimensional verification, fixed threshold misjudgment, and low efficiency in the existing technology can be systematically solved, ensuring that the cargo volume-draft data conforms to physical logic and navigation specifications, and providing high-quality training data for subsequent cargo volume prediction models.

[0068] The method provided by this invention is applicable to the quality verification of voyage cargo load-draft data for various merchant ships (including bulk carriers, oil tankers, etc.), covering all operating conditions such as empty, half-loaded, and fully loaded navigation scenarios. It supports batch processing of historical data from a single ship and parallel detection of real-time data streams from multiple ships, and is adaptable to different regulatory requirements for domestic and international vessels (such as...). Figure 3 As shown, blue represents domestic ship data and red represents international ship data. Both types of data can be effectively detected using the method provided by this invention.

[0069] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A method for detecting anomalies in draft data for ship cargo load prediction, characterized in that, Includes the following steps: S1: Collect data related to the ship's cargo load and draft, and preprocess the data; the data includes dynamic data and static parameters; S2: Based on IMO navigation safety requirements and the draft and cargo data collected in S1, a rectangular coordinate system is established with the ship's draft as the x-axis and the ship's cargo as the y-axis, and a quadrilateral normal data boundary is generated in the coordinate system; the four points of the quadrilateral are determined according to the maximum and minimum permissible draft when the ship is unloaded and the maximum and minimum permissible draft when the ship is loaded with the maximum cargo. S3: Data corresponding to ship cargo load and draft outside the quadrilateral formed in S2 are abnormal data; otherwise, they are normal data.

2. The method for detecting draft data anomalies for ship cargo load prediction according to claim 1, characterized in that, The dynamic data includes: the ship's real-time draft and the ship's actual cargo load; the static data includes: the ship's length between perpendiculars, the ship's identification code, the design draft, and the deadweight tonnage.

3. The method for detecting draft data anomalies for ship cargo load prediction according to claim 2, characterized in that, The four points of the square are represented by B, C, D, and E. Then the four points of the quadrilateral are: point B is the minimum permissible draft corresponding to the maximum cargo capacity of the ship: 90%-95% of the design draft; point C is the maximum permissible draft corresponding to the maximum cargo capacity of the ship: 105%-110% of the design draft; point E is the maximum permissible draft when unloaded; and point D is the point in the coordinate system corresponding to the minimum permissible draft when unloaded.

4. The method for detecting draft data anomalies for ship cargo load prediction according to claim 3, characterized in that, The dynamic data is collected in real time through the Automatic Identification System (AIS) and draft sensors; the static parameters are obtained through the ship's technical files and ship inspection certificates.

5. The method for detecting draft data anomalies for ship cargo load prediction according to any one of claims 1-4, characterized in that, The preprocessing refers to the standardization of the data format, specifically: unifying units, removing spatiotemporally missing data, and forming a structured dataset. The data in the structured dataset includes the following fields: ship identification number (MMSI), collection timestamp, length between perpendiculars (LBP), design draft (draught), deadweight tonnage (DWT), real-time midship draft (start_draught), and actual cargo load (realcargo). The spatiotemporally missing data refers to data without a timestamp or ship identification number.

6. The method for detecting draft data anomalies for ship cargo load prediction according to claim 5, characterized in that, According to IMO navigation safety requirements, the basic requirements for draft of vessels sailing empty are as follows: For vessels with a length of 150 meters or less, when sailing empty, the minimum forward draft must be greater than or equal to 0.025 times the vessel's length, and the minimum midship draft must be greater than or equal to 0.02 times the vessel's length plus 2; For vessels with a length greater than 150 meters, when sailing empty, the minimum forward draft must be greater than or equal to 0.12 times the vessel's length plus 2, and the minimum midship draft must be greater than or equal to 0.015-0.025 times the vessel's length plus 2; the minimum permissible draft when empty in S2 corresponds to the minimum midship draft; the vessel length refers to the hull length.

7. The method for detecting draft data anomalies for ship cargo load prediction according to claim 6, characterized in that, The maximum permissible draft under no-load conditions in S2 is the minimum permissible draft under no-load conditions plus 20% of the design draft.

8. The method for detecting draft data anomalies for ship cargo load prediction according to claim 7, characterized in that, The maximum deadweight tonnage in S2 is obtained directly by matching the ship identification number.

9. The method for detecting draft data anomalies for ship cargo load prediction according to claim 8, characterized in that, The specific steps for S3 are as follows: S31: Extract the current ship's dynamic parameters and dynamic data for each piece of data obtained from preprocessing; S32: Calculate the minimum permissible draft when unloaded based on the length between the ship's perpendiculars, and determine the coordinates of the four vertices of the quadrilateral based on the minimum permissible draft and the design draft; S33: Determine whether the current data point of the ship is within the quadrilateral based on dynamic data; S34: If the data point corresponding to the dynamic data is within the quadrilateral, it is considered a normal data point; otherwise, the corresponding data point is considered abnormal.

Images