A port container volume prediction analysis method based on AIS big data

Abstract

The present application relates to a kind of port container quantity prediction analysis method based on AIS big data, collate and collect ship AIS data, ship capacity data, ship schedule data and port container throughput data, extract the information data of all container ships in specific port each month;According to the name number of ship, correlate AIS ship track data, ship profile and ship schedule data, construct ship size, new features of time in port, generate ship information data feature table;The ship information data feature table is respectively input into neural network and gradient boosting tree, and port container quantity prediction model is obtained by training respectively;The neural network model and gradient boosting tree model are weighted and fused, and the final prediction model is generated, and the monthly port container quantity prediction result is obtained.The present application is based on multi-source data fusion and hybrid machine learning model, realizes the high-precision prediction of port throughput and provides guidance for the dynamic optimization of port production and operation strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a port container volume prediction analysis method based on AIS big data. BACKGROUND

[0002] At present, the analysis method for container port operation is mainly based on big data technology, and related data reflecting the operation capacity of the port is collected and arranged. Then, on the basis of processing and analyzing the data, the relevant theories of statistics and big data technology are used to solve various specific indexes reflecting the operation of the port.

[0003] The existing technology mainly has the following problems: first, the data island problem. The traditional prediction method does not effectively integrate multi-source data and actual operation records of the port, resulting in single feature dimension. For example: AIS data (Automatic Identification System) only contains basic information such as ship position and speed, and the actual capacity data lacks correlation analysis with the physical parameters of the ship. Second, the feature construction is rough. For example: the existing method uses a fixed threshold to divide the classification of ship size, without considering the following factors: the dynamic change of the actual loading rate of the ship along the route, and the nonlinear influence of the matching degree of the ship type and the port facilities. Third, the model adaptability is poor. For example: a single model is difficult to handle both structured data and time series data. SUMMARY

[0004] The present application designs a port container volume prediction analysis method based on AIS big data, which solves the technical problems of the existing port container volume prediction analysis method, such as data island, rough feature construction and poor model adaptability.

[0005] In order to solve the above-mentioned technical problems, the present application adopts the following scheme:

[0006] A method for predicting and analyzing port container volume based on AIS big data includes the following steps: S1. Organize and collect ship AIS data, ship capacity data, shipping schedule data, and port container throughput data; extract information data of all container ships in a specific port each month; and clean and verify the AIS data, including removing trajectory drift points and repairing missing fields; S2. Based on the ship name and number, associate AIS ship trajectory data, ship files, and shipping schedule data to construct new features such as ship size and berthing time, generating a ship information data feature table; S3. Input the feature table into a multilayer perceptron (MLP) neural network and an XGBoost gradient boosting tree for training, with the objective functions being minimizing the mean square error and optimizing the gradient boosting tree loss function, respectively, to obtain port container volume prediction models; S4. Based on the root mean square error (RMSE) of the MLP neural network model and the XGBoost gradient boosting tree model on the test set, calculate the weights in reverse proportion, and weightedly fuse the prediction results of the two models to generate the final port container volume prediction value. Calculate the RMSE of the fused model to verify its accuracy.

[0007] Preferably, step S1 includes cleaning and verifying AIS data:

[0008] Specifically, S101. Clean and verify the ship's AIS data, remove trajectory drift points and abnormal records, and repair missing data fields; S102. Use multi-source data fusion technology to interpolate, repair, and reconstruct missing data.

[0009] Preferably, S201. Ship dimensions are determined based on a dynamic threshold, and the calculation formula is as follows:

[0010] ;

[0011] in, To study the average rated TEU of ships over a time period, Standard deviation;

[0012] S202. Determine the ship berthing docks based on AIS latitude and longitude information, and count the number of ships berthed at each dock;

[0013] S203. The formula for calculating the total berthing time of a vessel is as follows:

[0014] ;

[0015] in, For the first Total berthing time for this type of vessel The value can be either large, medium, or small (L, M, S) sizes. For each ship The time when this type of vessel leaves its berth. For each ship The time it takes for a vessel of this type to enter the berth; n is the size of the sliding window.

[0016] Step S2 further includes: introducing a dynamic sliding window, using the historical data mean as a new feature, calculated using the following formula:

[0017] ;

[0018] Where k is the size of the sliding window. New features generated for the data in month d. For the dth month ago Monthly port container throughput;

[0019] Preferably, step S2 further includes AIS data association and integration: based on the ship name and number, the AIS ship trajectory data is associated with the ship file and the ship schedule data to obtain the relevant characteristic attributes of each ship, specifically: ship latitude and longitude, data collection time, sampling frequency, ship type, actual carrying capacity and ship operator information.

[0020] Preferably, step S2 further includes introducing berthing terminal features, using latitude and longitude information from AIS data to determine the terminal where the ship berths, and counting the number of ships berthing at each terminal of a specific port within the research time range. This is used as a new feature for subsequent model training; the formula for determining which terminal a ship berths at is as follows:

[0021] ;

[0022] ;

[0023] in , For the ship's latitude and longitude, , The upper and lower limits of the longitude of the wharf under study. , The latitude limits of the pier under study are defined as follows.

[0024] Preferably, in step S3: the objective function of the multilayer perceptron (MLP) neural network model is to minimize the mean squared error.

[0025] ;

[0026] in, This represents the objective function. This represents the number of records in the training set. This represents the target vector on the training set, i.e., the port container throughput on the training set. Indicates the process The result of layer training;

[0027] The objective function of the XGBoost gradient boosting tree model is:

[0028] ;

[0029] in, Let be the objective function. For the target vector and the first The mean square error between the outputs of the trees, where For the first The residuals of the input features of each tree, i.e., the deviation between the current prediction and the true value; when Sometimes, Port container throughput in the ship information data characteristic table; The first gradient of the mean square error. The second gradient of the mean square error. For the first The output of each tree, This is a regularization term used to penalize complexity.

[0030] Preferably, the data preprocessing in step S3 includes: normalizing the numerical features using the following formula:

[0031] ;

[0032] in, The result after normalization For features that need to be normalized, and These are the maximum and minimum values ​​of this feature, respectively.

[0033] In step S3, the root mean square error on the test set is used as the basis for judging the model performance. The optimization parameters are adjusted according to the model performance to obtain the optimal model. The root mean square error formula is as follows:

[0034] ;

[0035] in, The actual target quantity on the test set, i.e., the port container throughput on the test set. The output of the trained model. This represents the number of data records in the test set.

[0036] Preferably, the weight calculation formula for model fusion in step S4 is as follows:

[0037] ;

[0038] in For the final output, Output results for the neural network model. Output results for the tree model. , where represents the weights of the two models, and the specific calculation formula is shown below:

[0039] ;

[0040] in , These represent the performance of the neural network model and the tree model on the test set, respectively, expressed as root mean square error.

[0041] Preferably, step S4 further includes: if the RMSE of the fusion model exceeds a preset threshold, dynamically adjust the model weights or re-optimize the parameters of the multilayer perceptron (MLP) neural network model and the XGBoost gradient boosting tree model.

[0042] It also includes step S5. Based on the monthly port container volume forecast results, generate port production and operation strategy optimization suggestions and adjust resource allocation. Adjusting resource allocation includes: increasing or decreasing terminal equipment scheduling, increasing or decreasing human resources, and increasing or decreasing yard storage strategies according to the forecasted container throughput.

[0043] This port container volume prediction and analysis method based on AIS big data has the following beneficial effects:

[0044] (1) Based on multi-source data fusion and hybrid machine learning model, this invention realizes high-precision prediction of port throughput and provides guidance for dynamic optimization of port production and operation strategies.

[0045] (2) The MLP and XGBoost of this invention can complement each other through weighted fusion, feature collaboration or stacking integration, which can significantly improve prediction accuracy and robustness. In complex scenarios such as port throughput prediction, this kind of hybrid model strategy is an effective means to solve the problems of data multi-source and task complexity.

[0046] (3) This invention deeply integrates AIS ship trajectory data, ship file data, and ship schedule data with machine learning models to construct an integrated analysis framework of "data-driven-model prediction-decision closed loop", breaking through the prediction bottleneck of traditional methods in complex port competition situation and dynamic environment. Attached Figure Description

[0047] Figure 1 : Flowchart of port container volume prediction based on AIS big data in this embodiment of the invention;

[0048] Figure 2 The method of this invention is applicable to the prediction error map of monthly container throughput at the Port of Singapore. Detailed Implementation

[0049] The following is combined with Figure 1 and Figure 2 The present invention will be further described as follows:

[0050] like Figure 1 As shown, the port container volume prediction and analysis method based on AIS big data of the present invention includes the following steps:

[0051] S1. Organize and collect ship AIS data, ship capacity data, shipping schedule data, and port container throughput data. Extract information data for all container ships within a specific port each month, and clean and verify the AIS data, including removing trajectory drift points and repairing missing fields. S2. Based on ship name and number, associate AIS ship trajectory data, ship files, and shipping schedule data to construct new features such as ship size and berthing time, generating a ship information data feature table. S3. Input the feature table into a Multilayer Perceptron (MLP) neural network model and an XGBoost gradient boosting tree model for training. The objective functions are minimizing the mean squared error and optimizing the gradient boosting tree loss function, respectively, to obtain port container volume prediction models. S4. Based on the root mean square error (RMSE) of the MLP neural network model and the XGBoost gradient boosting tree model on the test set, calculate the weights in reverse proportion, and weightedly fuse the prediction results of the two models to generate the final port container volume prediction value. Calculate the RMSE of the fused model to verify its accuracy.

[0052] Specifically, step S1 includes the following steps: S101: AIS data cleaning and verification.

[0053] The core tasks of this phase include: removing noisy data such as trajectory drift points; identifying and verifying abnormal records (e.g., IMO codes of the same vessel conflicting geographically at the same time, records exceeding actual navigation capabilities are considered abnormal, and sudden changes in latitude and longitude exceeding thresholds are considered drift points); repairing missing data and completing information; and identifying and handling abnormal berthing points (including anchorages, docks, and other special areas). For AIS data gaps, the system uses multi-source data fusion technology to interpolate and reconstruct missing fields during the cleaning process, ensuring information integrity.

[0054] Step S2 specifically includes the following steps:

[0055] AIS data association and integration: Based on the vessel name and number, AIS vessel trajectory data is associated with vessel files, shipping schedules, and other data to obtain relevant characteristic attributes of each vessel, such as vessel latitude and longitude, data collection time, sampling frequency, vessel type, actual carrying capacity, and vessel operator.

[0056] Feature generation:

[0057] 1) An adaptive threshold adjustment mechanism is introduced for ship size, and the judgment condition formula is as follows:

[0058] ;

[0059] in, To study the average rated TEU of ships over a time period, Standard deviation; To study the average rated TEU of ships over a time period, To study the standard deviation of the rated TEU of ships over a time period, To study ship carrying capacity within a time frame, this formula is used to dynamically classify ship sizes into large, medium, and small vessels, with the following value ranges: , Based on this standard, the number of large, medium and small ships each month can be statistically determined.

[0060] 2) Introduce the berthing dock feature.

[0061] The latitude and longitude information from AIS data is used to determine the docking berths of ships. The number of ships berthed at each dock in a specific port within the research time period is statistically analyzed and used as a new feature for subsequent model training. The formula for determining which dock a ship is berthed at is shown below:

[0062] ;

[0063] ;

[0064] in , For the ship's latitude and longitude, , The upper and lower limits of the longitude of the wharf under study. , The latitude limits of the pier under study are defined as follows:

[0065] 3) Use AIS data to generate the total berthing time for ships of all sizes.

[0066] The system uses the latitude and longitude information from the vessel's AIS data to determine whether the vessel is berthed, and then calculates the vessel's entry and exit times. The berthing times of all vessels are then statistically calculated. The calculation formula is shown below:

[0067] ;

[0068] in, For the first Total berthing time for this type of vessel The value can be either large, medium, or small (L, M, S) sizes. For each ship The time when this type of vessel leaves its berth. For each ship The time it takes for a vessel of a certain type to enter the berth; n is the size of the sliding window.

[0069] 4) Introduce a dynamic sliding window to study the impact of historical data.

[0070] The average of k historical data periods is used as the new feature. The calculation formula is as follows:

[0071] ;

[0072] Where k is the size of the sliding window. New features generated for the data in month d. For the dth month ago Monthly port container throughput;

[0073] Step S3 specifically includes the following steps:

[0074] S301. Using the ship information data feature table as the input dataset for the model, normalize the numerical features. The calculation formula is as follows:

[0075] ;

[0076] in, The result after normalization For features that need to be normalized, and These are the maximum and minimum values ​​of this feature, respectively.

[0077] S302. Divide all records in the dataset into a training set and a test set according to a certain strategy. The specific strategy can be random sampling or division according to time.

[0078] S303. Input the training set data into the neural network model and the tree model respectively for training. The neural network model uses the Multiple Perceptron (MLP) algorithm, and the tree model uses the Extreme Gradient Boosting (XGBoost) algorithm.

[0079] MLP is a feedforward artificial neural network model that receives raw data through the input layer, performs weighted summation and nonlinear activation function transformation through multiple hidden layers to extract high-order features, optimizes network weights through backpropagation algorithm, and finally generates target prediction results at the output layer.

[0080] The objective function of MLP is shown below. The ultimate goal of this algorithm is to minimize the objective function.

[0081] ;

[0082] in, This represents the objective function. This represents the number of records in the training set. This represents the target vector on the training set, i.e., the port container throughput on the training set. Indicates the process The result of layer training;

[0083] For each layer, the input and output are:

[0084] ;

[0085] ;

[0086] in, It is the first The weight matrix of the layer, It is the first The layer's bias vector, It is the first The activation function of the layer, It is the output of the previous layer (for the input layer, , which represents all input feature vectors in the ship information data feature table.

[0087] The backpropagation algorithm calculation formula is shown below:

[0088] ;

[0089] ;

[0090] in, It is the first The error term of the layer propagates through the derivative of the activation function. The learning rate is used to optimize network weights using the backpropagation algorithm. This ultimately forms a complete neural network model.

[0091] In a specific implementation, the details of the MLP parameters that yielded the optimal result are as follows: The neural network has 4 layers, with 256, 128, 32, and 1 neurons in each layer; the activation function is swish; the learning rate is 0.01; and the optimizer is Lion.

[0092] XGBoost is an efficient gradient boosting decision tree (GBDT) algorithm that uses the given function as the objective function to generate a tree model by minimizing the objective function.

[0093] ;

[0094] in, Let be the objective function. For the target vector and the first The mean square error between the outputs of the trees, where The residual of the input features for the tree-th tree is the deviation between the current prediction and the true value; when Sometimes, Port container throughput in the ship information data characteristic table; The first gradient of the mean square error. The second gradient of the mean square error. For the first The output of each tree, This is a regularization term used to penalize complexity.

[0095] XGBoost is generated iteratively using the above formula. A decision tree, which will eventually The results from each tree are fused to output a high-precision predicted value.

[0096] In a specific implementation, the XGBoost parameters that yielded the optimal results are as follows: tree depth set to 3; regularization parameter set to 0.1; learning rate set to 0.1. The co-optimization of MLP and XGBoost reduces the risk of overfitting, thus supporting the reasonableness of the RMSE value.

[0097] S304. Using the root mean square error (RMSE) on the test set as the criterion for judging model performance, adjust and optimize the parameters based on the model performance to obtain the optimal model. The formula for RMSE is shown below:

[0098] ;

[0099] in, The actual target quantity on the test set, i.e., the port container throughput on the test set. The output of the trained model. This represents the number of data records in the test set.

[0100] In step S3, the Multilayer Perceptron (MLP) algorithm and the Extreme Gradient Boosting (XGBoost) algorithm can significantly improve the performance of prediction results through model fusion strategies.

[0101] The two have the following cooperative relationship:

[0102] (1) Weighted fusion can combine the nonlinear fitting ability of MLP with the feature interpretability of XGBoost. At the same time, it reduces the overfitting risk of a single model and improves generalization performance.

[0103] (2) Collaborative feature engineering between the two: MLP extracts high-order features and uses the output of the MLP hidden layer (such as activation values) as the input features of XGBoost to enhance the tree model's ability to capture complex patterns. The feature importance of XGBoost output is used to guide the selection or weighting of MLP input features.

[0104] (3) Stacking and integration of the two: The prediction results of MLP and XGBoost are used as new features and input into the second-layer model (such as linear regression or another neural network) for final prediction. The prediction bias and complementarity of the two are learned through the meta-model, which further improves the accuracy.

[0105] In summary, although MLP and XGBoost are independent algorithms, they can complement each other through weighted fusion, feature collaboration, or stacked integration, significantly improving prediction accuracy and robustness. In complex scenarios such as port throughput prediction, this hybrid model strategy is an effective means to address the challenges of data diversity and task complexity.

[0106] Step S4 specifically includes the following steps:

[0107] S401: By fusing the two models, the final port container throughput prediction output is obtained. The calculation formula is shown below:

[0108] ;

[0109] in, For the final output, Output results for the neural network model. Output results for the tree model. , where represents the weights of the two models, and the specific calculation formula is shown below:

[0110] ;

[0111] in , These represent the performance of the neural network model and the tree model on the test set, respectively, expressed as root mean square error.

[0112] S402: Using the predicted port container throughput as the basis for analyzing port production and operation, providing strategic support for the dynamic optimization of port production and operation strategies.

[0113] In this invention, multi-source data processing is used to establish and integrate neural network models and tree models to analyze and predict the container throughput of important ports, providing strategic support for the dynamic optimization of port production and operation strategies.

[0114] like Figure 2As shown, RMSE (Root Mean Square Error) is a core metric for evaluating the performance of a prediction model; a lower value indicates that the prediction is closer to the actual value. In the scenario of port container volume prediction, taking Singapore Port as an example, the typical monthly throughput of Singapore Port is in the range of 3.4-3.5 million TEUs (standard containers). Therefore, an RMSE of 35,811 indicates an error rate of approximately 1%, demonstrating practical application value. This RMSE value indicates that the prediction results can provide a reliable reference for the port's monthly throughput, supporting dynamic resource allocation (such as terminal scheduling and manpower configuration).

[0115] , Used to calculate the fusion weights, RMSE=35,811 is an independent evaluation value of the fusion model prediction results.

[0116] 1. Base model error → Fusion weights;

[0117] Weights are allocated using a reverse proportional distribution:

[0118] ;

[0119] 2. Fusion weights → Predicted value F(x);

[0120] Weighted generation of final prediction:

[0121] ;

[0122] Output: Forecast of monthly throughput (e.g., 3.45 million TEUs).

[0123] 3. Predicted value F(x) → RMSE = 35,811

[0124] Calculated by comparing the actual values ​​on the test set:

[0125] ;

[0126] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the implementation of the present invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A method for predicting and analyzing port container volume based on AIS big data, comprising the following steps: S1. Organize and collect ship AIS data, ship capacity data, shipping schedule data and port container throughput data, extract information data of all container ships in a specific port each month, and clean and verify the AIS data, including removing trajectory drift points and repairing missing fields. S2. Based on the vessel name and serial number, link AIS vessel trajectory data, vessel files, and schedule data to construct new characteristics such as vessel size and berthing time, and generate a vessel information data characteristic table; step S2 includes the following: S201. Ship dimensions are classified based on dynamic thresholds, and the calculation formula is as follows: ;in, To study the average rated TEU of ships over a time period, Standard deviation; S202. Determine the ship berthing docks based on AIS latitude and longitude information, and count the number of ships berthed at each dock; S203. The formula for calculating the total berthing time of a vessel is as follows: ； in, For the first Total berthing time for this type of vessel The value can be either large, medium, or small (L, M, S) sizes. For each ship The time when this type of vessel leaves its berth. For each ship The time it takes for a vessel of this type to enter the berth; n is the size of the sliding window. Step S2 further includes: introducing a dynamic sliding window, using the historical data mean as a new feature, calculated using the following formula: ; Where k is the sliding window size, New features generated for the data in month d. For the dth month ago Monthly port container throughput; S3. Input the feature table into the multilayer perceptron (MLP) neural network and the XGBoost gradient boosting tree for training, respectively. The objective functions are minimizing the mean squared error and optimizing the gradient boosting tree loss function, respectively. The port container volume prediction models are trained to obtain the port container volume prediction models. S4. Based on the root mean square error (RMSE) of the MLP neural network model and the XGBoost gradient boosting tree model on the test set, calculate the weights in reverse proportion, and weightedly fuse the prediction results of the two models to generate the final port container volume prediction value. Calculate the RMSE of the fused model to verify the accuracy. The formula for calculating the weights in the model fusion process in step S4 is as follows: ;in For the final output, Output results for the neural network model. Output results for the tree model. , where represents the weights of the two models, and the specific calculation formula is shown below: ； in , The performance of the neural network model and the tree model on the test set are respectively represented by the root mean square error. Step S4 further includes: if the RMSE of the fusion model exceeds a preset threshold, dynamically adjust the model weights or re-optimize the parameters of the multilayer perceptron (MLP) neural network model and the XGBoost gradient boosting tree model; it also includes step S5. generating port production and operation strategy optimization suggestions based on the monthly port container volume forecast results, and adjusting resource allocation; the adjustment of resource allocation includes: increasing or decreasing terminal equipment scheduling, increasing or decreasing human resources, and increasing or decreasing yard storage strategies according to the predicted container throughput.

2. The port container volume prediction and analysis method based on AIS big data according to claim 1, characterized in that: Step S1 includes cleaning and verifying AIS data: S101. Clean and verify the ship's AIS data, remove trajectory drift points and abnormal records, and repair missing data fields; S102. Use multi-source data fusion technology to interpolate, repair, and reconstruct missing data.

3. The port container volume prediction and analysis method based on AIS big data according to claim 1, characterized in that: Step S2 further includes AIS data association and integration: based on the ship name and number, the AIS ship trajectory data is associated with the ship file and the ship schedule data to obtain the relevant characteristic attributes of each ship, specifically: ship latitude and longitude, data collection time, sampling frequency, ship type, actual carrying capacity and ship operator information.

4. The port container volume prediction and analysis method based on AIS big data according to claim 1, characterized in that: Step S2 further includes introducing berthing terminal features. The berths where ships berth are determined using latitude and longitude information from AIS data. The number of ships berthing at each terminal in a specific port within the research time period is counted and used as a new feature for subsequent model training. The formula for determining which terminal a ship berths at is shown below: ； in , For the ship's latitude and longitude, , The upper and lower limits of the longitude of the wharf under study. , The latitude limits of the pier under study are defined as follows.

5. The port container volume prediction and analysis method based on AIS big data according to claim 1, characterized in that: In step S3: The objective function of the multilayer perceptron (MLP) neural network model is to minimize the mean squared error. ;in, This represents the objective function. This represents the number of records in the training set. This represents the target vector on the training set, i.e., the port container throughput on the training set. Indicates the process The result of layer training; The objective function of the XGBoost gradient boosting tree model is: ； in, Let be the objective function. For the target vector and the first The mean square error between the outputs of the trees, where The residual of the input features for the tree-th tree is the deviation between the current prediction and the true value; when Sometimes, Port container throughput in the ship information data characteristic table; The first gradient of the mean square error. The second gradient of the mean square error. For the first The output of each tree, This is a regularization term used to penalize complexity.

6. The port container volume prediction and analysis method based on AIS big data according to claim 5, characterized in that: The data preprocessing in step S3 includes: normalizing the numerical features, using the following formula: ； in, The result after normalization For features that need to be normalized, and These are the maximum and minimum values ​​of the feature, respectively. In step S3, the root mean square error on the test set is used as the basis for judging the model performance. The optimization parameters are adjusted according to the model performance to obtain the optimal model. The root mean square error formula is as follows: ； in, The actual target quantity on the test set, i.e., the port container throughput on the test set. The output of the trained model. This represents the number of data records in the test set.

Images