Port container quantity prediction analysis method based on AIS big data

Abstract

The invention relates to an AIS big data-based port container quantity prediction analysis method, which comprises the following steps of: sorting and collecting ship AIS data, ship transport capacity data, shipping schedule data and port container quantity throughput data, and extracting information data of all container ships in a specific port per month; according to the ship name number, the associated AIS ship trajectory data, the ship archive and the ship schedule plan data, new features of the ship size and the berthing time are constructed, and a ship information data feature table is generated; respectively inputting the ship information data feature table into a neural network and a gradient boosting tree, and respectively training to obtain a port container quantity prediction model; and carrying out weighted fusion on the neural network model and the gradient boosting tree model to generate a final prediction model, and obtaining a monthly port container quantity prediction result. According to the invention, based on the multi-source data fusion and hybrid machine learning model, high-precision prediction of port throughput is realized, and guidance is provided for dynamic optimization of port production and operation strategies.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method for predicting and analyzing the container volume of a port based on AIS big data. Background Art

[0002] Currently, the analysis methods for the operation of container ports are mainly based on big data technology, collecting and collating relevant data that can reflect the port operation capacity. Then, on the basis of processing and analyzing the data, using the relevant theories of statistics and big data technology to solve various specific indicators that can reflect the port operation situation.

[0003] The existing technologies mainly have the following problems: First, the data island problem. Traditional prediction methods do not effectively integrate multi-source data with the actual operation records of the port, resulting in a single feature dimension. For example, AIS data (Automatic Identification System) only contains basic information such as ship position and speed, and the lack of correlation analysis between the actual transport capacity data and the ship physical parameters. Second, the feature construction is rough. For example, the existing methods use a fixed threshold to classify the ship size, without considering the following factors: the dynamic change of the actual loading rate of the ship with the route, and the non-linear influence of the ship type and port facilities matching degree. Third, the model adaptability is poor. For example, a single model is difficult to process structured data and time series data at the same time. Summary of the Invention

[0004] The present invention designs a method for predicting and analyzing the container volume of a port based on AIS big data, and the technical problem it solves is that the existing methods for predicting and analyzing the container volume of a port have data islands, rough feature construction, and poor model adaptability.

[0005] To solve the above-mentioned existing technical problems, the present invention adopts the following solutions: A method for predicting and analyzing the container volume of a port based on AIS big data, comprising the following steps: S1. Sort and collect ship AIS data, ship carrying capacity data, ship schedule data, and port container throughput data, extract information data of all container ships within a specific port each month, and clean and verify the AIS data, including removing trajectory drift points and repairing missing fields; S2. According to the ship name number, associate the AIS ship trajectory data, ship archives, and ship schedule data, construct new features of ship size and berthing time, and generate a ship information data feature table; S3. Input the feature table into a multi-layer perceptron MLP neural network and an XGBoost gradient boosting tree for training respectively. The objective functions are respectively the minimization of mean square error and the optimization of the gradient boosting tree loss function, and a port container volume prediction model is obtained through training respectively; S4. According to the root mean square error RMSE of the MLP neural network model and the XGBoost gradient boosting tree model on the test set, calculate the weights in reverse proportion, weighted fuse the prediction results of the two models, generate the final port container volume prediction value, and calculate the RMSE of the fusion model to verify the accuracy.

[0006] Preferably, the cleaning and verification of AIS data are included in step S1: Among them, S101. Clean and verify the ship AIS data, remove trajectory drift points and abnormal records, and repair missing data fields; among them, S102. Interpolate and reconstruct the missing data through multi-source data fusion technology.

[0007] Preferably, S201. Divide the ship size based on a dynamic threshold, and the calculation formula is: Among them, is the average value of the rated TEU of the ship within the research time range, is the standard deviation; S202. Determine the ship berthing dock according to the AIS longitude and latitude information, and count the number of ships berthing at each dock; S203. Calculate the total berthing time of the ship, and the formula is: ; Among them, among them is the total berthing time of the type of ship, takes values of three sizes: large, medium, and small (L, M, S), is the time when each type of ship leaves the berth, is the time when each type of ship enters the berth.

[0008] Preferably, the step S2 further includes: introducing a dynamic sliding window, and taking the average value of the historical n data as a new feature. The calculation formula is: where is the size of the sliding window, is the newly generated feature of the month's data, is the month before the month's port container throughput.

[0009] Preferably, the step S2 further includes the AIS data association and integration: according to the ship name number, associating the AIS ship trajectory data with the data such as ship archives and shipping schedule plans to obtain the relevant characteristic attributes of each ship, including ship longitude and latitude, data acquisition time, sampling frequency, ship type, actual carrying capacity, and ship operator information.

[0010] Preferably, the step S2 further includes introducing the berthing terminal feature. Determine the terminal where the ship berths based on the longitude and latitude information in the AIS data, and count the number of ships berthing at each terminal in a specific port within the statistical research time range. Use this as a new feature for subsequent model training. The formula for judging whether a ship berths at a specific terminal is as follows: where , are the longitude and latitude of the ship, , are the upper and lower limits of the longitude of the studied terminal, , are the upper and lower limits of the latitude of the studied terminal.

[0011] Preferably, in the step S3: the objective function of the multi-layer perceptron MLP neural network model is to minimize the mean square error: ; where represents this objective function, represents the number of records in the training set, represents the target vector, that is, the port container throughput represents the result output after layers of training; The objective function of the XGBoost gradient boosting tree model is: ; where, is the objective function, is the difference between the target vector and the The mean squared error between the output results of the trees, where is the residual of the input features of the -th tree (the deviation between the current prediction and the true value). In particular, when , there is The port container throughput in the ship information data feature table. is the first-order gradient of the mean squared error, is the second-order gradient of the mean squared error, is the output of the -th tree, is the regularization term for complexity penalty.

[0012] Preferably, the data preprocessing in step S3 includes: normalizing the numerical features, and the formula is: where is the result after normalization, is the feature to be normalized, and are the maximum and minimum values in this feature respectively.

[0013] In step S3, the root mean squared error on the test set is used as the basis for judging the model performance, and the optimization parameters are adjusted according to the model performance to obtain the optimal model. The root mean squared error formula is as follows:

[0014] where is the actual target value, that is, the port container throughput, is the output of the model after training, is the number of data records on the test set.

[0015] Preferably, the weight calculation formula for model fusion in step S4 is: where is the final output result, is the output result of the neural network model, is the output result of the tree model, , are the weights of the two models, and the specific calculation formula is as follows:

[0016] where 、 respectively represent the performance of the neural network model and the tree model on the test set, which is reflected by the root mean squared error.

[0017] Preferably, step S4 further includes: if the RMSE of the fusion model exceeds a preset threshold, dynamically adjust the model weights or re-optimize the parameters of the multi-layer perceptron (MLP) neural network model and the XGBoost gradient boosting tree model.

[0018] It further includes step S5. Generate optimization suggestions for the port production and operation strategy based on the monthly port container volume prediction results to adjust the resource allocation. Adjusting the resource allocation includes: according to the predicted container throughput, increasing or decreasing the terminal equipment scheduling, increasing or decreasing the human resources, and increasing or decreasing the yard storage strategy.

[0019] The port container volume prediction and analysis method based on AIS big data has the following beneficial effects: (1) Based on multi-source data fusion and hybrid machine learning models, the present invention realizes high-precision prediction of port throughput and provides guidance for the dynamic optimization of port production and operation strategies.

[0020] (2) Through weighted fusion, feature collaboration or stacked integration of MLP and XGBoost in the present invention, the advantages can be complementary, significantly improving the prediction accuracy and robustness. In complex scenarios such as port throughput prediction, such hybrid model strategies are effective means to solve the problem of data multi-source and task complexity.

[0021] (3) The present invention deeply combines AIS ship trajectory data, ship profile data, ship schedule data with machine learning models to construct an integrated analysis framework of "data-driven - model prediction - decision closed-loop", breaking through the prediction bottleneck of traditional methods in the complex port competition pattern and dynamic environment. Description of the Drawings

[0022] Figure 1 : Flowchart of port container volume prediction based on AIS big data in the embodiment of the present invention; Figure 2 : Prediction error chart of the monthly container throughput of the Singapore Port applicable to the method of the present invention. Detailed Embodiments

[0023] The following Figure 1 and Figure 2 are used to further illustrate the present invention: As Figure 1 shown, the port container volume prediction and analysis method based on AIS big data of the present invention includes the following steps: S1. Organize and collect ship AIS data, ship capacity data, ship schedule data, and port container throughput data. Extract information data of all container ships within a specific port each month, and clean and verify the AIS data, including removing trajectory drift points and repairing missing fields; S2. According to the ship name number, associate the AIS ship trajectory data, ship archives, and ship schedule data, construct new features of ship size and berthing time, and generate a ship information data feature table; S3. Input the feature table into a multi-layer perceptron MLP neural network model and an XGBoost gradient boosting tree model for training respectively. The objective functions are to minimize the mean square error and optimize the gradient boosting tree loss function respectively, and obtain port container volume prediction models through training respectively; S4. According to the root mean square error RMSE of the MLP neural network model and the XGBoost gradient boosting tree model on the test set, calculate the weights in reverse proportion, fuse the prediction results of the two models, generate the final port container volume prediction value, and calculate the RMSE of the fusion model to verify the accuracy.

[0024] Specifically, step S1 specifically includes the following steps: S101: Cleaning and verifying AIS data.

[0025] The core tasks of this stage cover the following contents: removing noise data such as trajectory drift points; identifying and verifying abnormal records (for example, when the IMO code of the same ship has geographical location conflicts in the same time period, records with speeds exceeding the actual navigation ability are regarded as abnormal, and longitude and latitude mutations exceeding the threshold are regarded as drift points); repairing data missing problems and completing information supplementation; identifying and processing abnormal operation berthing points (including special areas such as anchorages and shipyards). In the case of missing AIS data, the system uses multi-source data fusion technology to interpolate and repair missing fields and reconstruct data during the cleaning process to ensure information integrity.

[0026] Step S2 specifically includes the following steps: Associating and integrating AIS data: According to the ship name number, associate the AIS ship trajectory data with ship archives, ship schedule and other data to obtain relevant characteristic attributes of each ship, such as ship longitude and latitude, data collection time, sampling frequency, ship type, actual capacity, ship operator, etc.

[0027] Feature generation: 1) Introduce an adaptive threshold adjustment mechanism for ship size. The judgment condition formula is as follows: Among them, is the average rated TEU of the ship within the research time range, is the standard deviation of the rated TEU of the ship within the research time range, To study the ship capacity within the research time range, the ship size is dynamically divided into large ships, medium ships, and small ships using this formula, and their value ranges are respectively , . Based on this standard, the number of large, medium, and small ships per month is statistically obtained.

[0028] 2) Introduce the characteristics of the berthing terminals.

[0029] Determine the terminal where the ship is berthed based on the longitude and latitude information in the AIS data, and count the number of ships berthed at each terminal in a specific port within the research time range. This is used as a new feature for subsequent model training. The formula for determining whether a ship is berthed at a specific terminal is as follows:[[]]

[0030] where , are the longitude and latitude of the ship, , are the upper and lower limits of the longitude of the studied terminal, , are the upper and lower limits of the latitude of the studied terminal.

[0031] 3) Use the AIS data to generate the total berthing time of various sizes of ships.

[0032] Determine whether the ship is berthed at the berth based on the longitude and latitude information in the ship's AIS data, and obtain the time when the ship enters and leaves the berth. Statistically calculate the berthing time of all ships. The calculation formula is as follows:[[]] where is the total berthing time of the th type of ship, takes values of large, medium, and small (L, M, S) sizes, is the time when each th type of ship leaves the berth, is the time when each th type of ship enters the berth.

[0033] 4) Introduce a dynamic sliding window to study the influence of historical data.

[0034] Take the average of n periods of historical data as a new feature. The calculation formula is as follows:[[]] where is the size of the sliding window, is the new feature generated from the data of the th month, is the months ago The container throughput of a specific port per month.

[0035] The specific steps of step S3 include the following steps: S301. Using the ship information data feature table as the dataset of the input model, normalize the numerical features. The calculation formula is as follows:

[0036] Among them, is the result after normalization, is the feature to be normalized, and are the maximum and minimum values of this feature respectively.

[0037] S302. Divide all records in the dataset into a training set and a test set according to a certain strategy. The specific strategy can be random sampling or time-based partitioning.

[0038] S303. Input the training set data into a neural network model and a tree model respectively for training. Among them, the neural network model uses the multi-layer perceptron (MLP) algorithm, and the tree model uses the extreme gradient boosting (XGBoost) algorithm.

[0039] MLP is a feedforward artificial neural network model that receives raw data through the input layer, performs weighted summation and nonlinear activation function transformation layer by layer through multiple hidden layers, extracts high-order features, and optimizes the network weights through the backpropagation algorithm, and finally generates the target prediction result in the output layer.

[0040] The objective function of MLP is as follows, and the ultimate goal of this algorithm is to minimize the objective function.

[0041] Among them, represents this objective function, represents the number of records in the training set, represents the target vector, that is, the port container throughput represents after the result output by the

[0042] For each layer, the input and output are: Among them, is the weight matrix of the th layer, is the bias vector of the th layer, is the The activation function of the layer, is the output of the previous layer (for the input layer, , it is all the input feature vectors in the ship information data feature table).

[0043] The calculation formula of the backpropagation algorithm is as follows:

[0044] Among them, is the error term of the th layer, which is propagated through the derivative of the activation function, is the learning rate. The network weights are optimized through the backpropagation algorithm , and finally a complete neural network model is formed.

[0045] In the specific case implementation, the details of the MLP parameters for obtaining the optimal results are as follows. The number of neural network layers is set to 4 layers, and the number of neurons in each layer is 256, 128, 32, 1 respectively; the activation function is selected as swish; the learning rate is set to 0.01; the optimizer is selected as Lion.

[0046] XGBoost is an efficient gradient boosting decision tree (GBDT) algorithm, which uses the following function as the objective function. This algorithm generates a tree model by minimizing the objective function: Among them, is the objective function, is the mean square error between the target vector and the output result of the th tree, where is the residual of the input features of the th tree (the deviation between the current prediction and the true value). In particular, when , there is The port container throughput in the ship information data feature table. is the first-order gradient of the mean square error, is the second-order gradient of the mean square error, is the output of the th tree, is the regularization term, which is used for complexity penalty.

[0047] XGBoost generates decision trees through the above formula iteration, and finally fuses the results of trees to output a high-precision prediction value.

[0048] In the implementation of specific cases, the detailed XGBoost parameters for obtaining the optimal results are as follows: the tree depth is set to 3; the regularization parameter is set to 0.1; the learning rate is set to 0.1. The collaborative optimization of MLP and XGBoost reduces the overfitting risk, thus supporting the rationality of the RMSE value.

[0049] S304. The root mean square error on the test set is used as the basis for judging the model performance, and the optimization parameters are adjusted according to the model performance to obtain the optimal model. The root mean square error formula is as follows: where, is the actual target value, that is, the port container throughput, is the output of the model after training, is the number of data records on the test set.

[0050] In step S3, the multi-layer perceptron (MLP) algorithm and the extreme gradient boosting (XGBoost) algorithm can significantly improve the performance of the prediction results through the model fusion strategy.

[0051] The two have the following collaborative relationship: (1) Weighted fusion can combine the non-linear fitting ability of MLP and the feature interpretability of XGBoost. At the same time, it reduces the overfitting risk of a single model and improves the generalization performance.

[0052] (2) The two conduct feature engineering collaboration: MLP extracts high-order features and uses the output of the MLP hidden layer (such as activation values) as the input features of XGBoost to enhance the tree model's ability to capture complex patterns. Use the feature importance output by XGBoost to guide the input feature selection or weighting of MLP.

[0053] (3) The two are stacked and integrated: The prediction results of MLP and XGBoost are used as new features and input into the second-layer model (such as linear regression or another neural network) for final prediction. By learning the prediction biases and complementarities of the two through the meta-model, the accuracy is further improved.

[0054] In short, although MLP and XGBoost are independent algorithms, through weighted fusion, feature collaboration or stacked integration, they can achieve complementary advantages and significantly improve the prediction accuracy and robustness. In complex scenarios such as port throughput prediction, such hybrid model strategies are effective means to solve data multi-source and task complexity.

[0055] Step S4 specifically includes the following steps: S401: Integrate the two models to obtain the final port container throughput prediction output. The calculation formula is as follows: Among them, is the final output result, is the output result of the neural network model, is the output result of the tree model, , are the weights of the two models, and the specific calculation formula is as follows: Among them 、 respectively represent the performance of the neural network model and the tree model on the test set, which is reflected by the root mean square error.

[0056] S402: Use the predicted port container throughput as the basis for analyzing the port production and operation situation, and provide strategic support for the dynamic optimization of the port production and operation strategy.

[0057] In the present invention, through multi-source data processing, a neural network model and a tree model are established and fused, and then the throughput of important port containers is analyzed and predicted, providing strategic support for the dynamic optimization of the port production and operation strategy.

[0058] As Figure 2 shown, RMSE (root mean square error) is the core index for measuring the performance of the prediction model, and the lower the value, the closer the prediction is to the true value. In the scenario of port container volume prediction, taking the Port of Singapore as an example, the monthly throughput of the Port of Singapore is usually in the order of 3.4 - 3.5 million TEU (twenty-foot equivalent unit), then the error rate with RMSE = 35,811 is about 1%, which has practical application value. This RMSE value indicates that the prediction result can provide a reliable monthly throughput reference for the port, supporting dynamic resource allocation (such as terminal scheduling, manpower allocation).

[0059] 、 are used to calculate the fusion weight, and RMSE = 35,811 is the independent evaluation value of the prediction result of the fusion model.

[0060] 1. Basic model error → fusion weight; Allocate weights by inverse proportion:

[0061] 2. Fusion weight → predicted value F(x); Generate the final prediction by weighting: Output: The predicted value of the monthly throughput (such as 3.45 million TEU).

[0062] 3. Predicted value F(x) → RMSE = 35,811 Calculate by comparing with the true value of the test set: The present invention has been described exemplarily above in conjunction with the accompanying drawings. Obviously, the implementation of the present invention is not limited by the above-mentioned manner. As long as various improvements are made by adopting the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A method for predicting and analyzing the port container volume based on AIS big data, comprising the following steps: S1. Sort and collect ship AIS data, ship carrying capacity data, shipping schedule data, and port container throughput data, extract the information data of all container ships within a specific port each month, and clean and verify the AIS data, including removing trajectory drift points and repairing missing fields; S2. According to the ship name number, associate the AIS ship trajectory data, ship archives, and shipping schedule data, construct new features of ship size and berthing time, and generate a ship information data feature table; S3. Input the feature table into a multi-layer perceptron MLP neural network and an XGBoost gradient boosting tree for training respectively. The objective functions are respectively the minimization of mean square error and the optimization of the gradient boosting tree loss function, and port container volume prediction models are obtained through separate training; S4. According to the root mean square error RMSE of the MLP neural network model and the XGBoost gradient boosting tree model on the test set, calculate the weights in reverse proportion, weighted fuse the prediction results of the two models, generate the final port container volume prediction value, and calculate the RMSE of the fusion model to verify the accuracy.

2. The method for predicting and analyzing the port container volume based on AIS big data according to claim 1, wherein: Step S1 includes the cleaning and verification of AIS data: Among them, S101. Clean and verify the ship AIS data, remove trajectory drift points and abnormal records, and repair data missing fields; Among them, S102. Interpolate, repair, and reconstruct the missing data through multi-source data fusion technology.

3. The port container volume prediction and analysis method based on AIS big data according to claim 1, characterized in that: Step S2 includes the following contents: S201. Divide the ship size based on a dynamic threshold, and the calculation formula is: Among them, is the average value of the rated TEU of the ship within the research time range, is the standard deviation; S202. Determine the ship docking wharf according to the AIS longitude and latitude information, and count the number of ships berthing at each wharf; S203. Calculate the total berthing time of the ship, and the formula is: ； Among them, among them is the total berthing time of the class of ships, whose values are three sizes: large, medium, and small (L, M, S), is the time when each class of ship leaves the berth, is the time when each class of ship enters the berth.

4. The method for predicting and analyzing the port container volume based on AIS big data according to claim 3, wherein: Step S2 further includes: introducing a dynamic sliding window, taking the average value of the historical n data as a new feature, and the calculation formula is: Among them, is the sliding window size, is the newly generated feature of the month's data, is the month before month's port container throughput.

5. The method for predicting and analyzing the port container volume based on AIS big data according to claim 3, wherein: Step S2 further includes the association and integration of AIS data: According to the ship name number, associate the AIS ship trajectory data with data such as ship archives and shipping schedules to obtain the relevant characteristic attributes of each ship, including ship longitude and latitude, data collection time, sampling frequency, ship type, actual carrying capacity, and ship operator information.

6. The method for predicting and analyzing the port container volume based on AIS big data according to claim 3, wherein: Step S2 further includes introducing the berthing wharf feature. Determine the wharf where the ship berths according to the longitude and latitude information in the AIS data, count the number of ships berthing at each wharf in a specific port within the research time range, and use this as a new feature for subsequent model training; the formula for judging whether a ship berths at a specific wharf is as follows: wherein and are the longitude and latitude of the ship, and are the upper and lower limits of the longitude of the pier under study, and are the upper and lower limits of the latitude of the pier under study.

7. The port container volume prediction and analysis method based on AIS big data according to claim 1, characterized in that: In the step S3: The objective function of the multi-layer perceptron MLP neural network model is to minimize the mean square error: ； wherein represents the objective function represents the number of records in the training set represents the target vector, i.e., the throughput of port containers represents after the result of the output after training for The objective function of the XGBoost gradient boosting tree model is: ； Among them, is the objective function, is the mean squared error between the target vector and the output result of the th tree, where is the residual of the input features of the th tree (the deviation between the current prediction and the true value). In particular, when , there is The port container throughput in the ship information data feature table; is the first-order gradient of the mean squared error, is the second-order gradient of the mean squared error, is the output of the th tree, is the regularization term for complexity penalty.

8. The port container volume prediction and analysis method based on AIS big data according to claim 7, characterized in that: The data preprocessing in step S3 includes: normalizing the numerical features, and the formula is: Among them, is the result after normalization, is the feature to be normalized, and are respectively the maximum value and the minimum value in this feature; In the step S3, the root mean square error on the test set is used as the basis for judging the model performance, and the optimization parameters are adjusted according to the model performance to obtain the optimal model. The root mean square error formula is as follows: Among them, is the actual target volume, that is, the port container throughput, is the output of the model after training, is the number of data records in the test set.

9. The port container volume prediction and analysis method based on AIS big data according to claim 1, characterized in that: The weight calculation formula for model fusion in the step S4 is: Among them is the final output result, is the output result of the neural network model, is the output result of the tree model, , is the weight of the two models, and the specific calculation formula is as follows: Among them and represent the performance of the neural network model and the tree model on the test set, respectively, which is reflected by the root mean square error.

10. The port container volume prediction and analysis method based on AIS big data according to claim 9, characterized in that: The step S4 further includes: if the RMSE of the fusion model exceeds the preset threshold, dynamically adjust the model weights or re-optimize the parameters of the multi-layer perceptron MLP neural network model and the XGBoost gradient boosting tree model; It further includes step S5. Generate optimization suggestions for the port production and operation strategy based on the monthly port container volume prediction results, and adjust the resource allocation; the adjustment of the resource allocation includes: according to the predicted container throughput, increase or decrease the terminal equipment scheduling, increase or decrease the human resources, and increase or decrease the yard storage strategy.

Images