Hierarchical clustering Q learning-based ship equipment spare part demand prediction method
By using the Q-learning method based on hierarchical clustering, multiple prediction models were constructed and the best model was selected using the Q-learning algorithm. This solved the problem of the difficulty in accurately predicting the intermittent demand for ship equipment spare parts, reduced inventory costs, and improved prediction accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-18
- Publication Date
- 2026-03-27
AI Technical Summary
Existing technologies struggle to accurately predict the intermittent demand for ship equipment spare parts, leading to high spare parts inventory costs and potential equipment downtime. Improved forecasting methods are urgently needed.
A hierarchical clustering-based Q-learning method is adopted. By preprocessing, clustering and feature extraction of historical spare parts demand data, multiple prediction models are constructed. The Q-learning algorithm is used to select the best model for prediction, thereby reducing the error.
It enables the automatic selection of the best prediction model based on the distribution of spare parts characteristics, reducing spare parts inventory costs, improving prediction accuracy, and avoiding the risk of equipment downtime.
Smart Images
Figure CN121745904A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to ship equipment maintenance technology, and more particularly to a method for predicting the demand for ship equipment spare parts based on hierarchical clustering and Q-learning. Background Technology
[0002] In the marine industry, spare parts are a vital resource for equipment maintenance and support. They are essential components and tools for maintaining the normal operation of equipment, playing a crucial role in ensuring its operation and safety. Overstocking spare parts not only increases warehousing costs but may also lead to some spare parts becoming expired or obsolete; understocking spare parts may necessitate emergency procurement or obtaining spare parts through more expensive means, causing ship downtime and increasing maintenance costs. Therefore, accurate spare parts demand forecasting and planning are essential for the normal operation of ships.
[0003] In actual operation, spare parts demand is closely related to maintenance. Aside from regular preventative maintenance, most maintenance needs occur randomly, meaning the demand for spare parts with a large number of zero-demand components also exhibits significant fluctuations. This type of demand is known as the intermittent demand characteristic of spare parts, and its non-normal distribution makes accurate demand forecasting extremely difficult.
[0004] Existing methods for forecasting intermittent spare parts demand can be mainly categorized into Crosston method and its variants, Boosterping and its variants, time aggregation, Crosston-like methods, and machine learning / deep learning methods. Currently, the most widely used methods are traditional forecasting methods such as Crosston, Simple Exponential Smoothing (SES), and ARIMA. Among them, Crosston is specifically designed for forecasting intermittent demand, decomposing the intermittent demand sequence into two sequences: demand size and demand interval. These two sequences are then forecasted separately and combined.
[0005] However, due to the limitations of single forecasting methods for intermittent demand sequences with multiple demand patterns, how to further reduce the forecasting error of intermittent spare parts, better provide decision-makers with spare parts procurement references, and reduce spare parts inventory costs is an urgent problem to be solved. Summary of the Invention
[0006] The technical problem to be solved by the present invention is to provide a method for predicting the demand for ship equipment spare parts based on hierarchical clustering Q-learning, which addresses the deficiencies in the existing technology.
[0007] The technical solution adopted by this invention to solve its technical problem is: a method for predicting the demand for ship equipment spare parts based on hierarchical clustering Q-learning, comprising the following steps: 1) Obtain historical demand data for ship equipment spare parts, sort and organize the collected data in chronological order, and form a complete time series with a set period as the unit; The historical demand data for ship equipment spare parts includes the amount of spare parts required during maintenance and the time when maintenance occurs. 2) Preprocess the collected dataset to identify and handle missing and outlier values. 3) Divide the dataset into a training dataset and a test dataset, and perform clustering operations on the two datasets respectively; Extract intermittent feature representations from all sequences in the training and test datasets; A feature-based hierarchical clustering method is constructed to cluster the training dataset and the test dataset separately to obtain the classification results of the dataset; 4) Construct a spare parts prediction model Based on historical demand data for ship spare parts, a set of prediction models is constructed, including a univariate classical prediction model and a multivariate global prediction model for intermittent demand forecasting. 5) Train the Q-learning spare parts prediction model using the training dataset; 5.1) Calculate the comprehensive prediction error of each prediction model in the prediction model set, where the comprehensive prediction error is the average error of the three errors: NRMSE, MASE, and MAE. 5.2) Take each feature category of the data in the training dataset as the state space, take each prediction model as the action space, take the comprehensive prediction error as the reward in the form of rules, train the spare parts demand prediction model based on hierarchical clustering Q learning, and repeat the prediction n times. Take the average Q value of n times from the Q value table and select the best prediction model for each class. 6) Based on the ship spare parts demand data to be predicted, use the trained model to predict subsequent ship spare parts demand data.
[0008] According to the above scheme, in step 2), the mean and standard deviation of non-zero demand are used as Z-score parameters, as follows: 2.1) Calculate the mean of non-zero demand, the standard deviation of non-zero demand, and the median of non-zero demand for each spare part demand sequence; 2.2) Calculate the Z-score value for each spare part in each sequence cycle. If the absolute value of the Z-score value is greater than a set threshold, it is considered an outlier. The calculation is as follows:
[0009] in, It is the demand value. It is the mean of non-zero demand. It is the standard deviation of non-zero demand; 2.3) Replace outliers with the median of non-zero demand.
[0010] According to the above scheme, in step 3), the number of clusters is determined by maximizing the mean of the four indicators: Dunn index, silhouette coefficient, DB index, and CH index.
[0011] According to the above scheme, in step 3), the intermittent features of all sequences in the training dataset and test dataset are extracted using the following features: average demand interval (ADI), squared coefficient of variation of non-zero demand (CV2), approximate entropy, percentage of zero demand, percentage of demand outside the mean standard deviation of non-zero demand, coefficient of linear least squares regression, average absolute value of continuous demand change, ratio of the sum of squares of the last block to the sum of squares of the entire sequence, and percentage of consecutive zero values at the end.
[0012] According to the above scheme, the univariate classical prediction models are the SBA prediction model and the ETS prediction model; the multivariate global prediction models are the DeepAR, LSTM, LightGBM and Deep Renewal Exact prediction models.
[0013] According to the above scheme, the process of generating the Q-value table in step 5.2) is as follows: 5.2.1) Initialization: Initialize the Q-value table Q(s, a) by setting all Q-values to 0; initializing Q-values to 0 indicates that all state-action pairs have the same initial value. Where s represents the state and a represents the action; 5.2.2) Action Selection: The initial state is randomly selected, and an ε-greedy strategy is adopted. The current optimal action is selected with a probability of ε, and an action is randomly selected with a probability of 1-ε. 5.2.3) State transition: Execute action a, observe reward r and the next state s'. State s' is determined based on error information in the R table. The smaller the error, the higher the probability of the state being selected, thus realizing the transition to a better state. 5.2.4) Q-value update: Update the current Q-value based on the current reward and the maximum Q-value of the next state; The reward is derived from the prediction error; the smaller the error, the greater the reward. 5.2.5) When the Q value no longer changes significantly or the preset number of iterations is reached, the learning process terminates and a Q table is generated.
[0014] The beneficial effects of this invention are: This invention provides a spare parts demand forecasting method based on hierarchical clustering and Q-learning. This method can automatically select the best forecasting model according to different spare parts feature distributions, reduce the forecasting error of spare parts demand, and achieve the goal of reducing spare parts inventory costs. Attached Figure Description
[0015] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0017] like Figure 1 As shown, a method for predicting the demand for ship equipment spare parts based on Q-learning using hierarchical clustering includes the following steps: Step 1: Obtain historical demand data for ship spare parts from relevant databases or records. This mainly includes information such as the demand quantity and the time of demand occurrence. Sort and organize the collected data in chronological order to form a complete time series. Due to the intermittent nature of ship spare parts, the organized time series will be represented in months. (1.1) Collect historical demand data for ship spare parts: It is necessary to obtain historical demand data for ship spare parts from relevant databases or records. This mainly includes information such as the demand quantity and the time when the demand occurred; (1.2) Constructing a spare parts time series: Sort and organize the collected data in chronological order to form a complete time series. Due to the intermittent nature of ship spare parts, the organized time series is represented in months. (1.3) Spare parts sequence visualization: Select typical spare parts demand sequences for visualization to highlight the intermittent, regular, obsolete, and fluctuating characteristics of intermittent spare parts.
[0018] Since ship maintenance is carried out in accordance with the maintenance manual, the demand for spare parts and the timing of maintenance have periodic characteristics. Step 2: Preprocess the collected dataset to identify and handle missing and outlier values. The outlier identification and handling steps are as follows: (2.1) Calculate the non-zero mean, non-zero standard deviation, and non-zero median of the demand for each spare part; (2.2) Calculate the Z-score value for each spare part for each month, and set the threshold to ±3. If the Z-score value is greater than 3 or less than 3, the Z-score is considered invalid. 3. If the data is greater than the mean plus or minus three standard deviations, then it is considered an outlier.
[0019] in, It is the demand value. It is the mean of non-zero demand. The standard deviation of non-zero demand.
[0020] (2.3) Replace outliers with the median of non-zero demand.
[0021] Step 3: Divide the dataset into dataset A and dataset B, and perform clustering and classification operations on the two datasets; (3.1) Extract the average demand interval (ADI), the coefficient of variation squared for non-zero demand (CV2), the approximate entropy, the percentage of zero demand, and the demand location for all sequences in dataset A. Nine features are included: percentage of out-of-range values, coefficients of linear least squares regression, average absolute value of continuous demand changes, ratio of the sum of squares of the last block to the sum of squares of the entire sequence, and percentage of consecutive zero values at the end. The formulas for ADI and CV2 are as follows:
[0022] in, This represents the total number of periods of the i-th sequence; This represents the number of non-zero periods in the i-th sequence; This represents the standard deviation of the non-zero demand for the i-th sequence; This represents the mean of non-zero demand. The ratio of the sum of squares of the last block to the sum of squares of the entire sequence, where a block is a continuous subsequence segment obtained by dividing the sequence into segments of fixed length. It is equivalent to a time window sequence.
[0023] (3.2) Based on the features in Section 3.1, construct a feature-based hierarchical clustering method to cluster dataset A; (3.3) Extract the nine features from all sequences in dataset B as shown in step 3.1; (3.4) Use the nine features of dataset B as input to the hierarchical clustering method in step 3.2 to classify the sequences in dataset B.
[0024] Step 4: Based on historical demand data for ship spare parts, construct six prediction models, namely SBA, ETS, DeepAR, LSTM, LightGBM, and DeepRenewal Exact, and make predictions on dataset A.
[0025] SBA forecasting is a variant of Croston's method; ETS is an exponential smoothing method, also a commonly used method for forecasting spare parts demand; both of these are classic univariate forecasting models. DeepAR, LSTM, LightGBM, and Deep Renewal Exact are all multivariate global forecasting models.
[0026] Step 5: Use dataset A to train a spare parts prediction model based on hierarchical clustering Q-learning, and use dataset B to verify the prediction effect of the model.
[0027] (5.1) Divide both dataset A and dataset B into training and test sets. The training set is used to train the prediction model for each class, and the test set is used to verify the performance of the prediction model. NRMSE, MAE, and MASE are used as evaluation metrics. The specific formulas are as follows:
[0028] Where n is the sample size. and and are the actual value and predicted value of the i-th observation, respectively.
[0029] Because the distribution and scale of each data type differ, the RMSE metric is normalized, and the Normalized Root Mean Square Error (NRMSE) is used to evaluate the performance of the forecasting model. Two other metrics are also commonly used to evaluate the accuracy of intermittent demand forecasting, and their formulas are as follows:
[0030] (5.2) Using the feature categories of each data class in dataset A as the state space, the six prediction models as the action space, and the comprehensive prediction error as a reward in the form of rules, train a spare parts demand prediction model based on hierarchical clustering Q-learning. Repeat the prediction 10 times, and take the average Q value of the 10 predictions from the Q table to guide the selection of the best prediction model for each class. The comprehensive prediction error is the average error of three errors: NRMSE, MASE, and MAE. The specific steps of the prediction model based on hierarchical clustering Q-learning are as follows: (5.2.1) State space s: Based on the clustering results, nine features are calculated for each class of sequences, including discontinuity, regularity, obsolescence, and fluctuation, and these are used as class features; Action space a: Executes six prediction models, including SBA, ETS, DeepAR, Deep Renewal Exact, LightGBM, and LSTM.
[0031] (5.2.2) Initialization: 1) Initialize the Q-value table Q(s, a), setting all Q-values to 0; 2) Randomly select initial state data s; (5.2.3) Action a selection: Given the current data state s, action a is selected using the ε-greedy strategy; Execute action a, observe reward r and the next state s'. Selection of the next state s': Compile the average error of the three errors (NRMSE, MASE, MAE) of the actions executed in each state into a table, which is the R table. Randomly select the next state according to the R table, thereby determining the next state. Update the Q value according to the Bellman equation;
[0032] Update the current state to the next state s'.
[0033] (5.2.4) Repeat step 2 until convergence or the specified number of iterations is reached, and generate the Q table.
[0034] (5.3) Based on the trained hierarchical clustering-based Q-learning prediction model, the best prediction model for each class is selected using the dataset B labeled with the class in step 3.4, and the spare parts in dataset B are predicted to evaluate the prediction generalization performance of the hierarchical clustering-based Q-learning model on dataset B.
[0035] (6) Based on the ship spare parts demand data to be predicted, use the trained model to predict the subsequent ship spare parts demand data, and update the inventory quantity according to the prediction results.
[0036] Example 2 This embodiment uses the RAF dataset for experiments. The dataset contains 5,000 spare parts with a total requirement of 84 months. The best example is used to illustrate how the present invention solves the technical problem.
[0037] Step 1: Obtain information on the demand for ship spare parts, including the timing of demand, from relevant databases or records. Organize the data by month. Due to the large amount of data, only a portion is listed in Table 1. To highlight the intermittent, regular, obsolete, and fluctuating characteristics of intermittent spare parts, select typical spare parts demand sequences for visualization. Step 2: Handling Missing and Outlier Values Identification and handling of missing values: The data is preprocessed. Assuming that the months in which the demand occurred are all recorded correctly, the presence of missing values indicates that no demand occurred in that month. In this case, the missing values are filled with 0. Outlier Mistakes and Handling: Outliers arise due to recording errors or the occurrence of emergencies. First, the mean and standard deviation of non-zero demands are calculated using the improved Z-score parameter. An improved Z-score method with a threshold of ±3 is used to identify outliers among the non-zero demands, and these outliers are replaced with the median of the non-zero demands. A total of 196 outliers were identified and processed.
[0038] Step 3: Clustering and Classification of Data (3.1) The first 4000 spare parts in the RAF dataset are used as dataset A, and the last 1000 spare parts are used as dataset B; (3.2) Extract nine features such as ADI, CV2, and approximate entropy from the two datasets respectively. Use hierarchical clustering to cluster dataset A. Use the mean of four indicators, Dunn index, silhouette coefficient, DB index and CH index, to evaluate the optimal number of clusters. The comprehensive index is maximized when K=7.
[0039] (3.3) Input the nine features of dataset B and use the trained hierarchical clustering model to classify dataset B.
[0040] Step 4: Based on historical demand data for ship spare parts, construct six prediction models, including SBA, DeepAR, and Deep Renewal Exact, as detailed below: (4.1) The SBA method models a single spare parts sequence, decomposes the intermittent demand sequence into two sequences: demand size and demand interval, predicts these two sequences separately and then combines them, searches the parameter space using the Nelder-Mead method to find the model parameters that minimize the sum of squared residuals, and finally adjusts the final predicted value by adding a damping coefficient to the Croston predicted value.
[0041] (4.2) The ETS model models a single spare parts sequence and uses the AIC and BIC criteria to determine the optimal model for each sequence; (4.3) The DeepAR model models multiple spare parts sequences. It is based on the GRU model and sets up two RNN layers, each containing 128 units, with a learning rate of 0.01. (4.4) The Deep Renewal Exact model models multiple spare parts sequences. It is based on LSTM and has an RNN layer with 64 units per layer and a learning rate of 0.01.
[0042] (4.5) The LightGBM model calculates the characteristics of historical data for each spare part, such as the average demand over twelve months, the total demand over twelve months, and the average demand interval over twelve months. Using lag and sliding as feature variables and historical demand as the target variable, a tree model is constructed using LightGBM to learn the relationship between features and the target, thereby making accurate predictions.
[0043] (4.6) The LSTM model processes all sequences through a sliding window, adjusts the data format as input to the LSTM model, considers the interrelationships between spare parts, and realizes the prediction of spare parts demand.
[0044] Step 5: Use dataset A to train a spare parts prediction model based on hierarchical clustering Q-learning, and use dataset B to verify the prediction effect of the model.
[0045] (5.1) Divide both dataset A and dataset B into training and test sets. The training set is used to train the prediction model for each class, and the test set is used to verify the performance of the prediction model. NRMSE, MAE, and MASE are used as evaluation metrics. The specific formulas are as follows:
[0046] Where n is the sample size. and and are the actual value and predicted value of the i-th observation, respectively.
[0047] Because the distribution and scale of each data type differ, the RMSE metric is normalized, and the Normalized Root Mean Square Error (NRMSE) is used to evaluate the performance of the forecasting model. Two other metrics are also commonly used to evaluate the accuracy of intermittent demand forecasting, and their formulas are as follows:
[0048] (5.2) Take the feature category of each data class in dataset A as the state space, take the six prediction models as the action space, take the comprehensive prediction error as the reward in the form of rules, train the spare parts demand prediction model based on hierarchical clustering Q learning, and repeat the prediction 10 times. Take the average Q value of 10 times in the Q table to guide the selection of the best prediction model for each class.
[0049] Step 6: Use the best prediction model for each class to make predictions on the labeled dataset B, and evaluate the prediction performance using NRMSE, MASE, and MAE.
[0050] Step 7: Use the trained model to predict subsequent ship spare parts demand data.
[0051] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A method for predicting the demand for ship equipment spare parts based on Q-learning using hierarchical clustering, characterized in that, Includes the following steps: 1) Obtain historical demand data for ship equipment spare parts, sort and organize the collected data in chronological order to form a complete demand data sequence with a set period as the unit; The historical demand data for ship equipment spare parts includes the amount of spare parts required during maintenance and the time when maintenance occurs. 2) Preprocess the collected dataset to identify and handle missing and outlier values. 3) Divide the dataset into a training dataset and a test dataset, and perform clustering operations on the two datasets respectively; Extract intermittent feature representations from all sequences in the training and test datasets; A feature-based hierarchical clustering method is constructed to cluster the training dataset and the test dataset separately to obtain the classification results of the dataset; 4) Construct a spare parts prediction model; Based on historical demand data for ship spare parts, a set of prediction models is constructed, including a univariate classical prediction model and a multivariate global prediction model for intermittent demand forecasting. 5) Train the Q-learning spare parts prediction model using the training dataset; 5.1) Calculate the combined prediction error of each prediction model in the prediction model set; 5.2) Use each feature category of the data in the training dataset as the state space, each prediction model as the action space, and the comprehensive prediction error as a reward in the form of rules to train a spare parts demand prediction model based on hierarchical clustering Q-learning. 6) Based on the ship spare parts demand data to be predicted, use the trained model to predict the subsequent ship spare parts demand data, and update the inventory quantity according to the prediction results.
2. The method for predicting the demand for ship equipment spare parts based on hierarchical clustering Q-learning according to claim 1, characterized in that, In step 2), the mean and standard deviation of non-zero demand are used as Z-score parameters, as follows: 2.1) Calculate the mean of non-zero demand, the standard deviation of non-zero demand, and the median of non-zero demand for each spare part demand sequence; 2.2) Calculate the Z-score value for each spare part in each sequence cycle. If the absolute value of the Z-score value is greater than a set threshold, it is considered an outlier. The calculation is as follows: in, It is the demand value. It is the mean of non-zero demand. It is the standard deviation of non-zero demand; 2.3) Replace outliers with the median of non-zero demand.
3. The method for predicting the demand for ship equipment spare parts based on hierarchical clustering Q-learning according to claim 1, characterized in that, In step 3), the number of clusters is determined by maximizing the mean of the four indices: Dunn index, silhouette coefficient, DB index, and CH index.
4. The method for predicting the demand for ship equipment spare parts based on hierarchical clustering Q-learning according to claim 1, characterized in that, In step 3), the intermittent features of all sequences in the training and test datasets are extracted using the following features: average demand interval (ADI), squared coefficient of variation of non-zero demand (CV2), approximate entropy, percentage of zero demand, percentage of demand outside the mean standard deviation of non-zero demand, coefficient of linear least squares regression, average absolute value of continuous demand change, ratio of the sum of squares of the last block to the sum of squares of the entire sequence, and percentage of consecutive zero values at the end.
5. The method for predicting the demand for ship equipment spare parts based on hierarchical clustering and Q-learning according to claim 1, characterized in that, The univariate classical prediction models are the SBA prediction model and the ETS prediction model; the multivariate global prediction models are the DeepAR, LSTM, LightGBM, and Deep Renewal Exact prediction models.
6. The method for predicting the demand for ship equipment spare parts based on Q-learning with hierarchical clustering according to claim 1, characterized in that, In step 5.1), the comprehensive prediction error is the average error of the three errors: NRMSE, MASE, and MAE.
7. The method for predicting the demand for ship equipment spare parts based on hierarchical clustering Q-learning according to claim 1, characterized in that, In step 5.2), during training, the prediction is repeated n times, and the average Q value of the n predictions is taken from the Q value table to select the best prediction model for each class.
8. The method for predicting the demand for ship equipment spare parts based on Q-learning with hierarchical clustering according to claim 1, characterized in that, The process of generating the Q-value table in step 5.2) is as follows: 5.2.1) Initialization: Initialize the Q-value table Q(s, a) by setting all Q-values to 0; initializing Q-values to 0 indicates that all state-action pairs have the same initial value. Where s represents the state and a represents the action; 5.2.2) Action Selection: The initial state is randomly selected, and an ε-greedy strategy is adopted. The current optimal action is selected with a probability of ε, and an action is randomly selected with a probability of 1-ε. 5.2.3) State transition: Execute action a, observe reward r and the next state s'. State s' is determined based on error information in the R table. The smaller the error, the higher the probability of the state being selected, thus realizing the transition to a better state. 5.2.4) Q-value update: Update the current Q-value based on the current reward and the maximum Q-value of the next state; The reward is derived from the prediction error; the smaller the error, the greater the reward. 5.2.5) When the Q value no longer changes significantly or the preset number of iterations is reached, the learning process terminates and a Q table is generated.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.