Multi-decoder dynamic attention model for container turnover
By designing a multi-decoder dynamic attention model in container thrust problem, using Transformer architecture and greedy baseline reinforcement learning algorithm, the problem of insufficient diversity in the existing technology is solved, and the solution quality and efficiency of container thrust problem is significantly improved. .
Patent Information
- Application Number
- CN202410935787.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-12
AI Technical Summary
The existing dynamic attention model only trains one construction strategy in the container rummaging problem, resulting in insufficient diversity and the inability to effectively find the optimal solution.
Design a multi-decoder dynamic attention model, based on the Transformer architecture, extract container stack semantics through feature enhancers, and train multiple decoders using a reinforcement learning algorithm with greedy baselines to increase the diversity of outputs, and use Kullback-Leibler divergence The loss forces the decoder to output different probability distributions.
The quality of solutions to container rumbling problems has been improved, especially in large-scale problems, which significantly improves the diversity and efficiency of solutions and reduces the number of rumbling times.
Smart Images

Figure CN118965953A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of container packing and tumbling, and in particular to a multi-decoder dynamic attention model for container tumbling. Background Art
[0002] In container terminal yards, container relocation is a common and important operation. Reducing the number of container relocations can not only improve the terminal's operating efficiency, but also reduce the moving distance of handling equipment, thereby reducing the non-productive costs of the terminal. Therefore, it is particularly important to study the container relocation problem and reduce the number of container relocations in the yard. The container relocation problem (CRP), also known as the block relocation problem (BRP), is an important combinatorial optimization problem that arises in yard management. The goal of CRP is to minimize the total number of relocations (or reprocessing) required to retrieve all containers from the bay.
[0003] There are many variations of CRP, which can be classified from the following two perspectives. First, consider the uniqueness of the priority value of the container. In a CRP with different priorities, each container is assigned a unique priority. Therefore, the retrieval order of the containers is determined. In contrast, in a CRP with duplicate priorities (or group priorities), the priorities in a group of containers can be the same, and the retrieval order between them is arbitrary. Another classification method is to consider the restrictions on container roaming. In a restricted CRP (RCRP), only containers in the stack where the highest priority container is located are allowed to be relocated. In contrast, in an unrestricted CRP (UCRP), there is no such restriction, and any container on the top of the stack can be relocated.
[0004] Deep learning has shown good capabilities in solving NP-hard decision problems and has achieved remarkable results in various fields, such as using deep reinforcement learning (DRL) to solve the game of Go. However, for CRP problems, research using deep learning methods is still limited. Unlike the traditional attention model (AM), the dynamic attention model (DAM) considers dynamic state information. After each operation, the new bay structure is re-input into the DAM to accurately decide the next operation. In DAM, the bay structure is represented by a two-dimensional matrix and used as the input of DAM. The multi-head attention layer extracts the features of the stack in the bay structure. Based on the features of these stacks, the decoder gradually constructs the solution in an autoregressive manner. Autoregression means that the decoder selects a target stack to add to the partial solution each time according to the output probability distribution until the complete solution is constructed. In order to obtain a better solution, a sampling strategy is adopted to generate a set of solutions from the trained neural network model, and then the best one is selected from this set of solutions.
[0005] However, DAM has the following defects. First, directly inputting the two-dimensional matrix into the neural network model may lose some important information of the stack, and multi-head attention may not be able to extract some key features. Secondly, in the decoding stage of DAM, a single decoder is used to output the solution, and the generated solution will lack diversity. Intuitively, increasing the diversity of solutions will potentially lead to better solutions. Because there are multiple optimal solutions for CRP, by generating diverse solutions, the possibility of finding the optimal solution can be increased. In addition, in a set of solutions generated by sampling, due to insufficient generation diversity, some solutions are the same, which will reduce the search range of the solution space. The existing method DAM only trains one construction strategy and samples solutions from this strategy. Diversity comes only from sampling, but since the probability distribution is deterministic, this leads to insufficient diversity. Summary of the invention
[0006] The purpose of the present invention is to provide a multi-decoder dynamic attention model for container rummage, aiming to solve the problem that the existing dynamic attention model only trains one construction strategy and lacks diversity.
[0007] To achieve the above objectives, the present invention provides a multi-decoder dynamic attention model for container flipping, comprising the following steps:
[0008] Design a multi-decoder dynamic attention model with feature enhancement based on the Transformer architecture;
[0009] Training the multi-decoder dynamic attention model using a greedy baseline based reinforcement learning algorithm;
[0010] Input a bay structure, extract the container stack semantics through the feature enhancer, and form a bay structure containing the inter-stack semantics;
[0011] Using the trained multi-decoder dynamic attention model, predict the bin flipping sequence of the input bay.
[0012] Among them, the multi-decoder dynamic attention model including feature enhancement based on Transformer architecture design includes designing a feature enhancer, an encoder and a decoder.
[0013] Among them, the specific method of designing the special enhancer is:
[0014] Five primitive features representing inter-stack semantics are defined based on heuristic rules and manual experience;
[0015] Based on the input bay structure, the five original features are calculated and expressed in a matrix form to obtain an inter-stack semantic feature matrix;
[0016] The bay-level construction matrix and the inter-stack semantic feature matrix are connected to form a bay-level construction matrix containing inter-stack semantics.
[0017] The original features include: the height percentage of the current stack, the percentage of the number of empty spaces in the stack, the stack priority, the priority of the target container and the number of disordered stacks in the stack.
[0018] Among them, the specific method of designing the special encoder is:
[0019] Input the output of the special enhancer into the long short-term memory network to form the initial embedding required by the Transformer;
[0020] The output of the long short-term memory network is input into the encoding end of the Transformer, and the global semantic features of the bay-level structure are extracted through the self-attention mechanism and residual connection.
[0021] The invention discloses a multi-decoder dynamic attention model for container flipping. The multi-decoder dynamic attention model including feature enhancement is designed based on the Transformer architecture. The multi-decoder dynamic attention model is trained based on a reinforcement learning algorithm with a greedy baseline: a bay structure is input, and the container stack semantics is extracted by a feature enhancer to form a bay structure including inter-stack semantics; the trained multi-decoder dynamic attention model is used to predict the flipping operation sequence of the input bay. The method designs a stack feature enhancer to enhance the input bay structure information, which effectively helps the encoder capture the relationship between stacks. The decoder dynamic attention model trains multiple construction strategies to increase the diversity of the output. It uses a Transformer-like encoder to encode the stack information, and uses multiple different attention decoders with non-shared parameters to sample different solutions. During the training process, each decoder learns a different solution mode and is regularized by the Kullback-Leibler (KL) divergence loss to force the decoder to output different probability distributions, thereby solving the problem that the existing dynamic attention model only trains one construction strategy and lacks diversity. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0023] Figure 1 It is an overall flow chart of a multi-decoder dynamic attention model for container rummage provided by the present invention.
[0024] Figure 2 Schematic diagram of training the multi-decoder dynamic attention model based on the greedy baseline reinforcement learning algorithm.
[0025] Figure 3 is a schematic diagram of using the trained multi-decoder dynamic attention model to predict the box flipping operation sequence of the input bay. DETAILED DESCRIPTION
[0026] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0027] See also Figures 1 to 3 , the present invention provides a multi-decoder dynamic attention model for container flipping, comprising the following steps:
[0028] S1 is a multi-decoder dynamic attention model based on the Transformer architecture with feature enhancement;
[0029] Specifically, the design of a multi-decoder dynamic attention model with feature enhancement based on the Transformer architecture includes designing feature enhancers, encoders, and decoders. Based on the Transformer network architecture, the multi-decoder dynamic attention model (MDDAM) is used to train multiple construction strategies to increase the diversity of outputs. The encoding end of MDDAM integrates a stack feature enhancer (SFE) to improve the encoder's ability to capture the semantics between container stacks; the decoding end integrates multiple decoders, each of which is based on multi-head attention, and the diversity between decoders is maintained by KL divergence.
[0030] Specific methods for designing special enhancer (SFE):
[0031] S101 defines five primitive features representing inter-stack semantics based on heuristic rules and manual experience;
[0032] Specifically, based on heuristic rules and manual experience, five primitive features are defined to represent the semantics of stacks. These five features are: the height percentage of the current stack (1 / T, 2 / T, …, 1); the percentage of empty spaces in the stack; the stack priority; the priority of the target container; and the number of unordered stacks (US) in the stack.
[0033] S102 calculates the five original features based on the input bay structure and expresses them in a matrix form to obtain an inter-stack semantic feature matrix;
[0034] Specifically, five inter-stack semantic features are calculated based on the input bay structure and expressed in a matrix form. A row of the matrix has 5 components, and each component corresponds to an inter-stack semantic feature value.
[0035] S103 connects the Bayeux construction matrix and the inter-stack semantic feature matrix to form a Bayeux construction matrix containing inter-stack semantics.
[0036] Specific ways to design the encoder:
[0037] S111 inputs the output of the special enhancer into a long short-term memory network to form an initial embedding required by the Transformer;
[0038] Specifically, the output of SFE enters the Long Short-Term Memory (LSTM) network to form the initial embedding required by Transformer.
[0039] S112 inputs the output of the long short-term memory network into the encoding end of the Transformer, and extracts the global semantic features of the bayeux structure through the self-attention mechanism and residual connection.
[0040] Specifically, the output of LSTM enters the encoding end of Transformer, and extracts the global semantic features of the bay structure through the self-attention mechanism and residual connection.
[0041] Design a decoder.
[0042] design decoders with the same structure, each decoder consists of a multi-head attention network;
[0043] Define the KL divergence between decoders and add it to the loss function in training as a regular term to maintain the diversity of output solutions;
[0044] Ablation experiments verify the effectiveness of SFE and multi-decoder (MD) design.
[0045] Specifically, using the average number of box flipping as the metric, both SFE and MD effectively improve the quality of CRP solutions, especially when the problem scale is large, the effect is more obvious.
[0046] S2 trains the multi-decoder dynamic attention model based on a greedy baseline reinforcement learning algorithm;
[0047] Specifically, each of the multiple decoders outputs a different probability distribution and independently samples a solution π d , to obtain a separate reinforcement learning loss:
[0048] Where L(π d )
[0049] For the solution π d The sequence length is the total number of box flips. Since the stack selection action is sampled from a probability distribution and the total number of box flips is used as the cost, the cost recorded in different episodes has a large variance due to the uncertainty of sampling, which affects the convergence speed (Li et al., 2022). In order to reduce the variance and speed up learning, a baseline b(X) is usually introduced. The baseline we use is similar to the baseline in (Kool et al., 2019). b(X) is the baseline strategy p θ*The total number of rollouts for the deterministic greedy rollout. The baseline policy baselinepolicy is trained to the current best model. At the end of each training cycle, once the currently trained model is better than the baseline policy, we will replace the baseline policy with the current model. The model parameters θ can be updated by gradient descent using the REINFORCE algorithm:
[0050]
[0051] in represents the REINFORCE loss gradient, is the gradient of the KL divergence between the decoder output probability distributions, k KL is the coefficient of KL loss. KL The reason why it is negative in the beginning is that we want to maximize the diversity between decoders. Generally speaking, for each state encountered by each decoder, the KL loss should be calculated during training to encourage the decoder to generate different solutions. However, we only apply the KL loss in the first step for the following reasons: on the one hand, it is to avoid expensive calculations, and on the other hand, for the same instance, after choosing different steps, the state of the instance is different, and it does not make sense to pursue different choices for different instance outputs.
[0052] In the current model θ The sampled solutions for each decoder are obtained (line 6), and the best model p θ* The greedy solution for each decoder is obtained (line 7). Lines 8-9 define the loss function of DRL. The parameters are optimized by the Adam optimizer (line 10). If the model p θ The solution is better than the best model p θ* The solution is to update the parameter θ * (Lines 12-13).
[0053]
[0054]
[0055] 64,000 instances are randomly generated. The input is an S×T bay structure, where S is the number of stacks in the bay and T is the height of the bay. The output is a box-turning operation sequence π=(π1,π2,…,π K ), where π i (i∈{1,2,…,K}) represents the i-th box flipping operation.
[0056] Set the number of iterations and the number of iteration steps, using Figure 2The reinforcement learning algorithm shown in the figure trains MDDAM. In the figure, epoch represents the current training round of the model, and step represents the number of iterations in a training round. Step++ and epoch++ respectively represent: step = step + 1 and epoch = epoch + 1.
[0057] S3 inputs a bay structure, extracts the container stack semantics through the feature enhancer, and forms a bay structure containing the inter-stack semantics;
[0058] Specifically, the feature enhancer (SFE) extracts the inter-stack semantics of the bay structure, which is expressed by an S×5 matrix, where each row of the matrix represents the semantic features of a stack: stack height percentage, stack empty space percentage, stack priority, target container priority, and the number of unordered stacking (US) in the stack. S and T represent the number of stacks and the layer height limit in the bay structure, respectively.
[0059] The S×T bay structure is connected with the S×5 inter-stack semantic features to output the S×(T+5) bay structure.
[0060] S4 uses the trained multi-decoder dynamic attention model to predict the box flipping operation sequence of the input bay.
[0061] Specifically, the trained MDDAM is used to predict the sequence of container flipping operations. For a new input container structure, MDDAM processes each container one by one and predicts the container flipping operation π for the container that needs to be flipped. i , until all containers in the bay structure are processed, MDDAM outputs the container turning operation sequence π of the bay.
[0062] Specific method: Step 1: Input the bay structure of the box-turning task into the multi-decoder dynamic attention model (MDDAM);
[0063] Step 2: Determine the target container to be processed according to the priority order of the containers;
[0064] Specifically, the higher the priority of the container, the more priority it needs to be handled.
[0065] Step 3: Determine whether there is an obstruction box above the target container. If not, jump to step 4; otherwise, jump to step 5.
[0066] Specifically, a container without an obstruction box can be directly retrieved, and the meaning of retrieval is to delete the container from the bay structure.
[0067] Step 4: Delete the target container from the bay structure and jump to step 2;
[0068] Specifically, after deletion, the number of containers in the bay structure will be reduced by one.
[0069] Step 5: Use MDDAM to predict and execute the turning operation of all obstructing boxes above the target container;
[0070] Specifically, after execution, the number of containers in the bay structure will not decrease, but the distribution of containers in the bay structure will change.
[0071] Step 6: Determine whether the container in the bay has been processed. If so, jump to step 7, otherwise jump to step 4.
[0072] Specifically, the container in the bay structure has been processed, which means that the bay is empty and the container turning task of the bay is completed.
[0073] Step 7: Output the box-turning operation sequence corresponding to the bay position and end.
[0074] The above disclosure is only a preferred embodiment of the multi-decoder dynamic attention model for container turning of the present invention. Of course, it cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.
Claims
1. A multi-decoder dynamic attention model for container flipping, characterized in that: The following steps are involved: Design a multi-decoder dynamic attention model with feature enhancement based on the Transformer architecture; Training the multi-decoder dynamic attention model using a greedy baseline reinforcement learning algorithm; Input a bay structure, extract the container stack semantics through the feature enhancer, and form a bay structure containing the inter-stack semantics; Using the trained multi-decoder dynamic attention model, predict the bin flipping sequence of the input bay.
2. A multi-decoder dynamic attention model for container flipping as claimed in claim 1, characterized in that: The multi-decoder dynamic attention model including feature enhancement designed based on Transformer architecture includes designing a feature enhancer, an encoder and a decoder.
3. A multi-decoder dynamic attention model for container flipping as claimed in claim 2, characterized in that: The specific method of designing the special enhancer is as follows: Five primitive features representing inter-stack semantics are defined based on heuristic rules and manual experience; Based on the input bay structure, the five original features are calculated and expressed in a matrix form to obtain an inter-stack semantic feature matrix; The bay-level construction matrix and the inter-stack semantic feature matrix are connected to form a bay-level construction matrix containing inter-stack semantics.
4. A multi-decoder dynamic attention model for container flipping as claimed in claim 3, characterized in that: The original features include: the height percentage of the current stack, the percentage of the number of empty spaces in the stack, the stack priority, the priority of the target container, and the number of disordered stacks in the stack.
5. A multi-decoder dynamic attention model for container flipping as claimed in claim 2, characterized in that: The specific method of designing the special encoder is as follows: Input the output of the special enhancer into the long short-term memory network to form the initial embedding required by the Transformer; The output of the long short-term memory network is input into the encoding end of the Transformer, and the global semantic features of the bay-level structure are extracted through the self-attention mechanism and residual connection.
Citation Information
Patent Citations
Container position distribution method, device and equipment for container yard and medium
CN115330227A
Method for pre-dumping containers in container yard
CN116451290A
Machine learning-based container terminal turnover method, device and equipment
CN117952292A
Cited By
Marine transportation container assembling method based on sequence generation and search
CN120069706A
