An online 3D bin packing method based on deep reinforcement learning

Abstract

The present invention discloses an online three-dimensional packing method based on deep reinforcement learning, which belongs to the technical field of packing, and includes: establishing a neural network model and training it according to a trust region actor-critic algorithm based on Kronecker factor approximate curvature and a generalized advantage estimation algorithm; setting a buffer zone, sequentially storing a number of boxes to be packed and obtaining information of a number of boxes to be packed and information of containers for packing, using the trained neural network model to make predictions, obtaining a packing strategy, and performing packing operations according to the packing strategy until the packing container cannot place any box in the buffer zone. The present invention establishes a neural network model based on the actor-critic algorithm and the generalized advantage estimation algorithm, which can reduce deviations and has stronger generalization ability and versatility. At the same time, the states of boxes and containers are separated, and a buffer zone and a separate direction strategy network are added, so as to reduce training costs, accelerate network convergence, and improve packing space utilization and test efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of box packing, and in particular relates to an online three-dimensional box packing method based on deep reinforcement learning. Background Art

[0002] With the rapid development of social economy, the business demand of logistics industry is increasing, and the total logistics cost is also growing. How to reduce logistics cost has become an urgent problem to be solved in logistics industry. Online 3D packing technology is a key technology used in logistics and transportation fields. It aims to effectively pack boxes of different sizes to maximize the use of the space of transportation containers. Reasonable packing scheme can improve space utilization while reducing packing cost, thereby reducing logistics cost.

[0003] At present, the algorithms commonly used to solve the online three-dimensional packing problem mainly include heuristic algorithms and deep reinforcement learning. Heuristic algorithms are a type of algorithm based on experience and rules, which is used to find the approximate optimal solution in a reasonable time. In online three-dimensional packing, heuristic algorithms need to model the size of the box, the capacity of the container and other related parameters, and design a heuristic rule to place the box and gradually optimize the current solution. Since the formulation of rules needs to be based on the characteristics and experience of the problem, when the size of the container or object changes, the rules need to be manually re-formulated, which has poor generalization and high complexity.

[0004] Compared with heuristic algorithms, deep reinforcement learning algorithms are more adaptable and have strong generalization capabilities. They can adapt to the situations of containers and boxes of different sizes. In recent years, the application of deep reinforcement learning has made significant progress in the field of online three-dimensional box packing. However, due to the lack of a buffer zone, the situation where the current box cannot be placed in the container but the next box can be placed is ignored. At the same time, the state representation of the box and container is complex, resulting in a large number of parameters in the deep neural network for feature extraction and low model prediction efficiency. The position selection method after adding the rotation direction of the box is unreasonable, resulting in the doubling of the number of action spaces and the difficulty of network training. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides an online three-dimensional packing method based on deep reinforcement learning, which solves the problems of low packing efficiency and unsatisfactory effect while considering the buffer zone and the six-dimensional rotation of the box.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: an online three-dimensional boxing method based on deep reinforcement learning, comprising the following steps:

[0007] S1: According to the trust region actor-critic algorithm ACKTR based on Kronecker factor approximate curvature and the generalized advantage estimation algorithm, a neural network model is established, and training is performed using training boxes and packing containers to obtain a trained neural network model;

[0008] S2: Set a buffer zone to sequentially store a number of boxes to be loaded, and obtain information of a number of boxes to be loaded and information of containers for loading;

[0009] S3: Based on the information of the first box to be packed and the information of the packing container, the trained neural network model is used to make predictions to obtain a packing strategy;

[0010] S4: According to the packing strategy, determine whether the box meets the static stability and boundary constraints. If so, perform the packing operation, obtain a new box to be packed and store it in the buffer, and return to S3. Otherwise, predict the packing strategy of the next box according to the storage order of the boxes to be packed in the buffer, and enter S5.

[0011] S5: Determine whether several boxes to be packed in the buffer cannot meet the static stability and boundary constraints. If so, end the packing; otherwise, return to S4.

[0012] The beneficial effects of the present invention are as follows: the present invention adopts a neural network model, and adds a broad advantage estimation and a KAFC algorithm on the basis of an advantage actor-critic algorithm, which can reduce deviations, make network training more stable, and have stronger generalization ability and versatility. At the same time, adding a buffer and a separate direction strategy network can make training easier, reduce time costs, accelerate network convergence, and improve space utilization. In addition, the present invention represents the states of boxes and containers separately, and uses a lighter network for feature extraction, so that the model parameters obtained by training are fewer, while ensuring space utilization and improving test efficiency.

[0013] Furthermore: the information of the box is a one-dimensional vector, including the length, width and height of the box, and the container information for packing is a height map of the container from top to bottom, and the height map is a two-dimensional matrix.

[0014] The beneficial effect of the above further scheme is: by representing the box information and the container information separately, it is convenient to use a lightweight network for feature extraction later, so that the trained model has fewer parameters, ensuring space utilization while improving test efficiency.

[0015] Further: the specific steps of S1 are as follows:

[0016] S101: Establish a neural network model based on the trust region actor-critic algorithm ACKTR based on Kronecker factor approximate curvature and the generalized advantage estimation algorithm;

[0017] S102: According to the training boxes and the packing containers, a neural network model is used to make predictions to obtain a packing strategy;

[0018] S103: Perform a packing operation according to the packing strategy and calculate an instant reward for the packing operation;

[0019] S104: Based on the immediate reward, the expected value of the cumulative reward of the neural network model is calculated, and it is determined whether the expected value of the cumulative reward reaches the maximum. If so, the training is completed to obtain the trained neural network model. Otherwise, the network parameters of the neural network model are optimized and returned to S102. The neural network model includes: feature extraction network, direction strategy network, position strategy network and value network.

[0020] The beneficial effect of the above further scheme is as follows: on the basis of the trust region actor-critic algorithm based on the Kronecker factor approximate curvature, the present invention introduces a broad advantage estimation algorithm to reduce calculation deviation, make network training more stable, obtain a stronger model generalization ability, and improve the versatility of the algorithm.

[0021] Further: The expression of the expected value of the cumulative reward is as follows:

[0022]

[0023]

[0024] Among them, J(π) is the expected value of the cumulative reward, which represents the cumulative packing space utilization after multiple packing operations, γ is the discount factor, T is the number of steps to collect each trajectory, and E τ～π () is the expected value calculated by accumulating and adding the instant rewards at each moment on multiple trajectories, τ represents a trajectory formed after a series of state transitions are completed according to the packing strategy π, R(s t ,a t ) is the instant reward calculated at time t, s t is the state of the trajectory at time t, a t is the action selected by the trajectory at time t, R is the immediate reward, which represents the increased packing space utilization after each packing operation, and l i is the length of the i-th box, w i is the width of the ith box, h i is the height of the i-th box, L is the length of the container, W is the width of the container, H is the height of the container, and i is the i-th box.

[0025] The beneficial effect of the above further scheme is that by calculating the expected value of the cumulative reward after completing each packing operation, it can provide a basis for judging whether the training performance of the neural network model meets the actual use standard.

[0026] Further: the neural network model includes: a feature extraction network, a direction strategy network, a position strategy network and a value network;

[0027] The feature extraction network is used to extract the coding information in the box information and the coding information in the container information for packing;

[0028] The direction strategy network is used to predict the direction of box placement based on the box encoding information and the container encoding information, and obtain a direction decision;

[0029] The location strategy network is used to predict the location of the box according to the box coding information and the container coding information, and obtain the location decision;

[0030] The value network is used to evaluate the direction decision and the position decision respectively to obtain the direction decision value and the position decision value.

[0031] The beneficial effect of the above further scheme is: through the feature extraction network, direction strategy network, location strategy network and value network, it is possible to analyze the information of the box and the information of the container used for packing, obtain the packing strategy and the corresponding packing strategy value, and better train the neural network model.

[0032] Further: the specific steps of S3 are as follows:

[0033] S301: Input the information of the first box to be loaded into the fully connected neural layer of the feature extraction network for encoding to obtain box encoding information;

[0034] S302: Inputting the container information for packing into the convolutional neural layer of the feature extraction network for encoding to obtain container encoding information;

[0035] S303: Add the box coding information and the container coding information, and input them into the direction strategy network and the value network to obtain the direction decision and the direction decision value respectively;

[0036] S304: according to the direction decision, obtain the index of the box placement direction, and obtain the information of the box after rotation according to the index correspondence;

[0037] S305: Input the information of the box rotation into the fully connected neural layer for encoding to obtain new box encoding information;

[0038] S306: Add the new box coding information to the container coding information and send them to the location strategy network and the value network to obtain the location decision and the location decision value respectively;

[0039] S307: According to the direction decision and the position decision, the index corresponding to the box placement direction and the index corresponding to the box placement position are obtained respectively, and they are combined to obtain a packing strategy. The neural network model includes: a feature extraction network, a direction strategy network, a position strategy network and a value network.

[0040] The beneficial effect of the above further scheme is: according to the feature extraction network, strategy network and value network of the neural network model, the packing strategy and decision value of the box can be obtained, which is convenient for subsequent judgment of whether the box can be placed in the packing container and the advantages and disadvantages of the packing strategy.

[0041] Further: whether the static stability is satisfied includes:

[0042] Whether the supported area of ​​the bottom of the box is greater than a first threshold of the bottom area of ​​the box;

[0043] Whether the supported area of ​​the bottom of the box is greater than the second threshold of the bottom area of ​​the box, and three corners of the bottom of the box are supported;

[0044] Whether the supported area of ​​the bottom of the box is greater than the third threshold of the bottom area of ​​the box, and the four corners of the bottom of the box are supported.

[0045] The beneficial effect of the above further solution is: according to the static stability, it is ensured that the box is in a stable and reasonable position when it is actually stored.

[0046] Further: whether the boundary constraint condition is satisfied includes:

[0047] Constrain the box in three directions: length, width and height, and determine whether any edge of the box exceeds the boundary of the container when it is placed in the container. If so, it means that the box cannot be placed in the container; otherwise, the box can be placed in the container.

[0048] The beneficial effect of the above further solution is: through the boundary constraint conditions, it is ensured that the placement position of the box is reasonable, and the placement of the box can be completed during actual use.

[0049] Further: the steps of packing operation in S4 are as follows:

[0050] B1: According to the packing strategy, determine whether the placement of the boxes meets the static stability and boundary constraints;

[0051] B2: According to the direction strategy in the box packing strategy, the box is rotated in 6 dimensions to obtain a box with accurate direction;

[0052] B3: According to the location strategy in the packing strategy, select a box with accurate location and orientation in the packing container to complete the packing operation.

[0053] The beneficial effect of the above further scheme is: the present invention can greatly reduce the number of actions when selecting the box position by dividing the packing strategy into a direction strategy and a position strategy, making training easier, reducing time cost and accelerating network convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of an online 3D binning method based on deep reinforcement learning;

[0055] Figure 2 It is a schematic diagram of a height diagram of the container state according to the present invention;

[0056] Figure 3 This is a schematic diagram of placing the boxes of the present invention;

[0057] Figure 4 A detailed schematic diagram of the deep neural network architecture of the present invention;

[0058] Figure 5 This is a schematic diagram of placing the box in six rotation directions according to the present invention;

[0059] Figure 6 It is a schematic diagram showing the best test result of the baseline algorithm model of the present invention without adding a buffer zone;

[0060] Figure 7 It is a visualization diagram of the best test result of the algorithm model not considering the rotation direction according to the present invention without adding a buffer zone;

[0061] Figure 8 It is a visualization diagram of the best test result of the algorithm model considering 6 rotation directions according to the present invention without adding a buffer zone;

[0062] Fig. 9 It is a visual schematic diagram of the best test result of the baseline algorithm model of the present invention when the buffer size is 4;

[0063] Fig.10 It is a visualization diagram of the best test result of the algorithm model of the present invention without considering the rotation direction when the buffer size is 4;

[0064] Fig.11 This is a visualization diagram of the best test result of the algorithm model considering 6 rotation directions described in the present invention when the buffer size is 4. DETAILED DESCRIPTION

[0065] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0066] like Figure 1 As shown, the present invention provides an online three-dimensional boxing method based on deep reinforcement learning, comprising the following steps:

[0067] S1: According to the trust region actor-critic algorithm ACKTR based on Kronecker factor approximate curvature and the generalized advantage estimation algorithm, a neural network model is established, and training is performed using training boxes and packing containers to obtain a trained neural network model;

[0068] S2: Set a buffer zone to sequentially store a number of boxes to be loaded, and obtain information of a number of boxes to be loaded and information of containers for loading;

[0069] S3: Based on the information of the first box to be packed and the information of the packing container, the trained neural network model is used to make predictions to obtain a packing strategy;

[0070] S4: According to the packing strategy, determine whether the box meets the static stability and boundary constraints. If so, perform the packing operation, obtain a new box to be packed and store it in the buffer, and return to S3. Otherwise, predict the packing strategy of the next box according to the storage order of the boxes to be packed in the buffer, and enter S5.

[0071] S5: Determine whether several boxes to be packed in the buffer cannot meet the static stability and boundary constraints. If so, end the packing; otherwise, return to S4.

[0072] In the present invention, whether static stability is satisfied in S4 includes:

[0073] Whether the supported area of ​​the bottom of the box is greater than a first threshold of the bottom area of ​​the box;

[0074] Whether the supported area of ​​the bottom of the box is greater than the second threshold of the bottom area of ​​the box, and three corners of the bottom of the box are supported;

[0075] Whether the supported area of ​​the bottom of the box is greater than the third threshold of the bottom area of ​​the box, and the four corners of the bottom of the box are supported.

[0076] In one embodiment of the present invention, the first threshold may be set to 95%, the second threshold may be set to 85%, and the third threshold may be set to 50%.

[0077] Whether the boundary constraints are met includes:

[0078] Constrain the box in three directions: length, width and height, and determine whether any edge of the box exceeds the boundary of the packing container when it is placed in the container. If so, it means that the box cannot be placed in the packing container. Otherwise, the box can be placed in the packing container.

[0079] Among them, if the box meets the static stability and boundary constraints, the box is rotated in 6 dimensions according to the direction strategy in the packing strategy; according to the position strategy in the packing strategy, a position is selected in the packing container to place the box to complete the packing operation.

[0080] In one embodiment of the present invention, the information of a box can be represented as a one-dimensional vector [l,w,h], where l is the length of the box, w is the width of the box, and h is the height of the box; the information of the container is represented by a top-down height map of the container, such as Figure 2 As shown in FIG. 1 , it is a schematic diagram of the height map of the packing container in the present invention. Assuming that L is the length of the packing container, W is the width of the packing container, and H is the height of the packing container, then a two-dimensional matrix L×W can be used to represent the height map, that is, Figure 2 In the X×Y matrix, each corresponding value in the matrix represents the height of the boxes stacked at the corresponding position of the container. In the present invention, the box information and the information of the packing container can be obtained by sensors, depth cameras, etc., and real-time information transmission is required.

[0081] In one embodiment of the present invention, the specific steps of training the neural network model according to the training box and the packing container are as follows:

[0082] S101: Establish a neural network model based on the trust region actor-critic algorithm ACKTR based on Kronecker factor approximate curvature and the generalized advantage estimation algorithm;

[0083] S102: According to the training boxes and the packing containers, a neural network model is used to make predictions to obtain a packing strategy;

[0084] S103: Perform a packing operation according to the packing strategy and calculate an instant reward for the packing operation;

[0085] S104: Based on the immediate reward, calculate the expected value of the cumulative reward of the neural network model, and determine whether the expected value of the cumulative reward reaches the maximum. If so, complete the training and obtain the trained neural network model. Otherwise, optimize the network parameters of the neural network model and return to S102.

[0086] Among them, the neural network model includes: a feature extraction network, a direction strategy network, a position strategy network and a value network; the feature extraction network is used to extract the coding information in the box information and the coding information in the container information for packing; the direction strategy network is used to predict the direction of the box placement according to the box coding information and the container coding information, and obtain the direction decision; the position strategy network is used to predict the location of the box placement according to the box coding information and the container coding information, and obtain the location decision; the value network is used to evaluate the direction decision and the position decision respectively, and obtain the direction decision value and the position decision value.

[0087] Specifically, the process of solving the packing problem is treated as a Markov decision process, which is represented by a tuple (S, A, P, R), where S is the environment state set, A is the action set, P is the state transition probability function, and R is the reward function; during the training process, the information of the box and the container used for packing are used as the current environment state s t ,s t ∈S, for s t Preprocessing is performed as the input of the feature extraction network. After feature extraction, the box information and container information are obtained as the input of the direction strategy network, the location strategy network and the value network. The packing strategy is obtained and the packing operation is performed according to the packing strategy as action a. t , a t ∈A, in S4, if the box meets the static stability and boundary constraints, the box is packed and the state is transferred to obtain the next state s t+1 Otherwise, no state transfer is performed, a round of box packing operation is ended, and the box packing space utilization rate is calculated, and the increased box packing space utilization rate is used as an immediate reward R(s t ,a t ); when the packing strategy is obtained, the value network will output a value related to the current state, which is used to evaluate the pros and cons of the current placement plan. Through the continuous optimization of network parameters, this value will continue to approach the expected long-term reward in each step state, and at the same time guide the policy network to update its strategy, making it more likely to take actions in high-value states. After several state transfers, the network parameters are continuously optimized, and finally the maximum cumulative reward expectation value J(π) is obtained, that is, the accumulated space utilization after completing multiple steps of packing operations, and the training is completed.

[0088] In one embodiment of the present invention, the core of the trust region actor-critic algorithm ACKTR based on Kronecker factor approximate curvature includes two parts: an agent and an environment. The environment represents an online three-dimensional packing scene, including container information, upcoming box information, rewards, constraints, and time step information. The agent represents an entity for receiving information from the environment and performing packing operations according to the packing strategy to achieve the goal of improving space utilization.

[0089] During the training phase, a random selection strategy is used when selecting directions and positions, with the aim of allowing the agent to explore more good actions. During the testing phase, since the model training has converged, a greedy strategy is used when selecting directions and positions, always selecting the action with the highest probability.

[0090] Among them, the expression of packing space utilization is as follows:

[0091]

[0092] Among them, utilization is the utilization rate of packing space, l i is the length of the i-th box, w i is the width of the ith box, h i is the height of the i-th box, L is the length of the container, W is the width of the container, H is the height of the container, and i is the i-th box.

[0093] The expression of the reward function is as follows:

[0094]

[0095] Among them, R is the reward function, which represents the increased packing space utilization after each packing operation, l i is the length of the i-th box, w i is the width of the ith box, h i is the height of the i-th box, L is the length of the container, W is the width of the container, and H is the height of the container.

[0096] The expression for the expected value of the cumulative reward is as follows:

[0097]

[0098] Among them, J(π) is the expected value of the cumulative reward, which represents the cumulative packing space utilization after multiple packing operations, γ is the discount factor, T is the number of steps to collect each trajectory, and E τ～π () is the expected value calculated by accumulating and adding the instant rewards at each moment on multiple trajectories, τ represents a trajectory formed after a series of state transitions are completed according to the packing strategy π, R(s t ,at ) is the instant reward calculated at time t, s t is the state of the trajectory at time t, a t is the action selected by the trajectory at time t, and R is the immediate reward, which represents the increased packing space utilization after each packing operation.

[0099] Among them, state transfer includes: height map update and box information update;

[0100] Height map update: Take the point corresponding to the lower left rear of the box as the coordinate point for the box to be placed. The placement diagram is as follows Figure 3 As shown, its bottom surface is mapped to a two-dimensional plane. In the two-dimensional matrix representing the container state, the area covered by the box after being placed in the container is traversed, and the maximum value max_height in the area is found, which is the height of the box. The maximum value in the area is added to the height of the box, and the calculation result is assigned to the area covered by the box. The update of the height map is completed, and the updated height map information is added to the new state list;

[0101] Box information update: In the training phase, the next box information is directly added to the new status list; in the testing phase, the next box information is first inserted into the buffer box information storage list, the insertion position is the position where the last box was taken away, and then the next box information is added to the new status list.

[0102] In one embodiment of the present invention, a buffer of size n is set, where n is the number of boxes that can be stored, and information of n boxes is obtained, where each box can be represented as a one-dimensional vector [l, w, h], where l is the length of the box, w is the width of the box, and h is the height of the box; in the order in which the boxes arrive, the information of the first box stored in the buffer is input into the neural network model to obtain decision information on the placement of the box, and to determine whether the box can be placed in the container. If so, the box is loaded into the container, and the next box to be loaded on the conveyor belt is placed at the buffer position where the box is located; otherwise, the next box is taken out of the buffer to determine whether it can be placed in the container; in the present invention, only one box needs to be placed at a time, so the setting of the buffer does not need to be added to the training stage. In the training stage, only one box to be loaded needs to be considered, which reduces the difficulty of training, and the setting of the buffer is added in the test. This method can be applied to any online three-dimensional packing problem to improve space utilization.

[0103] In the present invention, the specific steps of S3 are as follows:

[0104] S301: Input the information of the first box to be loaded into the fully connected neural layer of the feature extraction network for encoding to obtain box encoding information;

[0105] S302: Inputting the container information for packing into the convolutional neural layer of the feature extraction network for encoding to obtain container encoding information;

[0106] S303: Add the box coding information and the container coding information, and input them into the direction strategy network and the value network to obtain the direction decision and the direction decision value respectively;

[0107] S304: according to the direction decision, obtain the index of the box placement direction, and obtain the information of the box after rotation according to the index correspondence;

[0108] S305: Input the information of the box rotation into the fully connected neural layer for encoding to obtain new box encoding information;

[0109] S306: Add the new box coding information to the container coding information and send them to the location strategy network and the value network to obtain the location decision and the location decision value respectively;

[0110] S307: According to the direction decision and the position decision, the index corresponding to the box placement direction and the index corresponding to the box placement position are obtained respectively, and they are combined to obtain a packing strategy. The neural network model includes: a feature extraction network, a direction strategy network, a position strategy network and a value network.

[0111] In the present invention, if Figure 4As shown, the neural network model in S3 includes a feature extraction network, a direction policy network, a location policy network, and a value network. The feature extraction network includes a convolutional neural layer binstateencoder and a fully connected neural layer boxstateencoder. The convolutional neural layer is used to encode container information, where inputs is the input of each layer of the network, outputs is the output of each layer, Conv2d is the convolutional layer, relu is the activation function used, which can increase the nonlinearity of the network, Flatten is the flattening layer, which can flatten the multidimensional array into a one-dimensional vector, and finally obtain the container encoding information bin_state_features after processing through multiple convolutional layers; the fully connected neural layer is used to encode the box information to be loaded, where Linear is the linear layer, and the box encoding information box_state_features to be loaded is obtained after processing through multiple linear layers; the policy network includes a direction policy network actor1 and a location policy network actor2. The direction policy network is used to predict the direction in which the box is placed, and the location policy network is used to predict the location in which the box is placed. Among them, softmax is the activation function. After softmax processing, the original output of the network can be converted into a form representing probability distribution. Therefore, the direction policy network actor1 and the position policy network actor2 output action probability distribution action_distributions respectively. The value network critic is used to update the value function, and the evaluation strategy outputs the evaluation value value, which is used to optimize the strategy. The direction policy network actor1 and the position policy network actor2 share a value network critic for evaluation.

[0112] The direction strategy network and the position strategy network output the corresponding probability according to each action. The direction strategy network allows the box to rotate in 6 directions, such as Figure 5 As shown, k represents the index corresponding to each direction, which can be k=0, k=1, k=2, k=3, k=4 and k=5. l, w and h represent the length, width and height of the box in the initial state, respectively. l′, w′ and h′ represent the length, width and height of the box after rotation, respectively. The direction strategy network outputs the probabilities of the six directions of the box. According to the direction corresponding to the maximum probability, the direction of the box is adjusted to obtain a box with accurate direction. The position strategy network outputs an integer within the range of 0 to the bottom area of ​​the container. According to the height map of the container for packing and the certificate output by the position strategy network, the position where the box can be placed in the height map is selected, and the box with accurate direction is placed to complete the packing operation.

[0113] In one embodiment of the present invention, a trust region actor-critic algorithm based on Kronecker factor approximate curvature is selected as the reinforcement learning algorithm of the neural network model in the present invention, and the algorithm is mainly divided into three parts: position strategy and direction strategy update, loss function update of the value before and after the box is rotated, and trust region optimization of the neural network model;

[0114] Position policy and direction policy update: By performing actions in the environment, collecting trajectory data, and calculating the advantage function, the model can measure the performance advantage of the model relative to the average level when performing actions. Then maximize the objective function for the policy parameters, which includes the expected return of the policy and a penalty term. In the direction policy network, through the policy Select the direction to place the box. In the location strategy network, through the strategy Select the location where the box is placed. Therefore, the expressions of the loss functions for the direction strategy and position strategy update are as follows:

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121] Among them, Orientation loss OriEntropy is the policy gradient loss function of the direction policy network. loss is the policy entropy regularization term of the direction policy network, Position loss is the policy gradient loss function of the position policy network, PosEntropy loss is the policy entropy regularization term of the location policy network, τ is a sample trajectory generated under the current direction policy, θ1 is the update parameter of the direction policy network, is the current direction strategy, is the expected cumulative reward of the current direction strategy, is the expectation of trajectory τ, a t is the action sampled at time t, s t is the state at time t, a t |s t is the action sampled at the state at time t, θ2 is the update parameter of the direction strategy network, is the expected cumulative reward of the current position strategy, is the current location strategy, is the expectation of trajectory τ, θ v is the update parameter of the value network, A(s t ,a t ;θ,θ v ) is the generalized advantage estimation function, is the TD time difference error, θ is the update parameter of the policy network, γ is the discount factor, λ is the hyperparameter, t is a certain time step, T is the length of the trajectory collected when updating the network parameters, and l is the step increment starting from time step t; after the introduction of generalized advantage estimation, the n l-step estimators starting from time t can be weighted averaged, λ is used to balance the deviation and variance, and in the simulation environment, λ=0.95 can be taken.

[0122] Update the loss function of the value of the box before and after rotation: Collect trajectory data, including the state and the corresponding reward, and use the trajectory data to update the value network to more accurately estimate the value of the state. The expressions of the loss function of the value of the state before and after the box is rotated are as follows:

[0123]

[0124]

[0125] Among them, OriValue loss PosValue is the loss function used to update the parameters of the direction strategy network and the value network. loss is the loss function used to update the parameters of the location policy network and the value network.

[0126] Trust region optimization of neural network models: Use the Kronecker factorized trust region method to ensure that the update parameters are not too large to prevent instability during training. The ACKTR algorithm uses the second-order optimization algorithm natural gradient method to update the model, improve the optimization direction of the objective function, and thus improve sample utilization. However, because the natural gradient method consumes more computational resources and needs to ensure update efficiency, the ACKTR algorithm uses the efficient K-FAC algorithm to estimate the natural gradient, reducing the amount of computational matrix inversion, and can approximate the Fisher information matrix in a short time. The total loss function used to update the parameters of the neural network model is Totalloss, and its expression is as follows:

[0127] Total loss =(Orientation loss +Position loss )+α×(OriValue loss +PosValue loss)+β×(OriEntropy loss +PosEntropy loss )

[0128] Among them, α and β are both hyperparameters. In the simulation environment, α=0.5 and β=0.01 can be taken.

[0129] In one embodiment of the present invention, the performance of the embodiment is evaluated. In a simulation environment, given that the size of the packing container is 10×10×10, and the length, width and height of the box are between [2, 5], a series of boxes of different sizes are randomly generated as test data. ows The simulation training is carried out on the computer of the system, which is equipped with an Intel Xeon W-2135 processor (3.70GHz, 16GB memory, 6 cores), an NVIDIA GeForce GTX 1080Ti graphics card, and the programming language is Python. The deep learning network is completed under the Pytorch framework.

[0130] In this embodiment, a constrained deep reinforcement learning algorithm is used as a baseline algorithm for comparison with the present invention. Hereinafter, the baseline algorithm model refers to the constrained deep reinforcement learning algorithm, Proposed (not considering rotation) refers to the model trained by considering only one rotation direction, and Proposed (adding 6 rotation directions) refers to the model trained by considering 6 rotation directions.

[0131] 100 box sequences were tested and the average results were calculated. The model was evaluated using four evaluation indicators, namely space utilization, average number of boxes, time per sequence, and time per box. For space utilization and average number of boxes, larger values ​​indicate better performance, while for time per sequence and time per box, smaller values ​​indicate better performance.

[0132] As shown in Table 1, the test results of the three models without considering the buffer zone are shown. Without considering the rotation direction, the space utilization of the proposed (without considering the rotation) algorithm is improved compared with the baseline algorithm, and the test time is greatly reduced. The space utilization of the proposed (adding 6 rotation directions) algorithm is significantly improved, and the test time is faster than the baseline algorithm, but lower than the proposed (without considering the rotation). The algorithm of this embodiment is superior to the baseline algorithm in all four metrics, which clearly proves the advantages of the algorithm of this embodiment.

[0133] Table 1

[0134]

[0135] As shown in Table 2, Table 3, and Table 4, they are the test results of adding a buffer to the baseline algorithm, the test results of adding a buffer to the proposed algorithm (without considering rotation), and the test results of adding a buffer to the proposed algorithm (adding 6 rotation directions). By comparison, it can be seen that after adding a buffer, the space utilization of the three models has been significantly improved compared to that without adding a buffer. In terms of test time, the average test time for each sequence is less than 1 second, which meets the actual application requirements. The above results show that the method of adding a buffer proposed in this embodiment is effective and has obvious advantages.

[0136] Table 2

[0137]

[0138] Table 3

[0139]

[0140] Table 4

[0141]

[0142] In order to more intuitively display the model test results and conduct qualitative analysis on the results, we visualize the test results of different models. First, without considering the buffer zone, we test the algorithm model proposed by Zhao et al., the algorithm model without adding the rotation direction proposed in this embodiment, and the algorithm model with 6 rotation directions. The best test results are selected for visualization, as shown in Figure 2. Figure 6 , Figure 7 , Figure 8 As shown, the packing result diagram of the baseline algorithm without considering the buffer zone, the packing result diagram of the Proposed algorithm (without considering rotation) without considering the buffer zone, and the packing result diagram of the Proposed algorithm (adding 6 rotation directions) without considering the buffer zone are shown. The space utilization rates achieved are 73.1%, 77.9% and 84.9% respectively. It can be seen that the algorithm proposed in this embodiment has a significant improvement over the baseline algorithm. From the visualization results, after adding 6 rotation directions, a higher space utilization rate can be achieved. At the same time, the stacking formed after the packing is completed will be more orderly, so it has better stability.

[0143] like Fig. 9 , Fig.10 , Fig.11As shown in the figure, the results of the baseline algorithm considering the buffer zone for packing, the Proposed algorithm (not considering rotation) considering the buffer zone for packing, and the Proposed algorithm (adding 6 rotation directions) considering the buffer zone for packing. The buffer size of the three models is 4. The best test results are selected for visualization, and the space utilization rates achieved are 77.1%, 78.2%, and 89.3% respectively. It can be seen that after adding the buffer zone, the boxes are stacked more compactly and can achieve higher space utilization.

[0144] The beneficial effects of the present invention are as follows: by adding a buffer zone, the present invention can make network training easier, reduce time cost, and significantly improve space utilization; at the same time, by adopting a neural network model, a broad advantage estimation algorithm is added on the basis of a trust region actor-critic algorithm ACKTR based on Kronecker factor approximate curvature, which can reduce deviations, make network training more stable, and have stronger generalization ability and versatility; when placing a box, the present invention considers the rotation of the box in the 6-dimensional direction, and uses a separate direction strategy network to select the direction, which can avoid a significant increase in the number of actions when selecting the box position, resulting in difficulty in network convergence; in addition, by separately representing the states of the box and the container, the present invention can use a lighter network for feature extraction, so that the model parameters obtained by training are fewer, while ensuring space utilization, improving test efficiency.

Claims

1. An online three-dimensional boxing method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: According to the trust region actor-critic algorithm ACKTR based on Kronecker factor approximate curvature and the generalized advantage estimation algorithm, a neural network model is established, and training is performed using training boxes and packing containers to obtain a trained neural network model; S2: Set a buffer zone to sequentially store a number of boxes to be loaded, and obtain information of a number of boxes to be loaded and information of containers for loading; S3: Based on the information of the first box to be packed and the information of the packing container, the trained neural network model is used to make predictions to obtain a packing strategy; S4: According to the packing strategy, determine whether the box meets the static stability and boundary constraints. If so, perform the packing operation, obtain a new box to be packed and store it in the buffer, and return to S3. Otherwise, predict the packing strategy of the next box according to the storage order of the boxes to be packed in the buffer, and enter S5. S5: Determine whether several boxes to be packed in the buffer cannot meet the static stability and boundary constraints. If so, end the packing. Otherwise, return to S4. The specific steps of S1 are as follows: S101: Establish a neural network model based on the trust region actor-critic algorithm ACKTR based on Kronecker factor approximate curvature and the generalized advantage estimation algorithm; S102: According to the training boxes and the packing containers, a neural network model is used to make predictions to obtain a packing strategy; S103: Perform a packing operation according to the packing strategy and calculate an instant reward for the packing operation; S104: Based on the immediate reward, the expected value of the cumulative reward of the neural network model is calculated, and it is determined whether the expected value of the cumulative reward reaches the maximum. If so, the training is completed to obtain the trained neural network model. Otherwise, the network parameters of the neural network model are optimized and returned to S102. The neural network model includes: feature extraction network, direction strategy network, position strategy network and value network.

2. The online three-dimensional boxing method based on deep reinforcement learning according to claim 1 is characterized in that: The box information is a one-dimensional vector, including the length, width and height of the box, and the container information for packing is a height map of the container from top to bottom, and the height map is a two-dimensional matrix.

3. The online three-dimensional packing method based on deep reinforcement learning according to claim 1, characterized in that: The neural network model includes: a feature extraction network, a direction strategy network, a position strategy network and a value network; The feature extraction network is used to extract the coding information in the box information and the coding information in the container information for packing; The direction strategy network is used to predict the direction of box placement based on the box encoding information and the container encoding information, and obtain a direction decision; The location strategy network is used to predict the location of the box according to the box coding information and the container coding information, and obtain the location decision; The value network is used to evaluate the direction decision and the position decision respectively to obtain the direction decision value and the position decision value.

4. The online three-dimensional boxing method based on deep reinforcement learning according to claim 1, characterized in that: The expression of the expected value of the cumulative reward is as follows: Among them, J(π) is the expected value of the cumulative reward, which represents the cumulative packing space utilization after multiple packing operations, γ is the discount factor, T is the number of steps to collect each trajectory, and E τ～π () is the expected value calculated by accumulating and adding the instant rewards at each moment on multiple trajectories, τ represents a trajectory formed after a series of state transitions are completed according to the packing strategy π, R(s t ,a t ) is the instant reward calculated at time t, s t is the state of the trajectory at time t, a t is the action selected by the trajectory at time t, R is the immediate reward, which represents the increased packing space utilization after each packing operation, and l i is the length of the i-th box, w i is the width of the ith box, h i is the height of the i-th box, L is the length of the container, W is the width of the container, H is the height of the container, and i is the i-th box.

5. The online three-dimensional boxing method based on deep reinforcement learning according to claim 1, characterized in that: The specific steps of S3 are as follows: S301: Input the information of the first box to be loaded into the fully connected neural layer of the feature extraction network for encoding to obtain box encoding information; S302: Inputting the container information for packing into the convolutional neural layer of the feature extraction network for encoding to obtain container encoding information; S303: Add the box coding information and the container coding information, and input them into the direction strategy network and the value network to obtain the direction decision and the direction decision value respectively; S304: according to the direction decision, obtain the index of the box placement direction, and obtain the information of the box after rotation according to the index correspondence; S305: Input the information of the box rotation into the fully connected neural layer for encoding to obtain new box encoding information; S306: Add the new box coding information to the container coding information and send them to the location strategy network and the value network to obtain the location decision and the location decision value respectively; S307: According to the direction decision, the direction decision value, the position decision and the position decision value, respectively obtain the index corresponding to the box placement direction and the index corresponding to the box placement position, and combine them to obtain the packing strategy.

6. The online three-dimensional packing method based on deep reinforcement learning according to claim 1, characterized in that: Whether the static stability is satisfied includes: Whether the supported area of ​​the bottom of the box is greater than a first threshold of the bottom area of ​​the box; Whether the supported area of ​​the bottom of the box is greater than the second threshold of the bottom area of ​​the box, and three corners of the bottom of the box are supported; Whether the supported area of ​​the bottom of the box is greater than the third threshold of the bottom area of ​​the box, and the four corners of the bottom of the box are supported.

7. The online three-dimensional boxing method based on deep reinforcement learning according to claim 6, characterized in that: Whether the boundary constraint condition is satisfied includes: Constrain the box in three directions: length, width and height, and determine whether any edge of the box exceeds the boundary of the container when it is placed in the container. If so, it means that the box cannot be placed in the container; otherwise, the box can be placed in the container.

8. The online three-dimensional packing method based on deep reinforcement learning according to claim 7, characterized in that: The steps of packing in S4 are as follows: B1: According to the packing strategy, determine whether the placement of the boxes meets the static stability and boundary constraints; B2: According to the direction strategy in the box packing strategy, the box is rotated in 6 dimensions to obtain a box with accurate direction; B3: According to the location strategy in the packing strategy, select a box with accurate location and orientation in the packing container to complete the packing operation.

Images