An Automated Bin Packing Method Based on Reinforcement Learning and Dynamic Search

Abstract

This invention relates to the fields of automated packaging and logistics warehousing applications, and discloses an automated packing method based on reinforcement learning and dynamic search. This method addresses the difficulties and shortcomings of traditional algorithms when applied to automated online packing scenarios. It combines the advantages of reinforcement learning (which continuously adjusts its strategy based on environmental feedback to achieve automated learning) with the fast solution speed and strong adaptability of dynamic search algorithms. While ensuring maximum utilization of box space, it achieves the goal of quickly solving and optimizing the stacking actions of goods. Furthermore, during the stacking process, the algorithm can continuously adjust its stacking actions and positions based on real-time feedback from the box and the environment, achieving the effect of automatically learning the optimal stacking strategy and providing real-time online guidance for packing.

Description

Technical Field

[0001] This invention relates to the field of automated packaging and logistics warehousing applications, specifically to an automated packing method based on reinforcement learning and dynamic search. Background Technology

[0002] Three-dimensional packing refers to placing a certain number of small items into one or more boxes with a large capacity, under conditions such as volume constraints, geometric constraints, and stability constraints, in order to achieve the goal of using the fewest number of boxes, maximizing space utilization, ensuring the best stability, and maximizing the loading value and weight ratio.

[0003] 3D packing is widely used in the loading process of materials in industries such as transportation and packaging. Therefore, designing a 3D packing solution algorithm with high operability and loading efficiency to meet practical application needs is of great practical significance for reducing logistics and distribution costs and improving economic benefits.

[0004] Due to the numerous challenges in automated online packing scenarios, such as a huge solution space, high uncertainty, and complex and variable scenario logic, traditional algorithms tend to suffer from slow solution speed, low optimization efficiency, and inability to adapt to the diversity of boxes and goods when applied to automated online packing scenarios, thus failing to meet the requirements of automated online packing. Summary of the Invention

[0005] To address the difficulties and shortcomings of applying traditional algorithms to automated online packing scenarios, this invention combines the characteristics of reinforcement learning—which can continuously adjust its own strategy based on environmental feedback to achieve automated learning—with the advantages of dynamic search algorithms—which offer fast solution speed and strong adaptability—and proposes an automated packing method based on reinforcement learning and dynamic search.

[0006] An automated bin packing method based on reinforcement learning and dynamic search includes the following steps:

[0007] Step S1: Count the items to be stacked in the same box, record the quantity as m, and assign each item in the box a number I, where number i satisfies the following relationship:

[0008] i∈m

[0009] i = i + 1

[0010] Step S2: Begin stacking goods in the cargo box and determine if logic L1 is true.

[0011] Specifically, logic L1 refers to the boundary collision logic between the goods and the cargo box, and the calculation formula is as follows:

[0012] x′ i ≤L,y′ i ≤W,z′i ≤H,i=1,2,3…,n

[0013] Where L, W, and H represent the length, width, and height of the cargo box, respectively, and i represents the item number. i ,y i ,z i (x′) represents the lower right rear three-dimensional coordinates of the goods after they have been placed in the cargo box. i ,y′ i ,z′ i () represents the upper left three-dimensional coordinates after the goods are placed in the cargo box;

[0014] Furthermore, if the result of logic L1 is false, proceed to the next step; otherwise, execute action D1.

[0015] Specifically, action D1 refers to changing the stacking posture of the goods. There are a total of 6 possible postures when stacking goods. Each rotation uses the length-width, length-height, and width-height planes of the goods as the base, rotating 90 degrees clockwise. Each rotation updates the posture of the goods and records the rotation action a. i Store in the knowledge base.

[0016] Step S3: Continue stacking goods and check if logic L2 is true.

[0017] Specifically, logic L2 refers to the overlapping judgment logic between goods, and the calculation formula is:

[0018]

[0019]

[0020] Where, x′ i ,y′ i ,z′ i Let x' represent the three-dimensional coordinates of the current item's left front top x-axis, y-axis, and z-axis, respectively. i+1 ,y′ i+1 ,z′ i+1 These represent the three-dimensional coordinates of the newly placed item on the upper left front, along the x, y, and z axes, respectively. T i This indicates the contact area of ​​item i, meaning that at least half of the bottom area of ​​the item must be supported and it must not be stacked in mid-air.

[0021] Furthermore, if the result of logic L2 is false, proceed to the next step; otherwise, execute action D2.

[0022] Specifically, action D2 refers to changing the stacking position of the goods, and the process is as follows:

[0023] First, a luggage stacking state matrix G is established. Before each change in the stacking position of goods, a dynamic search algorithm is used to calculate the three-dimensional coordinates of the stackable goods.

[0024] The dynamic search algorithm refers to a method that aims to maximize the utilization of the container for loading goods. It uses dynamic programming to achieve this goal. The specific steps are as follows:

[0025] Step A1: Define P as the composite layer of goods formed by stacking them in the same orientation. Calculate V based on the given P. act :

[0026]

[0027]

[0028] Among them, V act It is the difference between the defined volume and the actual volume, representing the remaining stacking space in the container. Indicates the fill rate of the composite stacking layer of goods. H represents the minimum outer rectangle dimension corresponding to the bottom surface of the composite goods, and H is the height formed after stacking.

[0029] Step A2: Invoke the dynamic programming algorithm to generate a new round of P′ and calculate the corresponding V′. act ;

[0030] Step A3, select V′ act Minimum stacking location a j Place the goods at location a j The corresponding three-dimensional coordinate information is used as the parameter of the update matrix G, and the stacked goods position a is set. j Store in the knowledge base.

[0031] The knowledge base stores the rotating goods action a i and the location of stacked goods a j This is used as training data to generate a reinforcement learning model, which is then used to initialize subsequent goods stacking strategies. The specific steps are as follows:

[0032] Step B1: Set the objective function as the learning objective of the agent, specifically, to maximize the total space utilization of the boxes (maxf) and minimize the remaining stacking space (V). act Assuming the objective function is used, the formula for calculating the total space utilization rate of the enclosure is:

[0033]

[0034] Among them, l i w i h iLet L, W, and H represent the length, width, and height of item i, respectively, and let L, W, and H represent the length, width, and height of the box, respectively.

[0035] Step B2, set the reward function to evaluate the result of the action, and calculate the formula as follows:

[0036] R = D + ω(maxf, Vact)

[0037] Where ω is the weighting coefficient, representing the degree of importance attached to the two rewards, and D represents the distance from the goods to the top of the cargo box;

[0038] Step B3 employs an iteratively computeable Q-function to guide the agent's action strategy during the box stacking process. Through continuous trial and error, the agent approximates the objective function, thereby obtaining the optimal box stacking action. The specific calculation formula is as follows:

[0039] Q(s,a)=Q(s,a)+α[R+γMax Q(s′,a)-Q(s,a)]

[0040] a = a i +a j

[0041] Where 'a' represents the action space of the intelligent agent, including the action of rotating the goods. i and the location of stacked goods a j Q(s,a) represents the update parameter, α is the learning factor, R is the reward function, γ is the discount factor, which represents the importance attached to future rewards, and s' is the state after performing action a in the state of goods s;

[0042] Step B4, obtain the optimal rotating goods action a i and the location of stacked goods a j The data is used to train a reinforcement learning model.

[0043] Next, after querying matrix G and confirming that the coordinates are executable, the parameters in matrix G are updated. Finally, the action of changing the stacking position of the goods is executed.

[0044] The formula for matrix G is:

[0045]

[0046] Among them, (x i ,y i ,z i ) represents the coordinates of the optional stacking location for item number i.

[0047] Step S4: Determine whether all goods have been placed. If the determination is true, the goods have been stacked. Otherwise, proceed to step S2.

[0048] This invention proposes an automated bin packing method based on reinforcement learning and dynamic search, which has the following advantages compared with existing technologies:

[0049] This invention introduces a dynamic search algorithm during the container stacking process to calculate the optimal goods stacking position in real time, thereby achieving the goal of quickly solving and optimizing goods stacking actions while ensuring the highest utilization of container space.

[0050] This invention introduces reinforcement learning into 3D box packing, enabling the algorithm to continuously adjust its stacking actions and positions based on real-time feedback from the box and the environment during the stacking process. This achieves the effect of automatically learning the optimal stacking strategy and providing online real-time guidance for box packing. Attached Figure Description

[0051] Figure 1 This is the overall flowchart of the present invention;

[0052] Figure 2 This is a flowchart of the dynamic search algorithm of the present invention. Detailed Implementation

[0053] To make the objectives and features of this invention more apparent and understandable, the technical solution will be described in detail below through embodiments and in conjunction with the accompanying drawings.

[0054] like Figure 1 As shown, an automated bin packing method based on reinforcement learning and dynamic search includes the following steps:

[0055] Step S1: Count the items to be stacked in the same box, record the quantity as m, and assign each item in the box a number I, where number i satisfies the following relationship:

[0056] i∈m

[0057] i = i + 1

[0058] Step S2: Start stacking goods in the cargo box and determine whether the boundary collision logic L1 between the goods and the cargo box is valid. The specific calculation formula is as follows:

[0059] x′ i ≤L,y′ i ≤W,z′ i ≤H,i=1,2,3…,n

[0060] Where L, W, and H represent the length, width, and height of the cargo box, respectively, and i represents the item number. i ,y i ,z i (x′) represents the lower right rear three-dimensional coordinates of the goods after they have been placed in the cargo box. i ,y′i ,z′ i () represents the three-dimensional coordinates of the upper left front of the cargo box after the goods are placed inside.

[0061] Furthermore, if the result of logic L1 is false, proceed to the next step; otherwise, execute action D1 to change the stacking posture of the goods. There are six possible stacking postures. Each rotation uses the length-width, length-height, and width-height planes of the goods as the base, rotating 90 degrees clockwise. Each rotation updates the posture of the goods and records the rotation action a. i Store in the knowledge base.

[0062] Step S3: Continue stacking goods and determine whether the overlap judgment logic L2 between goods is true. The specific calculation formula is as follows:

[0063]

[0064]

[0065] Where, x′ i ,y′ i ,z′ i Let x' represent the three-dimensional coordinates of the current item's left front top x-axis, y-axis, and z-axis, respectively. i+1 ,y′ i+1 ,z′ i+1 These represent the three-dimensional coordinates of the newly placed item on the upper left front, along the x, y, and z axes, respectively. T i This indicates the contact area of ​​item i, meaning that at least half of the bottom area of ​​the item must be supported and it must not be stacked in mid-air.

[0066] Furthermore, if the result of logic L2 is false, the next step is executed; otherwise, action D2, which changes the stacking position of the goods, is executed, as follows:

[0067] First, a luggage stacking state matrix G is established. Before each action to change the stacking position of goods, the three-dimensional coordinates of the stackable goods are calculated using a dynamic search algorithm.

[0068] like Figure 2 As shown, the dynamic search algorithm aims to maximize the utilization of the container for loading goods. It uses dynamic programming to achieve this goal. The specific steps are as follows:

[0069] Step A1: Define P as the composite layer of goods formed by stacking them in the same orientation. Calculate V based on the given P. act .

[0070]

[0071]

[0072] Among them, V act It is the difference between the defined volume and the actual volume, representing the remaining stacking space in the container. Indicates the fill rate of the composite stacking layer of goods. H represents the minimum outer rectangle dimension corresponding to the bottom surface of the composite goods, and H represents the height formed after stacking.

[0073] Step A2: Invoke the dynamic programming algorithm to generate a new round of P′ and calculate the corresponding V′. act .

[0074] Step A3, select V′ act Minimum stacking location a j Place the goods at location a j The corresponding three-dimensional coordinate information is used as the parameter of the update matrix G, and the stacked goods position a is set. j Store in the knowledge base.

[0075] The knowledge base is used to train a reinforcement learning model, that is, by storing the action of rotating the goods a. i and the location of stacked goods a j The training data is used to generate a reinforcement learning model, which is then used to initialize the subsequent goods stacking strategy. The specific steps are as follows:

[0076] Step B1: Set the objective function as the learning objective of the agent, specifically, to maximize the total space utilization of the boxes (maxf) and minimize the remaining stacking space (V). act Assuming the objective function is used, the formula for calculating the total space utilization rate of the enclosure is:

[0077]

[0078] Among them, l i w i h i Let L, W, and H represent the length, width, and height of item i, respectively, and let L, W, and H represent the length, width, and height of the box body, respectively.

[0079] Step B2, set the reward function to evaluate the result of the action, and calculate the formula as follows:

[0080] R = D + ω(maxf, Vact)

[0081] Where ω is the weighting coefficient, representing the degree of importance attached to the two rewards, and D represents the distance from the goods to the top of the cargo box.

[0082] Step B3 employs an iteratively computeable Q-function to guide the agent's action strategy during the box stacking process. Through continuous trial and error, the agent approximates the objective function, thereby obtaining the optimal box stacking action. The specific calculation formula is as follows:

[0083] Q(s,a)=Q(s,a)+α[R+γMax Q(s′,a)-Q(s,a)]

[0084] a = a i +a j

[0085] Where 'a' represents the action space of the intelligent agent, including the action of rotating the goods. i and the location of stacked goods a j Q(s,a) represents the update parameter, α is the learning factor, R is the reward function, γ is the discount factor, which represents the importance attached to future rewards, and s' is the state after performing action a in the state of goods s.

[0086] Preferably, the γ value is set between 0.2 and 0.8.

[0087] Step B4, obtain the optimal rotating goods action a i and the location of stacked goods a j The data is used to train a reinforcement learning model and initialize subsequent product stacking strategies.

[0088] Next, after querying matrix G and confirming that the coordinates are executable, the parameters in matrix G are updated. The formula for matrix G is:

[0089]

[0090] Among them, (x i ,y i ,z i ) represents the coordinates of the optional stacking location for item number i.

[0091] Finally, the action of changing the stacking position of the goods is carried out.

[0092] Step S4: Determine whether all goods have been placed. If the determination is true, the goods have been stacked. Otherwise, proceed to step S2.

[0093] Specifically, it is determined whether the container has been filled with all the goods by judging whether the number i of the currently stacked goods is equal to the counted quantity m of the goods.

[0094] Thus, the working process of this invention has been completed once according to the method disclosed in this invention.

[0095] Although the present invention has been described in detail in this specification with general description and specific embodiments, some modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention are within the scope of protection claimed by the present invention.

Claims

1. An automated bin packing method based on reinforcement learning and dynamic search, comprising the following steps: Step S1: Count the items to be stacked in the same box, record the quantity as m, and assign each item in the box a number i. Step S2: Start stacking goods in the cargo box and determine whether logic L1 is true. If the result of logic L1 is false, proceed to the next step; otherwise, execute action D1. Step S3: Continue stacking goods and determine whether logic L2 is true. If the result of logic L2 is false, proceed to the next step; otherwise, execute action D2. Step S4: Determine whether all goods have been placed. If the determination is true, the goods have been stacked. Otherwise, proceed to step S2. In step S2, logic L1 refers to the boundary collision logic between the goods and the cargo box, and the specific calculation formula is as follows: x′ i ≤L,y′ i ≤W,z′ i ≤H,i=1,2,3…,n in, L, W, and H represent the length, width, and height of the cargo box, respectively, and i represents the item number. i ,y i ,z i (x') represents the lower right rear three-dimensional coordinates of the goods after they have been placed in the cargo box. i ,y' i ,z' i () represents the upper left three-dimensional coordinates after the goods are placed in the cargo box; In step S2, logic L2 refers to the logic for determining overlap between goods, and the specific calculation formula is as follows: Where, x' i ,y' i ,z' i These represent the three-dimensional coordinates of the current item's left front top x-axis, y-axis, and z-axis, respectively. i+1 ,y' i+1 ,z' i+1 These represent the three-dimensional coordinates of the newly placed item on the upper left front, along the x, y, and z axes, respectively. T i This indicates the contact area of ​​item i, meaning that at least half of the bottom area of ​​the item must be supported and it must not be stacked in mid-air.

2. In the automated packing method based on reinforcement learning and dynamic search according to claim 1, the item number i in step S1 satisfies the following relationship: i∈m i = i + 1.

3. The automated packing method based on reinforcement learning and dynamic search according to claim 1, wherein action D1 in step S2 refers to changing the stacking posture of the goods. Specifically, there are 6 possible stacking postures. Each rotation uses the length-width, length-height, and width-height planes of the goods as the base, rotating 90 degrees clockwise. Each rotation updates the posture of the goods, and the rotating action a is recorded. i Store in the knowledge base.

4. In the automated packing method based on reinforcement learning and dynamic search according to claim 1, action D2 in step S2 refers to changing the stacking position of the goods. Specifically, a baggage stacking state matrix G is established. Before each action to change the stacking position of the goods, the three-dimensional coordinates of the stackable goods are calculated using a dynamic search algorithm. After confirming that the coordinates are executable by querying matrix G, the parameters in matrix G are updated, and the action to change the stacking position of the goods is executed. The formula for matrix G is: in, (x i ,y i ,z i ) represents the coordinates of the optional stacking location for item number i.

5. The automated packing method based on reinforcement learning and dynamic search according to claim 4, wherein the dynamic search algorithm refers to using a dynamic programming algorithm to maximize the utilization of the packing volume with the goal of maximizing the use of the packing box for loading goods, and the specific steps include: Step A1: Define P as the composite layer of goods formed by stacking them in the same orientation. Calculate V based on the given P. act : Among them, V act It is the difference between the defined volume and the actual volume, representing the remaining stacking space in the container. Indicates the fill rate of the composite stacking layer of goods. H represents the minimum outer rectangle dimension corresponding to the bottom surface of the composite goods, and H is the height formed after stacking. Step A2: Invoke the dynamic programming algorithm to generate a new round of P' and calculate the corresponding V'. act ; Step A3, select V' act Minimum stacking location a j Place the goods at location a j The corresponding three-dimensional coordinate information is used as the parameter of the update matrix G, and the stacked goods position a is also used. j Store in the knowledge base.

6. The automated packing method based on reinforcement learning and dynamic search according to claim 3 or claim 5, wherein the knowledge base stores the rotating goods action a. i and the location of stacked goods a j This is used as training data to generate a reinforcement learning model, which is then used to initialize subsequent goods stacking strategies. The specific steps include: Step B1: Set the objective function as the learning objective of the agent, specifically, to maximize the total space utilization of the boxes (maxf) and minimize the remaining stacking space (V). act Assuming the objective function is used, the formula for calculating the total space utilization rate of the enclosure is: Among them, l i w i h i Let L, W, and H represent the length, width, and height of item i, respectively, and let L, W, and H represent the length, width, and height of the box body, respectively. Step B2, set the reward function to evaluate the result of the action, and calculate the formula as follows: R = D + ω(maxf, Vact) Where ω is the weighting coefficient, representing the degree of importance attached to the two rewards, and D represents the distance from the goods to the top of the cargo box; Step B3 employs an iteratively computeable Q-function to guide the agent's action strategy during the box stacking process. Through continuous trial and error, the agent approximates the objective function, thereby obtaining the optimal box stacking action. The specific calculation formula is as follows: Q(s,a)=Q(s,a)+α[R+γMax Q(s′,a)-Q(s,a)] a=a i +a j Where 'a' represents the action space of the intelligent agent, including the action of rotating the goods. i and the location of stacked goods a j Q(s,a) represents the update parameter, α is the learning factor, R is the reward function, γ is the discount factor, which represents the importance attached to future rewards, and s' is the state after performing action a in the state of goods s; Step B4, obtain the optimal rotating goods action a i and the location of stacked goods a j The data is used to train a reinforcement learning model.

Images