Dynamic candidate boxing method and system for large-scale three-dimensional boxing problem

Abstract

The invention belongs to the technical field of three-dimensional encasement, and discloses a dynamic candidate encasement method and system for a large-scale three-dimensional encasement problem, the three-dimensional encasement problem is modeled as a Markov decision process, and an encasement Q network and a candidate Q network which take Transform as a core are constructed. And the box selection module is responsible for selecting boxes from the processing set for boxing and dynamically selecting boxes from the candidate set for adding the boxes into the processing set. The encasement Q network is optimized through reinforcement learning in the training stage, and the dynamic candidate Q network directly multiplexes the trained encasement Q network parameters without additional training. According to the method, the box body is divided into the processing set and the candidate set, so that the calculation complexity of a large-scale boxing problem is effectively reduced, and meanwhile, a relatively high space utilization rate is kept. When large-scale instances are processed, compared with an existing method, the method has the remarkable advantage in the aspect of the space utilization rate, and the method is particularly suitable for meeting the efficient encasement requirement in logistics transportation and warehouse management.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional packing technology, and in particular to a dynamic candidate packing method and system for large-scale three-dimensional packing problems. Background Technology

[0002] The three-dimensional bin packing problem (3D-BPP), a fundamental component of logistics, is a combinatorial optimization problem in operations research and computer science. It primarily explores how to pack a series of rectangular boxes of given dimensions into one or more rectangular containers of given dimensions under specific constraints, maximizing the space utilization of the containers. 3D-BPP has wide applications in container loading for road, rail, and water transport, as well as palletized cargo storage in warehousing logistics. Efficient and automatically generated packing strategies are crucial for enhancing logistics automation and improving overall efficiency.

[0003] As a multidimensional combinatorial optimization problem, 3D-BPP, due to its strongly NP-hard nature, is mathematically difficult to solve mathematically using exact algorithms within a given time. In the past, heuristic 3D bin packing methods have been a research hotspot in the field of 3D-BPP. These methods typically find approximate optimal solutions to the bin packing problem, with a much higher probability of finding feasible solutions within an acceptable time than exact algorithms. However, these methods are based on expert experience and intuition, and may be influenced by subjective consciousness and cognitive biases, making them difficult to adapt to different bin packing scenarios. In recent years, methods using deep reinforcement learning to solve 3D-BPP have received widespread attention. Deep reinforcement learning can dynamically adjust its strategy based on environmental feedback, thus adapting to diverse bin packing scenarios and typically possessing high solution efficiency. However, most existing deep reinforcement learning bin packing methods are limited by narrow exploration spaces and excessively high computational complexity, making it difficult to handle large-scale bin 3D-BPP. Even if a few methods can handle large numbers of bins, their space utilization performance remains unsatisfactory.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a dynamic candidate bin packing method and system for large-scale 3D bin packing problems. This aims to address the limitations of most existing deep reinforcement learning bin packing methods, which are constrained by narrow exploration spaces and excessively high computational complexity, making it difficult to handle large-scale 3D bin packing. Even those few methods that can handle a large number of bins still suffer from unsatisfactory space utilization.

[0006] To achieve the above object, the present invention provides a dynamic candidate packing method for large-scale three-dimensional packing problems. The dynamic candidate packing method for large-scale three-dimensional packing problems includes the following steps: Formalize the three-dimensional packing problem and initialize the problem instance of three-dimensional packing; Describe the three-dimensional packing problem as a Markov decision process, and formulate corresponding state representation, action representation, reward function, transition function and discount factor for the problem instance; Construct a packing Q-network with Transformer as the backbone, and iteratively train and update the packing Q-network with the problem instance of three-dimensional packing as the training set until the cumulative reward converges. Among them, the input of the packing Q-network is the state without the candidate set of boxes, and the output is the value of the packing action. The packing action is selected based on the magnitude of the value of the packing action; Construct a dynamic candidate Q-network with Transformer as the backbone. The dynamic candidate Q-network does not need to be trained, and its parameters are directly taken from the trained packing Q-network. Among them, the input of the dynamic candidate Q-network is the state including the candidate set of boxes, and the output is the value of the candidate action. The candidate action is selected based on the magnitude of the value of the candidate action; Deploy the trained packing Q-network and dynamic candidate Q-network models to the actual large-scale three-dimensional packing environment for outputting three-dimensional packing solutions.

[0007] In one embodiment, the formalization of the three-dimensional packing problem is to give a rectangular container C with a fixed length of L, a fixed width of W and an unrestricted height and N square boxes I = {1, 2,..., N}. The goal of the problem is to pack all the boxes in I into the container C and minimize the height of the container used to maximize the space utilization rate of the container. The constraint conditions satisfied during the packing process include that the boxes placed in the container cannot overlap in space; the boxes placed in the container cannot exceed the boundary of the container, and the boxes are placed orthogonally in the container. The orthogonal placement in the container means that the length, width or height of the box is parallel or perpendicular to the length, width or height of the container; the problem instance is initialized as a three-dimensional packing problem instance, and Nmax boxes are randomly selected from the N boxes in the box set I and placed in the processing set U proc , and the other boxes are all placed in the candidate set U cand , where the size of Nmax is limited by the computational efficiency. If N < Nmax, all boxes are placed in the processing set U proc , and the candidate set U cand is set to be empty.

[0008] In one embodiment, the state representation is that at any time step t, the state s t consists of the state of the processing box s t proc , the state of the candidate box st cand and container state s t bin It consists of three parts, s t proc The data records the three-dimensional dimensions of all the boxes in the processing center. t cand The three-dimensional dimension information of all boxes in the candidate set was recorded. t bin The system records the position and size information of all spaces within the container; the action is represented as action a for any time step t. t From the packing action a t pack and candidate action a t sel It consists of two parts, a t pack This indicates selecting a container from the processing set and determining the loading direction and loading space of the container. t sel This indicates that a box is selected from the candidate set and moved into the processing set; the reward function is that for any time step t, the reward r is... t =g t -g t+1 , where g t This represents the ratio of the unoccupied volume in the container to the total volume of the container at time step t; the transfer function is a deterministic transfer function P(s) for any time step t. t+1 |s t ,a t )=1, and the discount factor γ is equal to 1.

[0009] In one embodiment, the state that does not contain candidate bins is when, for any time step t, the binning Q-network input s t 1 ={s t proc s t bin}

[0010] In one embodiment, the step of constructing a binning Q-network with Transformer as the backbone, and iteratively training and updating the binning Q-network using 3D binning problem instances as the training set until the cumulative reward converges includes: Step S1: Extract the e-th problem instance from the training set; Step S2: Initialize time step t=0, initialize state s t ={s t proc s t cand s tbin} and the state s that does not include the subsequent selection box t 1 ={s t proc s t bin}; Step S3: Pack Q network according to s t 1 Calculate the value of all actions Q pack (s t 1 ,˙); Step S4: Based on the ε-greedy strategy, from Q pack (s t 1 Select and execute the packing action a) t pack :

[0011] Step S5: If the candidate set is not empty, randomly select a box from the candidate set and move it into the processing set; Step S6: Calculate the reward r t And update the status to s t+1 ; Step S7: Transfer the state transition tuple (s) t 1 a t pack r t s t+1 1 Store in the experience pool; Step S8: Sample a batch of transfer tuples (s) from the experience pool. 1 a pack ,r,s 1' ), calculate the loss and update the parameter θ:

[0012] Where b is the batch size of the sampling, θ — The target network parameters are updated to θ after every certain number of iterations; Step S9: If there are still boxes in the processing set, return to step S3; otherwise, e = e + 1. Step S10: If e is less than the number of instances in the training set, return to step S1; otherwise, end the training.

[0013] In one embodiment, the state containing the candidate set box is defined as follows: for any time step t, the candidate Q-network input s t 2 ={s t' procs t cand s t+1 bin}, where t' represents the time state between t and t+1, in which a bin in the processing set has been loaded, while bins in the candidate set have not yet been moved into the processing set.

[0014] Furthermore, to achieve the above objectives, this invention also proposes a dynamic candidate packing system for large-scale three-dimensional packing problems. This dynamic candidate packing system is applied to the dynamic candidate packing method for large-scale three-dimensional packing problems described above. The system includes: The problem definition and initialization module is used to formally define the 3D bin packing problem and initialize instances of the 3D bin packing problem. The Markov description module is used to describe the 3D bin packing problem as a Markov decision process, and to formulate corresponding state representations, action representations, reward functions, transition functions and discount factors for problem instances; The bin packing decision module is used to construct a bin packing Q-network with Transformer as the backbone, and iteratively train and update the bin packing Q-network with three-dimensional bin packing problem instances as the training set until the cumulative reward converges. The input of the bin packing Q-network is the state that does not contain candidate bins, and the output is the bin packing action value. The bin packing action is selected based on the magnitude of the bin packing action value. The dynamic candidate module is used to construct a dynamic candidate Q-network with Transformer as the backbone. The dynamic candidate Q-network does not require training and its parameters are directly taken from the trained bin-packed Q-network. The input of the dynamic candidate Q-network is the state containing the candidate set bins, and the output is the candidate action value. The candidate action is selected based on the magnitude of the candidate action value. The deployment module is used to deploy the trained bin packing Q-network and dynamic candidate Q-network models to a real-world, large-scale 3D bin packing environment to output a 3D bin packing solution.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a dynamic candidate packing device for a large-scale three-dimensional packing problem. The dynamic candidate packing device for a large-scale three-dimensional packing problem includes: a memory, a processor, and a dynamic candidate packing program for a large-scale three-dimensional packing problem stored in the memory and executable on the processor. The dynamic candidate packing program for a large-scale three-dimensional packing problem is configured to implement the steps of the dynamic candidate packing method for a large-scale three-dimensional packing problem as described above.

[0016] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a dynamic candidate bin packing program for a large-scale three-dimensional bin packing problem. When the dynamic candidate bin packing program for a large-scale three-dimensional bin packing problem is executed by a processor, it implements the steps of the dynamic candidate bin packing method for a large-scale three-dimensional bin packing problem as described above.

[0017] This invention models the 3D packing problem as a Markov decision process. It constructs a packing Q-network and a candidate Q-network, both centered around a Transformer, responsible for selecting boxes from the processing set for packing and dynamically selecting boxes from the candidate set to add to the processing set, respectively. The packing Q-network is optimized through reinforcement learning during the training phase, while the dynamic candidate Q-network directly reuses the trained parameters without additional training. This method effectively reduces the computational complexity of large-scale packing problems by dividing boxes into processing and candidate sets, while maintaining high space utilization. Compared to existing methods, this invention offers significant advantages in space utilization when handling large-scale instances, making it particularly suitable for efficient packing requirements in logistics transportation and warehousing management. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the dynamic candidate packing method for large-scale three-dimensional packing problems according to the present invention. Figure 2 This is a schematic diagram of the dynamic candidate packing framework structure in the dynamic candidate packing method for large-scale three-dimensional packing problems of the present invention; Figure 3 This is a schematic diagram of the Q-network structure of the dynamic candidate bin packing method for large-scale three-dimensional bin packing problems in this invention; Figure 4 This is a schematic diagram of the candidate Q-network structure in the dynamic candidate bin packing method for large-scale three-dimensional bin packing problems of the present invention; Figure 5 This is a structural block diagram of the first embodiment of the dynamic candidate packing system for large-scale three-dimensional packing problems of the present invention.

[0019] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0021] This invention provides a dynamic candidate bin packing method for large-scale three-dimensional bin packing problems, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of a dynamic candidate packing method for large-scale three-dimensional packing problems according to the present invention.

[0022] In this embodiment, the dynamic candidate bin packing method for large-scale three-dimensional bin packing problems includes the following steps: Step S100: Formalize the 3D bin packing problem and initialize instances of the 3D bin packing problem.

[0023] In this embodiment, the execution subject is a dynamic candidate packing device for large-scale three-dimensional packing problems. This dynamic candidate packing device for large-scale three-dimensional packing problems has functions such as data processing, data communication, and program execution. The dynamic candidate packing device for large-scale three-dimensional packing problems can be a computer terminal device or other network device, or other devices with similar functions. This embodiment does not limit the scope of the application.

[0024] It should be noted that the three-dimensional bin packing problem (3D-BPP), as a fundamental component of logistics, is a class of combinatorial optimization problems in operations research and computer science. It primarily explores how to pack a series of cuboid boxes of given dimensions into one or more cuboid containers of given dimensions under specific constraints to maximize container space utilization. 3D-BPP has wide applications in container loading in road, rail, and water transport, as well as palletized cargo storage in warehousing logistics. Efficient and automatically generated packing strategies are crucial for enhancing logistics automation and improving efficiency. As a multidimensional combinatorial optimization problem, 3D-BPP is mathematically NP-hard, making it difficult to find the optimal solution within a given time using exact algorithms. In the past, heuristic 3D bin packing methods have been a research hotspot in the field of 3D-BPP. These methods typically seek approximate optimal solutions to the packing problem, with a much higher probability of finding a feasible solution within an acceptable timeframe compared to exact algorithms. However, these methods are based on expert experience and intuition, which may be influenced by subjective consciousness and cognitive biases, making them difficult to adapt to different bin-packing scenarios. In recent years, methods using deep reinforcement learning to solve 3D-BPP have received widespread attention. Deep reinforcement learning can dynamically adjust its strategy based on environmental feedback, thereby adapting to diverse bin-packing scenarios and typically possessing high solution efficiency. However, most existing deep reinforcement learning bin-packing methods are limited by narrow exploration space and excessively high computational complexity, making it difficult to handle large-scale 3D-BPP bins. Even if a few methods can handle the case of a large number of bins, their space utilization performance remains unsatisfactory.

[0025] To solve the above technical problems, in this embodiment, the three-dimensional bin-packing problem is modeled as a Markov decision process. By constructing a bin-packing Q-network and a candidate Q-network with Transformer as the core, they are respectively responsible for selecting bins from the processing set for bin-packing and dynamically selecting bins from the candidate set to be added to the processing set. The bin-packing Q-network is optimized through reinforcement learning during the training phase, while the dynamic candidate Q-network directly reuses the parameters of the trained bin-packing Q-network without additional training. This method effectively reduces the computational complexity of the large-scale bin-packing problem by dividing the bins into a processing set and a candidate set, while maintaining a high space utilization rate. When dealing with large-scale instances, the present invention has significant advantages in terms of space utilization compared with existing methods, and is particularly suitable for the efficient bin-packing requirements in logistics transportation and warehouse management. Specifically, it can be implemented as follows.

[0026] In this embodiment, the overall structure of the adopted dynamic candidate selection packing framework (DCSPF) is as Figure 2 shown, which consists of four modules: a problem definition and initialization module, a Markov description module, a bin-packing decision module, and a dynamic candidate module. The specific steps executed by DCSPF correspond to these 4 modules respectively.

[0027] It should be noted that the formal definition of the three-dimensional bin-packing problem is given a rectangular container C with a fixed length L, a fixed width W, and an unrestricted height, and N square bins I = {1, 2,..., N}. The goal of the problem is to pack all the bins in I into the container C and minimize the height of the container used to maximize the space utilization rate of the container. Among them, the constraint conditions satisfied during the bin-packing process include that the bins placed in the container cannot overlap in space; the bins placed in the container cannot exceed the boundaries of the container, and the bins are placed orthogonally in the container, and the orthogonally placed in the container means that the length, width, or height of the bin is parallel or perpendicular to the length, width, or height of the container; the problem instance is initialized as a three-dimensional bin-packing problem instance, and randomly select Nmax bins from the N bins in the bin set I and put them into the processing set U proc , and all other bins are put into the candidate set U cand , where the size of Nmax is limited by the computational efficiency. If N < Nmax, then all bins are put into the processing set U proc , and the candidate set U cand is set to be empty.

[0028] Step S200: Describe the three-dimensional bin-packing problem as a Markov decision process, and formulate corresponding state representation, action representation, reward function, transition function, and discount factor for the problem instance.

[0029] In this embodiment, the state representation is that for any time step t, the state st By processing box status s t proc Candidate box status s t cand and container state s t bin It consists of three parts, s t proc The data records the three-dimensional dimensions of all the boxes in the processing center. t cand The three-dimensional dimension information of all boxes in the candidate set was recorded. t bin The system records the position and size information of all spaces within the container; the action is represented as action a for any time step t. t From the packing action a t pack and candidate action a t sel It consists of two parts, a t pack This indicates selecting a container from the processing set and determining the loading direction and loading space of the container. t sel This indicates that a box is selected from the candidate set and moved into the processing set; the reward function is that for any time step t, the reward r is... t =g t -g t+1 , where g t This represents the ratio of the unoccupied volume in the container to the total volume of the container at time step t; the transfer function is a deterministic transfer function P(s) for any time step t. t+1 |s t ,a t )=1, and the discount factor γ is equal to 1.

[0030] It should be noted that under this reward design, achieving higher cumulative rewards means achieving higher container space utilization. The discount factor γ is equal to 1 because the time step of the bin packing task is finite (maximum N).

[0031] Step S300: Construct a binning Q-network with Transformer as the backbone, iteratively train and update the binning Q-network using 3D binning problem instances as the training set until the cumulative reward converges. The input of the binning Q-network is the state that does not contain candidate bins, and the output is the binning action value. The binning action is selected based on the magnitude of the binning action value.

[0032] In this embodiment, the structure of the boxed Q-network is as follows: Figure 3As shown, a bin packing Q-network is constructed with Transformer as the backbone. The bin packing Q-network is iteratively trained and updated with three-dimensional bin packing problem instances as the training set until the cumulative reward converges. The input of the bin packing Q-network is the state that does not contain candidate bins, and the output is the bin packing action value. The bin packing action is selected based on the magnitude of the bin packing action value.

[0033] The state that does not contain candidate bins is defined as the binning Q-network input s for any time step t. t 1 ={s t proc s t bin}

[0034] The specific process for obtaining the value of the packing action is to first use several linear layers to process the box state s. t proc Encoded as h t proc,(0) Then use N e Layer Transformer encoder encodes h t proc,(0) To capture dependencies between containers:

[0035] in, Indicates the Transformer encoder's first... The layer outputs, the Multi-Head Attention Module (MAB), and the Feedforward Module (FFB) are defined as follows: MAB(X,Y)=LN(X)+MHA(X,YY)) FFB(H)=LN(H+FF(H)) Where LN, MHA, and FF represent layer normalization, multi-head attention layer, and feedforward layer, respectively. Then, N is used... e The layer Transformer encoder will store the container state. t bin Encoded as h t bin,(Ne) Finally, in h t proc,(Ne) with h t bin,(Ne) Taking the dot product of the vectors in the matrix M yields the matrix M. 1 , unfold M 1 That is, the value Q of the packing action is obtained. pack (s t 1 ,˙).

[0036] Furthermore, the construction of a binning Q-network with Transformer as the backbone, and the iterative training and updating of the binning Q-network using 3D binning problem instances as the training set until the cumulative reward converges, includes: Step S1: Extract the e-th problem instance from the training set; Step S2: Initialize time step t=0, initialize state s t ={s t proc s t cand s t bin} and the state s that does not include the subsequent selection box t 1 ={s t proc s t bin}; Step S3: Pack Q network according to s t 1 Calculate the value of all actions Q pack (s t 1 ,˙); Step S4: Based on the ε-greedy strategy, from Q pack (s t 1 Select and execute the packing action a) t pack :

[0037] Step S5: If the candidate set is not empty, randomly select a box from the candidate set and move it into the processing set; Step S6: Calculate the reward r t And update the status to s t+1 ; Step S7: Transfer the state transition tuple (s) t 1 a t pack r t s t+1 1 Store in the experience pool; Step S8: Sample a batch of transfer tuples (s) from the experience pool. 1 a pack ,r,s 1' ), calculate the loss and update the parameter θ:

[0038] Where b is the batch size of the sampling, θ —The target network parameters are updated to θ after every certain number of iterations; Step S9: If there are still boxes in the processing set, return to step S3; otherwise, e = e + 1. Step S10: If e is less than the number of instances in the training set, return to step S1; otherwise, end the training.

[0039] Step S400: Construct a dynamic candidate Q-network with Transformer as the backbone. The dynamic candidate Q-network does not require training, and its parameters are directly taken from the trained bin-packed Q-network. The input of the dynamic candidate Q-network is the state containing the candidate set bins, and the output is the candidate action value. The candidate action is selected based on the magnitude of the candidate action value.

[0040] It should be noted that the state containing the candidate set box refers to the state of the candidate Q network input s for any time step t. t 2 ={s t' proc s t cand s t+1 bin}, where t' represents the time state between t and t+1, in which a bin in the processing set has been loaded, while bins in the candidate set have not yet been moved into the processing set.

[0041] The structure of the candidate Q network is as follows Figure 4 As shown, its input is defined as the state s containing candidate bins. t 2 ={s t' proc s t cand s t+1 bin},in This represents the time state between t and t+1, where the processing set is located. One of the boxes has been loaded, and the candidate set The container has not yet been moved in. The output is the value of the candidate action. Candidate actions can be selected based on the magnitude of this value. Specifically, firstly, several linear layers are used to separate s t' proc and s t cand Encoded as h t' proc,(0) and h t cand,(0) Then, for each layer ,calculate:

[0042]

[0043] The output of the last layer Encoded all The box was moved into The features that may be obtained later. Then, calculation. and The dot product of the vectors yields matrix M2, and the maximum value of its second dimension is the candidate action value. It is important to note that the candidate Q-network does not require training; the parameters in its linear layers, MAB, and FFB are directly taken from the trained binned Q-network.

[0044] Step S500: Deploy the trained bin packing Q-network and dynamic candidate Q-network models into a real large-scale 3D bin packing environment to output a 3D bin packing solution.

[0045] To further illustrate the effectiveness and reliability of the method of this invention, it was tested based on multiple sets of randomly generated 3D packing problem examples and compared with existing methods. Regarding the box data, seven different box numbers (20, 30, 50, 100, 200, 500, and 1000) were randomly generated for model training and testing. The length, width, and height of each box were randomly selected within the ranges [L / 10, L / 2], [W / 10, W / 2], and [min(L / 10, W / 10), max(L / 2, W / 2)]. For the container, a 100×100 dimension container was considered. It should be noted that the container height is unrestricted and therefore does not need to be pre-defined.

[0046] During training and testing, the Transformer in the Deep Q-Network had a maximum of 3 layers, and each attention head had 8 heads. Additionally, the feature vector dimension of the Deep Q-Network was set to 128. The model was trained using the Adam optimizer with a learning rate of 10. -5 During training, 512×64 bin packing problem instances are generated per cycle, with a batch size of 64. The DCSPF model proposed in this invention is implemented based on PyTorch, and all experiments were conducted on an Intel(R) Core(TM) 15-12400F CPU and an NVIDIA GeForce RTX 2080 TI GPU.

[0047] DCSPF was compared with the following 3D bin packing algorithms on test cases with 7 different bin counts: 1) Extreme Point Method (EP); 2) Maximum Area First Matching (LAFF); 3) Biased Random Bond Genetic Algorithm (BRKGA); 4) MTSL; 5) CQL; 6) MM; 7) TS-BP.

[0048] Table 1 shows the performance comparison results of various methods, recording the calculated average space utilization. Referring to previous research, for any number of bins, generating 1024 random bin packing instances, the average space utilization is equal to the average of the space utilization of all bin packing schemes. Table 1 shows that when N is less than or equal to 50, DCSPF improves space utilization by 1.0%-2.5% compared to the current best-performing TS-BP method. When N increases to 100, many methods struggle to solve the problem quickly due to excessive computational complexity, while DCSPF further improves space utilization by 7.9% compared to the current best-performing MM method. When N is greater than or equal to 200, existing deep reinforcement learning methods struggle to handle such large-scale problems, while DCSPF can still solve the problem quickly and maintain high space utilization. This indicates that DCSPF has a greater advantage in generating efficient bin packing schemes for large-scale 3D bin packing problems.

[0049] Table 1 (Comparison of methods for solving the packing problem of 100×100 containers):

[0050] In this embodiment, the 3D packing problem is modeled as a Markov decision process. A packing Q-network and a candidate Q-network, both centered around a Transformer, are constructed to select boxes from the processing set for packing and dynamically select boxes from the candidate set to add to the processing set, respectively. The packing Q-network is optimized through reinforcement learning during the training phase, while the dynamic candidate Q-network directly reuses the trained packing Q-network parameters without additional training. This method effectively reduces the computational complexity of large-scale packing problems by dividing boxes into processing and candidate sets, while maintaining high space utilization. Compared to existing methods, this invention has a significant advantage in space utilization when handling large-scale instances, and is particularly suitable for the efficient packing requirements in logistics transportation and warehousing management.

[0051] Furthermore, this embodiment of the invention also proposes a storage medium storing a dynamic candidate bin packing program for a large-scale three-dimensional bin packing problem. When the dynamic candidate bin packing program for a large-scale three-dimensional bin packing problem is executed by a processor, it implements the steps of the dynamic candidate bin packing method for a large-scale three-dimensional bin packing problem as described above.

[0052] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the dynamic candidate packing system for large-scale three-dimensional packing problems of the present invention.

[0053] like Figure 5 As shown, the dynamic candidate packing system for large-scale three-dimensional packing problems proposed in this embodiment of the invention includes: Problem definition and initialization module 10 is used to formally define the 3D bin packing problem and initialize instances of the 3D bin packing problem; Markov description module 20 is used to describe the three-dimensional bin packing problem as a Markov decision process and to formulate corresponding state representations, action representations, reward functions, transition functions and discount factors for problem instances; The bin packing decision module 30 is used to construct a bin packing Q network with Transformer as the backbone, and iteratively train and update the bin packing Q network with three-dimensional bin packing problem instances as the training set until the cumulative reward converges. The input of the bin packing Q network is the state that does not contain candidate bins, and the output is the bin packing action value. The bin packing action is selected based on the magnitude of the bin packing action value. The dynamic candidate module 40 is used to construct a dynamic candidate Q network with Transformer as the backbone. The dynamic candidate Q network does not require training and its parameters are directly taken from the trained bin-packed Q network. The input of the dynamic candidate Q network is the state containing the candidate set bins, and the output is the candidate action value. The candidate action is selected based on the magnitude of the candidate action value. Deployment module 50 is used to deploy the trained bin packing Q-network and dynamic candidate Q-network models to a real-world large-scale 3D bin packing environment to output a 3D bin packing solution.

[0054] In this embodiment, the 3D packing problem is modeled as a Markov decision process. A packing Q-network and a candidate Q-network, both centered around a Transformer, are constructed to select boxes from the processing set for packing and dynamically select boxes from the candidate set to add to the processing set, respectively. The packing Q-network is optimized through reinforcement learning during the training phase, while the dynamic candidate Q-network directly reuses the trained packing Q-network parameters without additional training. This method effectively reduces the computational complexity of large-scale packing problems by dividing boxes into processing and candidate sets, while maintaining high space utilization. Compared to existing methods, this invention has a significant advantage in space utilization when handling large-scale instances, and is particularly suitable for the efficient packing requirements in logistics transportation and warehousing management.

[0055] This application embodiment also provides a dynamic candidate packing device for large-scale three-dimensional packing problems, including a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other through the communication bus. The memory is used to store dynamic candidate packing programs for large-scale three-dimensional packing problems. When the processor executes the program stored in the memory, it implements the above-mentioned dynamic candidate packing method for large-scale three-dimensional packing problems.

[0056] The communication bus mentioned in the dynamic candidate packing devices for the large-scale 3D packing problem can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0057] The communication interface is used for communication between the dynamic candidate packing device for the large-scale three-dimensional packing problem and other devices.

[0058] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0059] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0060] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0061] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0062] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0063] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0064] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0065] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0066] In addition, for technical details not described in detail in this embodiment, please refer to the dynamic candidate packing method for large-scale three-dimensional packing problems provided in any embodiment of the present invention, which will not be repeated here.

[0067] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0068] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0070] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

[0071] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

Claims

1. A dynamic candidate packing method for large scale three-dimensional bin packing problem, characterized in that, The dynamic candidate packing method for the large-scale three-dimensional packing problem comprises the following steps: formal definition of the three-dimensional packing problem, and initialization of the problem instance of the three-dimensional packing problem; formal definition of the three-dimensional packing problem, and initialization of the problem instance of the three-dimensional packing problem; the three-dimensional packing problem is described as a Markov decision process, and the corresponding state representation, action representation, reward function, transition function and discount factor are formulated for the problem instance; a packing Q network is constructed with Transformer as the backbone, the problem instance of the three-dimensional packing is iteratively trained and the packing Q network is updated until the cumulative reward converges, wherein the input of the packing Q network is a state not containing a candidate set of boxes, the output is a packing action value, and the packing action is selected based on the size of the packing action value; a dynamic candidate Q network is constructed with Transformer as the backbone, the dynamic candidate Q network does not need to be trained, and the parameters are directly taken from the trained packing Q network, wherein the input of the dynamic candidate Q network is a state containing a candidate set of boxes, the output is a candidate action value, and the candidate action is selected based on the size of the candidate action value; 2. The dynamic candidate packing method for the large scale three-dimensional bin packing problem of claim 1, wherein, The formal definition of the three-dimensional bin packing problem is given a rectangular container C with a fixed length L, a fixed width W, and an unrestricted height, and N rectangular boxes I = {1, 2, …, N}. The goal of the problem is to pack all the boxes in I into container C and minimize the height of the container used to maximize the space utilization rate of the container. Among them, the constraint conditions satisfied during the packing process include that the boxes placed in the container cannot overlap in space; the boxes placed in the container cannot exceed the boundaries of the container, and the boxes are orthogonally placed in the container. The orthogonally placed in the container means that the length, width, or height of the box is parallel or perpendicular to the length, width, or height of the container; the problem instance is initialized as an instance of the three-dimensional bin packing problem, and Nmax boxes are randomly selected from the N boxes in the box set I and placed in the processing set U proc , and the other boxes are all placed in the candidate set U cand , where the size of Nmax is limited by the computing efficiency. If N < Nmax, then all the boxes are placed in the processing set U proc , and the candidate set U cand is set to be empty.

3. The dynamic candidate packing method for large scale three-dimensional bin packing problem of claim 1, wherein, The state is represented as s t The state is represented as s t proc The candidate state is represented as s t cand The container state is represented as s t bin The state is represented as s t proc The state is represented as s t cand The state is represented as s t bin The state is represented as s t The state is represented as s t pack The state is represented as s t sel The state is represented as s t pack The state is represented as s t sel The state is represented as s t =g t -g t+1 , where g t represents the ratio of the volume of the container that is not occupied to the total volume of the container at time step t; the transition function is represented as P(s t+1 |s t ,a t )=1 for any time step t, and the discount factor γ is equal to 1.

4. The dynamic candidate packing method for large scale three-dimensional bin packing problem of claim 1, wherein, The state that does not include the candidate set bin is for any time step t, the binning Q network input s t 1 = s t proc , s t bin}.

5. The dynamic candidate packing method for large scale three-dimensional bin packing problem of claim 2, wherein, the trained packing Q network and the dynamic candidate Q network model are deployed to the actual large-scale three-dimensional packing environment to output a three-dimensional packing solution. the dynamic candidate packing method for the large-scale three-dimensional packing problem comprises the following steps: Step S2: initialize time step t = 0, initialize state s t = s t proc , s t cand , s t bin} and states s t 1 = s t proc , s t bin} not containing the latter selected set of bins; Step S3: The packing Q-network computes Q t 1 Compute all action values Q pack (s t 1 ,˙); Step S4: Select and execute a binning action a based on an e-greedy policy from Q pack (s t 1 ,˙) based on an e-greedy policy from Q t pack : step S1: take the e-th problem instance from the training set; Step S6: Calculate the reward r t and update the state to s t+1 ; Step S7: store the state transition tuple (s t 1 , a t pack , r t , s t+1 1 ) into the experience pool; Step S8: Sample a batch of transition tuples (s 1 , a pack , r, s 1' ) from the experience pool, compute the loss and update the parameters θ: where b is the batch size of the samples, θ — is the target network parameter, which is updated to θafter every few iterations; step S5: if the candidate set is not empty, randomly select a box from the candidate set and move it to the processing set; step S9: if there are still boxes in the processing set, return to step S3, otherwise e=e+1; 6. The dynamic candidate packing method for large scale three-dimensional bin packing problem of claim 1, wherein, The state of the candidate set bin comprises, for any time step t, a candidate Q-network input s t 2 = s t' proc , s t cand , s t+1 bin} where t' denotes the time state between t and t+1, in which time one bin from the processing set has been loaded, while the bin from the candidate set has not yet moved into the processing set.

7. A dynamic candidate binning system for large scale three-dimensional bin packing problems, characterized in that, step S10: if e is less than the number of instances in the training set, return to step S1, otherwise end the training. The dynamic candidate packing system for the large-scale three-dimensional packing problem is applied to the dynamic candidate packing method for the large-scale three-dimensional packing problem according to any one of claims 1 to 7, and the system comprises: a problem definition and initialization module for formal definition of the three-dimensional packing problem and initialization of the problem instance of the three-dimensional packing problem; a Markov description module for describing the three-dimensional packing problem as a Markov decision process and formulating the corresponding state representation, action representation, reward function, transition function and discount factor for the problem instance; a packing decision module for constructing a packing Q network with Transformer as the backbone, iteratively training and updating the packing Q network with the problem instance of the three-dimensional packing as the training set until the cumulative reward converges, wherein the input of the packing Q network is a state not containing a candidate set of boxes, the output is a packing action value, and the packing action is selected based on the size of the packing action value; a dynamic candidate module for constructing a dynamic candidate Q network with Transformer as the backbone, wherein the dynamic candidate Q network does not need to be trained, and the parameters are directly taken from the trained packing Q network, wherein the input of the dynamic candidate Q network is a state containing a candidate set of boxes, the output is a candidate action value, and the candidate action is selected based on the size of the candidate action value; A deployment module is configured to deploy the trained binning Q-network and the dynamic candidate Q-network model into an actual large-scale 3D binning environment for outputting a 3D binning solution.

8. A dynamic candidate packing device for a large scale three-dimensional bin packing problem, characterized in that, The dynamic candidate binning device for the large-scale 3D binning problem comprises a memory, a processor, and a dynamic candidate binning program for the large-scale 3D binning problem stored on the memory and executable on the processor, and the dynamic candidate binning program is configured to implement the steps of the dynamic candidate binning method for the large-scale 3D binning problem according to any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium stores the dynamic candidate binning program for the large-scale 3D binning problem, and the dynamic candidate binning program implements the steps of the dynamic candidate binning method for the large-scale 3D binning problem according to any one of claims 1 to 6 when executed by the processor.

Images