A two-dimensional special-shaped parts layout method based on deep reinforcement learning
Through the method based on deep reinforcement learning, the state space and action space are defined, reward and punishment mechanisms are set, and the deep reinforcement learning training environment is configured, the complexity and automation problems of two-dimensional special-shaped parts are solved, and the efficient special-shaped parts are arranged in the design. Suitable for a variety of shapes and scenes.
Patent Information
- Application Number
- CN202111399771.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-24
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2041-11-24
AI Technical Summary
The prior art is difficult to effectively solve the problem of two-dimensional special-shaped parts sorting, especially due to the complexity and irregularity of boundary profiles, the complexity of the algorithm increases, making it difficult to automatically design.
A deep reinforcement learning method is adopted, and by defining state space, action space and reward and punishment mechanisms, the deep reinforcement learning training environment is configured, and the training data is generated, and the strategies are optimized to meet the conditions of the sampling task.
It realizes efficient arrangement of complex two-dimensional special-shaped parts, reduces the demand for manual design, can cope with a variety of shapes and scenarios, improves material utilization, and reduces calculation costs.
Smart Images
Figure CN114218624B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of two-dimensional special-shaped parts arrangement, and specifically relates to a two-dimensional special-shaped parts arrangement method based on deep reinforcement learning. Background Art
[0002] Nesting Problem is also called Cutting and stock problem or Packing Problem. Its goal is to find a higher material utilization rate in the material cutting process. Nesting problem belongs to the classic NP-Hard problem. Its time complexity increases rapidly with the increase of problem scale. It is difficult to accurately solve large-scale examples in a reasonable time. For example Figure 1 shown.
[0003] From a whole piece of flat material, cut out multiple sub-blocks of different styles (simple task: several rectangular wooden blocks of different sizes; complex task: in addition to the basic shape, it is also necessary to cut out irregular shaped special-shaped blocks), and meet the constraints: cut out no less than the specified number of sub-blocks of each type, and the remaining unallocated cutting area (waste ratio) is as small as possible.
[0004] When designing segmentation schemes, traditional methods such as rectangular nesting algorithms and genetic algorithms can only cope with simple scenarios. For example, they can only cut out rectangular sub-blocks with relatively consistent shapes. Compared with the rectangular nesting problem, the prominent feature of the special-shaped piece nesting problem is that the boundary contours of the pieces are complex, and complex geometric operations are required in the calculation process. The algorithm complexity will further increase, and it is a problem recognized by academia and industry as difficult to solve. In most cases, irregular shape nesting algorithms are mainly based on heuristic algorithms and intelligent search algorithms. Such methods require manual design of complex preprocessing methods and complex exploration strategies, and are difficult to automate. Summary of the invention
[0005] The present invention provides a two-dimensional special-shaped parts arrangement method based on deep reinforcement learning, which solves the problems of complex arrangement and difficulty in automation of two-dimensional special-shaped parts in the prior art.
[0006] The present invention provides a two-dimensional special-shaped parts arrangement method based on deep reinforcement learning, wherein the plane material used for cutting and arranging is called a mother block, and the special-shaped parts and non-special-shaped parts to be cut are collectively called sub-blocks, comprising the following steps:
[0007] Step 1: Obtain the information of the task and determine the state space and action space of the task;
[0008] Step 2: Introduce auxiliary decision making to reduce the action space: each time an action is taken, the remaining unsampled mother block area is corroded, and the erosion radius is the minimum radius of the candidate sub-block, which is defined as the minimum distance between the geometric center of the candidate sub-block shape and the shape contour point;
[0009] Step 3: Set up a reward and punishment mechanism;
[0010] Step 4: Configure the deep reinforcement learning training environment, perform training, and save the best model;
[0011] Step 5: According to the best model, substitute it into the task scenario for reasoning and calculation to obtain the final layout plan.
[0012] The specific method for determining the state space and action space of the task in step 1 of the present invention is:
[0013] (1) Determine the state space S: The regions on the parent block that have been divided into sub-blocks and the remaining undivided regions are marked according to whether they are divided or not, and together they constitute the state space S. The state space S consists of a region in a two-dimensional space. Initially, the entire parent block is in an undivided state.
[0014] (2) Determine the action space A: The action space A is determined by the attributes of the sub-blocks, including the size and placement posture of the sub-blocks.
[0015] The specific method for determining the action space in step (2) of the present invention is: specify a standard initial position for each candidate sub-block, and stipulate that when it is used for actual cutting and placement of shapes, it can only rotate by an angle of {30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 330, 360} relative to the initial position.
[0016] The reward and punishment mechanism of the present invention is set as follows:
[0017] (1) No overlap: If the placed block and the remaining block area do not satisfy the requirement that the block is completely surrounded, the round ends and the task is marked as over, with a penalty of -1.
[0018] (2) Each time a sub-block is successfully cut out in the simulation environment, a reward of 0.01 is given;
[0019] (3) Time cost: a penalty of -0.001 is imposed for each additional exploration step;
[0020] (3) Utilization rate: At the end of each round, calculate the remaining unsampled area ratio S_left / S_total, and return the reward for this round: γ(1–S_left / S_total), where γ is a hyperparameter;
[0021] (4) Minimum number of sub-blocks: Define the reward related to the minimum number of sub-blocks. Define this part of the reward as the negative value of the ratio of the number of unfinished sub-blocks to the total number of sub-blocks that need to be sampled: –N_unfinished / N_total.
[0022] In step (3) of the present invention, γ takes a value between 0.5 and 2.0 according to the strictness of the task's requirements on material utilization.
[0023] The present invention configures a deep reinforcement learning training environment, and the training content includes:
[0024] (1) Obtain the next row arrangement action in the cuttable area of the parent block according to the current strategy π: The input of the deep learning-based strategy π is the state of the currently observed task scene: the area of the parent block that has been segmented into sub-blocks and the unsegmented area. The unsegmented area is eroded according to the auxiliary strategy and then input. The output is the two-dimensional coordinate point P, the candidate sub-block n, and the rotation angle α;
[0025] (2) In the simulation environment, on the mother block, with the two-dimensional coordinate point P as the center point, the candidate sub-block n is rotated clockwise by an angle α relative to the default position and placed into the mother block, and the operation of splitting the sub-block is performed. This part of the area is marked as split, and according to the reward and punishment mechanism, the reward r given by the environment is obtained;
[0026] (3) Repeat steps (1) to (2) until the round ends. The end of the round is marked by one of the following conditions:
[0027] ① If the number of steps executed in this round exceeds the set number of steps, the current round will end immediately;
[0028] ② If the area of the remaining area in the parent block is smaller than the set threshold, it means that no more sub-blocks can be split under the current state, and the current round ends;
[0029] At the end of the round, record the cumulative reward R of this round, that is, the cumulative sum of the rewards r of each step of this round, and compare it with the preset best model cumulative reward threshold R_best. Compare whether the cumulative reward R of this round is greater than R_best to determine whether the current model is the best model. If so, save the model of this round as the best model and update the value of R_best to R of the current round.
[0030] (4) Save each step of the state transition process in this round to the memory pool for training the value function in the deep reinforcement learning algorithm, and use the gradient descent method to update and improve the strategy π;
[0031] (5) If no signal to stop training is received, the next round of exploration is carried out, and steps (1) to (4) are repeated until one or more of the following two states appear, and training is stopped:
[0032] ① If the cumulative number of executed rounds exceeds the set number of steps, the training will be stopped;
[0033] ②When the cumulative reward R is greater than the preset threshold, stop training.
[0034] The initialization value of R_best in step (3) of the present invention is -9999.
[0035] If the number of steps executed in this round in step (3) of the present invention exceeds the set number of steps 3000 times, the current round will be ended immediately.
[0036] The threshold in step (3) of the present invention is set to 3 times the area of the smallest sub-block in the candidate sub-blocks.
[0037] If the cumulative number of rounds executed in step (5) of the present invention exceeds the set number of steps of 5000, the training is stopped.
[0038] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description.
[0039] Beneficial effects of the present invention: The present invention converts the arrangement problem of two-dimensional special-shaped parts into a scenario applicable to reinforcement learning, sets a state space, an action space and a reward mechanism, and adds auxiliary functions to help improve the efficiency of the search. It can cope with the arrangement problem of complex two-dimensional special-shaped parts, and can be reused at a low cost in different demand scenarios, such as changing the shape of the mother block, the shape of the child block, the minimum number of cuts, etc. to obtain new solutions, and has the following advantages: ① No need to manually design complex preprocessing means and complex exploration strategies; ② It can cope with the arrangement problem of complex two-dimensional special-shaped parts, which can be concave, convex, annular, and have holes, because the action space can cover all the remaining areas that can be reasonably placed; ③ It can be a migration model at a low cost in different demand scenarios, such as changing the shape of the mother block, the shape of the child block, the minimum number of cuts, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is a schematic diagram of the sample arrangement problem case;
[0041] Figure 2 It is a schematic diagram of the overall process of the two-dimensional special-shaped parts arrangement method based on deep reinforcement learning of the present invention. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0044] This scheme proposes a two-dimensional special-shaped parts arrangement method with a wide range of applications, which can solve the two-dimensional special-shaped parts arrangement problem more flexibly. By using the characteristics of deep reinforcement learning, automatic exploration and sampling, generating training data, and continuously optimizing the strategy through back propagation until the conditions specified by the arrangement task are met, the whole process does not require much preprocessing.
[0045] In order to deal with the more complex two-dimensional special-shaped parts layout problem, this solution models the scene of the layout problem based on deep reinforcement learning. The observation space is constructed by the cut wood blocks and the shapes of the cut wood blocks, and the action space is built according to the size of the shapes to be segmented. A reward and punishment mechanism is set up, and a deep reinforcement learning training environment is configured. The layout strategy is trained through deep reinforcement learning, automatic exploration and sampling, and training data are generated. The strategy is continuously optimized through back propagation until the conditions specified by the layout task are met.
[0046] The specific method is:
[0047] 1. Obtain the arrangement task information and determine the state space of the task and the action space of the agent.
[0048] The plane material used for cutting and arranging is called the mother block, and the special-shaped parts and non-special-shaped parts that need to be cut are collectively called sub-blocks; the mother block used for segmentation can be described by a polygonal area on a plane, such as a common rectangular plane material, which can be described by the coordinates of two diagonal points. The modeling process is as follows:
[0049] Determine the state space: The areas on the parent block that have been divided into sub-blocks and the remaining undivided areas are marked according to whether they are divided or not, and together constitute the state space S, which consists of an area in a two-dimensional space. The state space is initially that the entire parent block is in an undivided state.
[0050] Determine the action space: The action space A is determined by the properties (size and pose) of the sub-blocks.
[0051] When the computational cost is not considered, the sub-block can be cut out from the parent block at any angle, but this will bring the problem of too large an action space and bring difficulties in sampling. In order to reduce the amount of calculation, the problem can be slightly simplified. For example, a standard initial pose (two-dimensional space) is specified for each candidate sub-block, and it is stipulated that when it is used for actual cutting and placing of shapes, it can only be rotated relative to the initial position by one of the angles {30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 330, 360}. Therefore, each time a sub-block shape is selected for cutting a wooden block, the action that can be taken is described as follows: select a two-dimensional point in the remaining unsampled area of the parent block as the center point of the candidate cutting shape, select an angle from the 12 candidate angles {30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 330, 360}, select one of the candidate sub-blocks, and split according to the shape of the sub-block. The action space contains two parts of information: one is the two-dimensional coordinates of the center point, which is a continuous value; the other is the selection of a sub-block shape and the determination of a rotation angle, which is a discrete value of 12*N.
[0052] Second, introduce auxiliary decision-making to reduce the action space and speed up the exploration process of the intelligent agent.
[0053] In order to increase the sampling efficiency, an auxiliary decision is introduced. Each time an action is taken, the remaining unsampled mother block area is eroded (a method in the field of image processing, which can reduce the area). The erosion radius is the minimum radius of the candidate sub-block. The minimum radius of the sub-block is defined as the minimum distance from the geometric center of the shape to the shape contour point.
[0054] 3. Set up a reward and punishment mechanism, specifically:
[0055] No overlap: If the placed block and the remaining block area do not satisfy the requirement that the block is completely surrounded (the corresponding actual meaning is that the shape of the block to be placed exceeds the range or overlaps with other arranged blocks), the round ends and is marked as mission completed, with a penalty of -1.
[0056] Each time a sub-block is successfully cut out in the simulation environment, a reward of 0.01 is given;
[0057] Time cost: For each additional exploration step, a penalty of -0.001 is given, because in the scenario of 2D special-shaped parts layout, there is generally no real-time requirement and the time cost requirement is not strict. Generally, it is only necessary to retrain and generate a new solution when the requirements are changed (for example, changing the shape of the parent block, the shape of the child block, or the minimum number of cuts);
[0058] Utilization rate: At the end of each round, calculate the remaining unsampled area ratio S_left / S_total, and return the reward for this round: γ(1–S_left / S_total), where γ is a hyperparameter. Depending on the strictness of the task's requirements for material utilization, it is recommended that the value be between 0.5 and 2.0;
[0059] Minimum number of sub-blocks required: Define the reward related to the minimum number of sub-blocks. Define this part of the reward as the negative value of the ratio of the number of unfinished sub-blocks to the total number of sub-blocks that need to be sampled: –N_unfinished / N_total. The complex version can be weighted averaged according to the importance of each sub-block.
[0060] 4. Configure the deep reinforcement learning training environment to train the intelligent agent.
[0061] After configuring the state space, action space, and reward and punishment mechanism, select a suitable deep reinforcement learning algorithm. The deep reinforcement learning algorithms that can be used include: deep Q learning algorithm (DQN), dual value network DDQN, etc., and start training the intelligent agent:
[0062] (1) The next arrangement action of the agent is obtained according to the current strategy π in the cuttable area of the parent block: the input of the deep learning-based strategy π is the state of the currently observed task scene: the area of the parent block that has been segmented into sub-blocks and the unsegmented area. The unsegmented area is eroded according to the auxiliary strategy and then input. The output is the two-dimensional coordinate point P, the candidate sub-block n, and the rotation angle α.
[0063] (2) At the position P of the parent block, rotate the candidate sub-block clockwise by an angle α relative to the default position to perform the sub-block segmentation operation, mark this area as segmented, and obtain the reward r given by the environment.
[0064] (3) Repeat steps (1) and (2) until the round ends. The end of the round is marked as:
[0065] ① If the number of steps executed in this round exceeds 3000, the current round will end immediately;
[0066] ② The area of the remaining area in the parent block is smaller than the set threshold, which can be set to three times the area of the smallest sub-block in the candidate sub-blocks. If the area of the remaining area in the parent block is smaller than the set threshold, it means that no sub-block can be segmented in the current state, and the current round ends;
[0067] At the end of the round, record the cumulative reward R of this round, that is, the cumulative sum of the rewards r of each step of this round, and compare it with the preset best model cumulative reward threshold R_best (which can be initialized to a minimum value -9999). Compare whether the cumulative reward R of this round is greater than R_best to determine whether the current model is the best model. If so, save the model of this round as the best model, and update the value of R_best to R of the current round;
[0068] (4) Save each step of the state transition process in this round to the memory pool for training the value function in the deep reinforcement learning algorithm and updating and improving the strategy π;
[0069] (5) If no signal to stop training is received, proceed to the next round of exploration and repeat steps (1) to (4) until one or more of the following two states appear, then stop training;
[0070] ① If the cumulative number of rounds executed exceeds 5000, stop training;
[0071] ② The current cumulative reward R. When the cumulative reward R is greater than the preset threshold, stop training.
[0072] 5. Based on the optimal model, substitute it into the task scenario for reasoning and calculation to obtain the final arrangement plan.
[0073] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A two-dimensional special-shaped parts arrangement method based on deep reinforcement learning, wherein the plane material used for cutting and arranging is called a mother block, and the special-shaped parts and non-special-shaped parts to be cut are collectively called sub-blocks, characterized in that: The following steps are involved: Step 1: Obtain the information of the task and determine the state space and action space of the task; Step 2: Introduce auxiliary decision making to reduce the action space: each time an action is taken, the remaining unsampled mother block area is corroded, and the erosion radius is the minimum radius of the candidate sub-block, which is defined as the minimum distance between the geometric center of the candidate sub-block shape and the shape contour point; Step 3: Set up a reward and punishment mechanism; Step 4: Configure the deep reinforcement learning training environment, perform training, and save the best model; Step 5: According to the best model, substitute the task scenario for reasoning and calculation to obtain the final layout plan; The specific method for determining the state space and action space of the task in step 1 is: 1.1 Determine the state space S: The areas on the parent block that have been divided into sub-blocks and the remaining undivided areas are marked according to whether they are divided or not, and together constitute the state space S. The state space S consists of an area in a two-dimensional space. The state space is initially that the entire parent block is in an undivided state; 1.2 Determine the action space A: The action space A is determined by the attributes of the sub-blocks, including the size and placement of the sub-blocks; The reward and punishment mechanism is set as follows: 3.1 No overlap: If the placed sub-block and the remaining wooden block area do not satisfy the requirement that the sub-block is completely surrounded, the round ends and the task is marked as over, with a penalty of -1. 3.2 Each time a sub-block is successfully cut out in the simulation environment, a reward of 0.01 will be given; 3.3 Time cost: For each additional exploration step, a penalty of -0.001 is given; 3.4 Utilization rate: At the end of each round, calculate the remaining unsampled area ratio S_left / S_total, and return the reward for this round: γ(1–S_left / S_total), where γ is a hyperparameter; 3.5 Minimum number of sub-blocks requirement: Define the reward related to the minimum number of sub-blocks, and define this part of the reward as the negative value of the ratio of the number of unfinished sub-blocks to the total number of sub-blocks that need to be sampled: –N_unfinished / N_total.
2. A two-dimensional special-shaped parts arrangement method based on deep reinforcement learning according to claim 1, characterized in that: The specific method for determining the action space in step 1.2 is: specify a standard initial position for each candidate sub-block, and stipulate that when it is used for actual cutting and placement of shapes, it can only rotate by an angle of {30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 330, 360} relative to the initial position.
3. The two-dimensional special-shaped parts arrangement method based on deep reinforcement learning according to claim 1, characterized in that: In step 3.4, γ takes a value between 0.5 and 2.0 according to the strictness of the task's requirements on material utilization.
4. The two-dimensional special-shaped parts arrangement method based on deep reinforcement learning according to claim 1, characterized in that: Configure the deep reinforcement learning training environment. The training content includes: 4.1 Obtain the next rowing action in the cuttable area of the parent block according to the current strategy π: The input of the deep learning-based strategy π is the state S of the currently observed task scene: the area of the parent block that has been segmented into sub-blocks and the unsegmented area. The unsegmented area is eroded according to the auxiliary strategy and then input. The output is the two-dimensional coordinate point P, the candidate sub-block n, and the rotation angle α; 4.2 In the simulation environment, on the mother block, with the two-dimensional coordinate point P as the center point, the candidate sub-block n is rotated clockwise by an angle α relative to the default position and placed into the mother block, and the operation of splitting the sub-block is performed. This part of the area is marked as split, and according to the reward and punishment mechanism, the reward r given by the environment is obtained; 4.3 Repeat steps 4.1 to 4.2 until the round ends. The end of the round is marked by one of the following conditions: 4.3.1 If the number of steps executed in this round exceeds the set number of steps, the current round will end immediately; 4.3.2 If the area of the remaining region in the parent block is smaller than the set threshold, it means that no more sub-blocks can be split under the current state, and the current round ends; At the end of the round, record the cumulative reward R of this round, that is, the cumulative sum of the rewards r of each step of this round, and compare it with the preset best model cumulative reward threshold R_best. Compare whether the cumulative reward R of this round is greater than R_best to determine whether the current model is the best model. If so, save the model of this round as the best model and update the value of R_best to R of the current round. 4.4 Save each step of the state transfer process in this round to the memory pool for training the value function in the deep reinforcement learning algorithm, and use the gradient descent method to update and improve the strategy π; 4.5 If no signal to stop training is received, proceed to the next round of exploration and repeat steps 4.1 to 4.4 until one or more of the following two states appear, then stop training: 4.5.1 If the cumulative number of executed rounds exceeds the set number of steps, training will be stopped; 4.5.2 When the cumulative reward R is greater than the preset threshold, stop training.
5. A two-dimensional special-shaped parts arrangement method based on deep reinforcement learning according to claim 4, characterized in that: The initialization value of R_best in step 4.3 is -9999.
6. The two-dimensional special-shaped parts arrangement method based on deep reinforcement learning according to claim 4, characterized in that: If the number of steps executed in this round in step 4.3 exceeds the set number of steps 3000 times, the current round will be ended immediately.
7. A two-dimensional special-shaped parts arrangement method based on deep reinforcement learning according to claim 4, characterized in that: In step 4.5, the threshold is set to 3 times the area of the smallest sub-block among the candidate sub-blocks.
8. The two-dimensional special-shaped parts arrangement method based on deep reinforcement learning according to claim 4, characterized in that: If the cumulative number of rounds executed in step 4.5 exceeds the set number of steps 5000, the training is stopped.