The unordered stacking model, the training method and the unordered stacking method

Abstract

The application relates to the technical field of industrial robots and discloses a trained disordered stacking model, a training method and a disordered stacking method. The trained disordered stacking model comprises an action decision module and a stacking state return expectation module. The action decision module receives state space information of a stacking tray and posture information of an article to be stacked, performs stacking action probability analysis on the article to be stacked according to the stacking state information, and outputs a stacking action of the article to be stacked to a stacking execution mechanism. The stacking state return expectation module performs return analysis on the stacking state information to obtain an expected return of the stacking state information. The action decision module is updated according to the expected return of the stacking state information, the stacking action of the article to be stacked and the stacking state information. The stacking state return expectation module is updated according to the stacking action of the article to be stacked. The trained disordered stacking model can adapt to different object postures and different stacking spaces in different scenes, and the adaptability of the disordered stacking is improved.

Description

Technical Field

[0001] This application relates to the field of industrial robot technology, and in particular to a trained unordered palletizing model, a training method, and an unordered palletizing method. Background Technology

[0002] Unordered palletizing refers to the task of placing a collection of randomly arranged items (e.g., boxes, crates, or other irregularly shaped objects) onto a pallet using a robot. The key characteristic of this task is the randomness of the items' order and position. The robot must be able to identify and handle these disordered items, placing them onto the pallet efficiently and reliably. Unordered palletizing tasks are common in the field of robotics, particularly in industrial scenarios related to warehousing, logistics, and production line automation, requiring robots to identify, grasp, and stack randomly arriving, irregularly shaped objects onto pallets in real time.

[0003] Traditional methods rely on pre-programmed trajectory planning, which enables robots to stack objects according to pre-defined positions and stacking orders. However, when faced with various industrial scenarios (various object postures and various stacking spaces), this traditional method may fail to perform the task correctly and has poor adaptability. Summary of the Invention

[0004] The purpose of this application is to provide a trained unordered stacking model, a training method, and an unordered stacking method to improve the adaptability of unordered stacking.

[0005] To address the aforementioned technical problems, embodiments of this application provide a trained unordered palletizing model, comprising: an action decision module and a palletizing state reward expectation module; the action decision module is used to receive externally input palletizing state information, perform palletizing action probability analysis on the items to be palletized based on the palletizing state information, and output the palletizing action of the items to be palletized to the palletizing execution mechanism; wherein, the palletizing state information includes: state space information of the palletizing pallet and posture information of the items to be palletized; the palletizing state reward expectation module is used to receive the palletizing state information, perform reward analysis on the palletizing state information, and obtain the expected reward of the palletizing state information; the action decision module is also used to perform self-iterative updates based on the expected reward of the palletizing state information, the palletizing action of the items to be palletized, and the palletizing state information; the palletizing state reward expectation module is also used to perform self-iterative updates based on the palletizing action of the items to be palletized.

[0006] An embodiment of this application also provides a trained unordered palletizing model, wherein the action decision module includes: an Actor network, an action search space optimization unit, and an action determination unit; the Actor network is used to receive the palletizing state information and determine the action probability distribution of the palletizing state information; the action search space optimization unit is used to filter the action probability distribution of the palletizing state information based on preset rules and determine the action probability distribution of the palletizing state information; the action determination unit is used to determine the palletizing action of the item to be palletized from the action probability distribution of the action probability distribution.

[0007] An embodiment of this application also provides a trained unordered palletizing model, wherein the action search space optimization unit is specifically used to: perform heuristic rule filtering on the action probability distribution of the palletizing state information to obtain an initial action probability distribution; and perform support area constraints and centroid constraints on the initial action probability distribution to determine the action probability distribution of the palletizing state information.

[0008] An embodiment of this application also provides a trained unordered palletizing model, wherein the Actor network is further used to: determine a palletizing value function based on the palletizing action of the item to be palletized and the palletizing state information; and update the Actor network using a policy gradient method based on the expected reward of the palletizing value function and the palletizing state information.

[0009] An embodiment of this application also provides a trained unordered palletizing model, wherein the palletizing state reward expectation module includes a Critic network.

[0010] An embodiment of this application also provides a trained unordered palletizing model, the unordered palletizing model further comprising: a reward module; the reward module is used to determine the palletizing reward value of the palletizing action of the item to be palletized according to a preset reward function; the Critic network is specifically used to: update the Critic network according to the palletizing action of the item to be palletized and the palletizing reward value of the palletizing action.

[0011] An embodiment of this application also provides a trained unordered palletizing model, wherein the reward function is preset in the following way: the reward function is determined based on the space utilization rate of the palletizing tray, the safe space information of the palletizing tray, and the stable value of the palletizing action of the item to be palletized.

[0012] An embodiment of this application also provides a trained unordered palletizing model, wherein the Critic network is specifically used to update the Critic network according to a temporal difference method, the palletizing action of the item to be palletized, and the palletizing reward value of the palletizing action.

[0013] An embodiment of this application also provides a training method for an unordered palletizing model, comprising: performing multiple iterations of training on the unordered palletizing model to obtain the trained unordered palletizing model; in each training iteration, a group of items to be palletized are sequentially palletized onto the palletizing tray corresponding to the group of items to be palletized, specifically including: inputting the posture information of one item to be palletized in the group and the current state space information of the palletizing tray; based on the posture information of the item to be palletized and the current state space information of the palletizing tray, palletizing the item to be palletized onto the palletizing tray until the training termination condition is met; wherein, the training termination condition is: all items in the group to be palletized have been placed in the palletizing tray, or the palletizing tray can no longer hold the items to be palletized.

[0014] An embodiment of this application also provides a disordered palletizing method, the method comprising: acquiring the current state space information of a palletizing pallet; determining the posture information of the item to be palletized based on an image of the item to be palletized; inputting the current state space information of the palletizing pallet and the posture information of the item to be palletized into the above-trained disordered palletizing model to obtain the palletizing action of the item to be palletized; and performing the palletizing action on the item to be palletized through a palletizing execution mechanism.

[0015] In this application, through the model's self-iterative update, the palletizing action of the item to be palletized corresponding to any palletizing state information input from the outside can be obtained and sent to the palletizing execution mechanism. This enables the trained disordered palletizing model to dynamically adjust the palletizing strategy when facing various different industrial scenarios, adapting to different object postures and different stacking spaces corresponding to different industrial scenarios, thereby improving the adaptability of disordered palletizing. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0017] Figure 1 This is a schematic diagram of the first type of trained unordered stacking model provided in the embodiments of this application;

[0018] Figure 2 This is a schematic diagram of the second type of unordered stacking model that has been trained according to the embodiments of this application;

[0019] Figure 3 This is a flowchart illustrating the probability distribution of possible actions for determining palletizing status information, provided in an embodiment of this application.

[0020] Figure 4 This is a schematic diagram of the third type of unordered stacking model trained according to the embodiments of this application;

[0021] Figure 5 This is a schematic diagram of the fourth type of unordered palletizing model trained according to the embodiments of this application;

[0022] Figure 6 This is a flowchart of a training method for an unordered stacking model provided in an embodiment of this application;

[0023] Figure 7 This is a flowchart of an unordered palletizing method provided in an embodiment of this application.

[0024] Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0026] In industrial settings, unordered palletizing tasks require robots to identify, grasp, and stack randomly arriving objects of various shapes in real time. Traditional methods typically rely on pre-programmed trajectory planning, which has the following problems: poor environmental adaptability, the need to pre-set object positions and stacking order, and the inability to handle dynamically changing objects and stacking space.

[0027] To address the aforementioned technical problems, this application proposes a trained unordered stacking model. Figure 1 This is a schematic diagram of the first type of trained unordered palletizing model provided in the embodiments of this application, as shown below. Figure 1 As shown, the trained unordered palletizing model includes: an action decision module 100 and a palletizing state reward expectation module 200.

[0028] The action decision module 100 is used to receive palletizing status information from external input. This external input can be input by a person or input from other devices to the action decision module 100; no specific limitations are imposed in this embodiment.

[0029] In this embodiment, the palletizing status information includes: the state space information of the palletizing pallet and the posture information of the item to be palletized. The state space information of the palletizing pallet may include: length, width, height, and volume information of the pallet; the posture information of the item to be palletized may include: length, width, height, and orientation information of the item to be palletized. Of course, the palletizing status information may also include other information about the current palletizing environment, which is not specifically limited in this embodiment.

[0030] After receiving the palletizing status information, the action decision module 100 performs a probability analysis of the palletizing action for the item to be palletized based on the palletizing status information and outputs the palletizing action for the item to be palletized. In this embodiment, the action decision module 100 outputs the palletizing action for the item to be palletized to the palletizing execution mechanism, so that the palletizing execution mechanism executes the palletizing action for the item to be palletized. Specifically, the palletizing action for the item to be palletized involves the palletizing execution mechanism grasping, moving, and placing the item to be palletized at the corresponding position on the palletizing pallet. This corresponding position can ensure the stable placement of the item to be palletized. Therefore, the most important parameter in the palletizing action for the item to be palletized is the palletizing position information of the item to be palletized.

[0031] The palletizing status reporting expectation module 200 is used to receive palletizing status information, which is the same as the palletizing status information received by the action decision module 100.

[0032] The palletizing status reward expectation module 200 performs reward analysis on the palletizing status information to obtain the expected reward of the palletizing status information. This expected reward, also known as the state value of the palletizing status information, represents the expected value of the robot's future cumulative reward under that palletizing status information and its current strategy. In the palletizing task, this value assessment helps the robot determine the "good" or "bad" of the current state and, based on this assessment, adopt a more appropriate action strategy to improve the efficiency and success probability of the palletizing task.

[0033] The trained unordered palletizing model can be internally trained based on data during both training and use to improve its performance. In this embodiment, both the action decision module 100 and the palletizing state reward expectation module 200 can be self-iteratio updated.

[0034] The action decision module 100 is further used to perform self-iterative updates based on the expected reward of the palletizing status information, the palletizing action of the item to be palletized, and the palletizing status information. The palletizing status reward expectation module 200 is further used to perform self-iterative updates based on the palletizing action of the item to be palletized.

[0035] In this embodiment, the self-iterative updates of both the action decision module 100 and the palletizing status reward expectation module 200 involve updating their own model parameters. Through continuous updating and adjustment of the model parameters, the action decision module 100 and the palletizing status reward expectation module 200 can take more effective actions when there is no palletizing task, thereby improving the effectiveness, completion rate, and efficiency of the task.

[0036] In this embodiment, through the model's self-iterative update, the palletizing action of the item to be palletized corresponding to any palletizing state information input from the outside can be obtained and sent to the palletizing execution mechanism. This allows the trained disordered palletizing model to dynamically adjust the palletizing strategy when facing various different industrial scenarios, adapting to different object postures and different stacking spaces corresponding to different industrial scenarios, thereby improving adaptability.

[0037] In the above Figure 1 Based on the embodiments described above, this application also provides another trained unordered palletizing model. Figure 2 This is a schematic diagram of the second type of trained unordered palletizing model provided in the embodiments of this application, as shown below. Figure 2 As shown, the action decision module 100 includes: an Actor network 101, an action search space optimization unit 102, and an action determination unit 103.

[0038] The Actor Network 101 receives palletizing state information and determines the probability distribution of actions based on that information. Specifically, the input to the Actor Network is the palletizing state information. After a Convolutional Neural Network (CNN) extracts useful features, these features are fused through a fully connected layer. The network then abstracts and processes the data, ultimately outputting a probability distribution π(a|s) containing all possible actions, where all possible actions refer to all possible palletizing actions for the item to be palletized. The Actor Network 101 can effectively perform tasks in complex palletizing environments, such as automated palletizing.

[0039] The action search space optimization unit 102 is used to filter the action probability distribution of palletizing state information based on preset rules, and determine the feasible action probability distribution of the palletizing state information. Specifically, in a palletizing task, the action space is usually limited, but there may be multiple action options, such as: selecting an item to move, placing an item in a specific location, adjusting the stacking method of items, etc. These actions may change according to changes in the environment and the current task state. Therefore, the goal of the action search space optimization unit 102 is to determine which actions in the action probability distribution are valid under the given palletizing state information through preset rules. After filtering, the action search space optimization unit 102 adjusts the action probability distribution, retains actions that meet the requirements, and updates their probabilities. For actions that do not meet the rules, their probabilities can be reduced or even reduced to zero. In this way, the optimized probability distribution reflects the feasible actions in the execution of the palletizing task. The final output of the action search space optimization unit 102 is a new, optimized action probability distribution (which can be called the feasible action probability distribution), which only contains those executable actions that meet the rules.

[0040] The action search space optimization unit 102 is used in reinforcement learning to filter and optimize the action probability distribution output by the Actor network according to preset rules, thereby ensuring that the selected actions are feasible and meet physical and task requirements. This mechanism is very important in automated palletizing tasks, as it can effectively improve the efficiency and stability of palletizing task execution.

[0041] The action determination unit 103 is used to determine the palletizing action of the item to be palletized from the possible actions in the possible action probability distribution. Specifically, after obtaining the optimized action probability distribution (possible action probability distribution), the action determination unit 103 can select actions based on this distribution. For example, it can select the action with the highest probability as the palletizing action of the item to be palletized by random sampling or by selecting the action with the highest probability.

[0042] In this embodiment, the action search space optimization is combined with the action probability output by the Actor to restrict and optimize the actions sampled by the agent, which can greatly improve the learning time. Furthermore, through the Actor network, the action search space optimization unit, and the action determination unit, the disordered palletizing model can effectively and automatically execute palletizing tasks in complex palletizing environments, and has high execution efficiency and stability.

[0043] In the above Figure 2 Based on the embodiments described above, this application also provides another trained unordered palletizing model. Figure 3 This is a flowchart illustrating the probability distribution of possible actions for determining palletizing status information, as provided in an embodiment of this application. Figure 3 As shown, the action search space optimization unit 102 is specifically used for:

[0044] Step 301: Perform heuristic rule filtering on the action probability distribution of the palletizing status information to obtain the initial action probability distribution.

[0045] In this embodiment, heuristic rule filtering refers to generating an initial placeable region based on a heuristic method, with a fallback limit. If an action in the action probability distribution is not within the initial placeable region, the probability of that action can be set to zero.

[0046] By using heuristic rules to filter the probability distribution of actions in palletizing tasks, inappropriate actions can be effectively eliminated, while those actions that meet the task objectives and environmental conditions can be retained, thus providing a more accurate probability distribution for the final action selection. This method can significantly improve the efficiency and stability of palletizing task execution.

[0047] Step 302: Apply support area constraints and centroid constraints to the initial feasible action probability distribution to determine the feasible action probability distribution of the palletizing status information.

[0048] In this embodiment, the initial feasible action probability distribution is subjected to physical constraint screening to eliminate candidate positions that do not meet physical requirements, further narrowing the action search space and ensuring the actual feasibility of the actions. Physical constraint screening includes support area constraints and centroid constraints, which ensure the stacking stability of the items to be stacked and prevent them from tipping over after stacking.

[0049] Specifically, the support area constraint requires that the area of ​​the supported region of the item to be stacked must exceed a preset threshold, and the mathematical expression is:

[0050] Area(S) / Area(S total )≥α

[0051] Where Area(S) represents the area of ​​the bottom surface of the item to be stacked that is supported. total ) represents the bottom area of ​​the item to be stacked, and α is a preset threshold. For example, α can be greater than or equal to 80%.

[0052] Specifically, the centroid constraint requires that the projection of the centroid of the item to be stacked must lie within the convex hull of the support region. The mathematical expression is:

[0053] (x c ,y c )∈ConvexHull(S)

[0054] Among them, (x c ,y c) represents the centroid projection of the item to be stacked, ConvexHull(S) represents the convex hull region of the support region, and S is the set of vertex coordinates of the support region.

[0055] In this embodiment of the application, heuristic rule filtering and physical constraint fine filtering (support area constraint and centroid constraint) are used to make the probability distribution of feasible actions of the palletizing status information practically feasible, retaining actions that conform to actual physical constraints, task objectives and environmental conditions, and providing a more accurate probability distribution for the final action selection.

[0056] In the above Figure 2 Based on the aforementioned embodiments, this application also provides another trained unordered palletizing model. The Actor network 101 is further used to: determine the palletizing value function based on the palletizing action and palletizing state information of the items to be palletized; and update the Actor network using a policy gradient method based on the expected return of the palletizing value function and the palletizing state information.

[0057] Specifically, the Actor network 101 contains model parameters. At each time step, the current state is obtained from the environment, and then an action (i.e., the palletizing action of the item to be palletized) is selected through the Actor network 101. The palletizing action of the item to be palletized is sent to the environment to obtain the new state and reward after execution. The expected value of the cumulative reward from the current state is calculated using a discount factor. The gradient is calculated using the policy gradient formula, and then the model parameters of the Actor network 101 are updated using the gradient ascent method.

[0058] In this embodiment, the Actor network is updated using the policy gradient method, causing the policy to change in the direction of maximizing cumulative reward. The Actor network continuously optimizes its model parameters through the policy gradient method, thereby achieving more efficient and stable palletizing operations.

[0059] In the above Figure 2 Based on the embodiments described above, this application also provides another trained unordered palletizing model. Figure 4 This is a schematic diagram of the third type of unordered palletizing model trained according to the embodiments of this application, as shown below. Figure 4 As shown, the palletizing status report expectation module 200 includes: a Critic network.

[0060] The Critic network receives palletizing status information, performs reward analysis on this information, and obtains the expected reward for the palletizing status information. The Critic network also iteratively updates itself based on the palletizing actions of the items to be palletized.

[0061] The input to the Critic network is the palletizing state information. After features are extracted by the convolutional neural network (CNN), the features extracted by the convolution are converted into a representation that can output the state value through a fully connected layer. Finally, the state value (i.e., the expected reward of the palletizing state information) is output.

[0062] In this embodiment of the application, the Critic network is used in the palletizing task to provide a value assessment based on the layout and stacking status of the items, thereby promoting the learning process of the entire palletizing model.

[0063] In the above Figure 4 Based on the embodiments described above, this application also provides another trained unordered palletizing model. Figure 5 This is a schematic diagram of the fourth type of unordered palletizing model trained according to the embodiments of this application, as shown below. Figure 5 As shown, the unordered stacking model also includes: reward module 300.

[0064] The reward module 300 is used to determine the palletizing reward value of the palletizing action of the item to be palletized according to the preset reward function.

[0065] The input to the preset reward function is the stacking action of the item to be stacked. Through calculation, the stacking reward value of the stacking action of the item to be stacked can be obtained and output.

[0066] In practical working conditions, if the robot's gripper opens and closes in one direction, a multi-objective optimization function needs to be designed to avoid collisions with already stacked items. In this embodiment, the reward function is preset as follows: the reward function is determined based on the space utilization rate of the pallet, the safe space information of the pallet, and the stable value of the stacking action of the items to be stacked. Specifically, the mathematical expression of the reward function is:

[0067]

[0068] Where α, β, and γ are adaptive weighting coefficients, which can be adjusted in real time based on the current stacking status (such as remaining space ratio, stability score), R is the palletizing reward value, and V current V represents the volume of the items currently to be palletized. pallet V is the volume of the current palletizing pallet. safe S is the volume of the remaining safe space on the pallet after placing the current item to be palletized on it. support D represents the support area of ​​the items to be stacked after stacking. centroid The position of the centroid of the items to be stacked is defined by k1 and k2, which are preset coefficients. This represents the ratio of the volume of the items to be palletized to the volume of the current palletizing pallet, used to maximize the space utilization of the palletizing pallet. This indicates the percentage of remaining safe space on the pallet after placing the current item to be palletized, encouraging the reservation of reasonable space. (k1·S) support +k2·D centroid The stability score (the stability value of the stacking action of the item to be stacked) is dynamically calculated based on the support area of ​​the item after stacking and the position of the centroid of the item after stacking.

[0069] In this embodiment, the Critic network is specifically used to update the Critic network based on the palletizing action of the item to be palletized and the palletizing reward value of the palletizing action. More specifically, the Critic network is specifically used to update the Critic network based on the temporal difference method, the palletizing action of the item to be palletized, and the palletizing reward value of the palletizing action.

[0070] For example, the Critic network contains model parameters. After the palletizing actuator performs a palletizing action on the item to be palletized, the environment returns the next state and the palletizing reward value for the palletizing action. The palletizing reward value reflects the contribution of the current palletizing action to the final goal (such as palletizing efficiency, stability, etc.). The TD (Temporal Difference target) error is calculated using the state values ​​(expected rewards) of the current state and the next state. The model parameters in the Critic network are updated by minimizing the error (the Adam optimizer and backpropagation algorithm can be used to update the network parameters to reduce the error between the predicted state value and the true value). The above steps are performed once at each time step. As training progresses, the Critic network gradually learns more accurate state value predictions, which helps the Actor network to better select the optimal palletizing action.

[0071] In this embodiment, a multi-objective reward function is designed to adapt to actual working conditions and improve palletizing stability. The reward function can also update the model parameters in the Critic network, making the prediction of the state value of the Critic network more accurate, thereby providing effective feedback to the Actor network and optimizing the palletizing effect of the entire model.

[0072] Based on the above embodiments, the embodiments of this application also provide a training method for an unordered palletizing model. The training method for the unordered palletizing model includes: performing multiple iterative training on the unordered palletizing model to obtain a trained unordered palletizing model as described in any of the above embodiments.

[0073] in, Figure 6 This is a flowchart of a training method for an unordered palletizing model provided in an embodiment of this application, as shown below. Figure 6 As shown, in each training session, a group of items to be stacked is sequentially stacked onto the corresponding stacking pallet, specifically including:

[0074] Step 601: Input the posture information of one item to be palletized in the group and the current state space information of the palletizing pallet.

[0075] In this embodiment, each group of items to be stacked corresponds to a stacking pallet. Stacking a group of items sequentially onto its corresponding pallet constitutes one training iteration. The trained unordered stacking model then stacks another group of items sequentially onto its corresponding pallet, completing another training iteration. The more times the model is trained, the better its stacking performance.

[0076] Optionally, having different orientation information for multiple sets of palletized items, and having multiple palletizing pallets corresponding to multiple sets of palletized items, can improve the palletizing effect of the unordered palletizing model.

[0077] In each training session, multiple palletizing operations are required. Each palletizing operation requires the posture information of the item to be palletized and the current state space information of the palletizing tray. The palletizing order can be sequential or random; this embodiment does not impose a specific restriction. Before each palletizing operation, the current state space information of the palletizing tray needs to be determined, rather than using the fixed state space information of the palletizing tray from before this training session. For example, if an item to be palletized has already been palletized on the pallet during the first palletizing operation, then during the second palletizing operation, the current state space information of the palletizing tray should be based on the state control information of the pallet that currently has one item palletized.

[0078] Optionally, each training set of items to be stacked can use a randomly generated sequence of items to simulate the stacking process.

[0079] Step 602: Based on the posture information of the items to be stacked and the current state space information of the stacking pallet, stack the items to be stacked onto the stacking pallet until the training termination condition is met.

[0080] The training termination condition is: all items in a group are placed in the palletizing tray, or the palletizing tray can no longer hold items to be palletized.

[0081] Through multiple training episodes, the model parameters of the Actor network and Critic network are continuously updated. At the beginning of each episode, an empty palletizing tray is initialized, and palletizing decisions are made one by one according to the sequence of items to be palletized, until all items to be palletized have been placed or the current policy can no longer place more items to be palletized. In each palletizing decision, the current Actor network is used to select the palletizing action of the items to be palletized, the reward obtained is calculated according to the reward function, the Critic network is used to estimate the value of the current palletizing state, the Critic network is updated through TD-error, and the Actor network is updated using policy gradient.

[0082] Optionally, before model training, you can set up the environment and parameters, install dependent libraries such as Python, PyTorch, and NumPy, and set training parameters; define the state and action space. The state space is the information of the current pallet and the information of the items to be stacked, which can be represented as a two-dimensional height map. The action space refers to all possible positions of the items to be stacked on the two-dimensional plane based on the size and orientation of the items to be stacked.

[0083] Optionally, a model can be trained in a simulation environment using randomly generated item sequences to obtain a highly efficient palletizing strategy.

[0084] In this embodiment, by using various items to be palletized and corresponding palletizing pallets, the disordered palletizing model is iteratively trained, so that the trained disordered palletizing model can dynamically adjust the palletizing strategy when facing various different industrial scenarios, adapt to different object postures and different stacking spaces, and improve adaptability.

[0085] Based on the above embodiments, embodiments of this application also provide an unordered palletizing method. Figure 7 This is a flowchart of an unordered palletizing method provided in an embodiment of this application, such as... Figure 7 As shown, unordered palletizing methods include:

[0086] Step 701: Obtain the current state space information of the palletizing pallet.

[0087] Optionally, at the beginning of the unordered palletizing method, the environment can be initialized first, loading the unordered palletizing model trained in the pre-learning stage, and setting the pallet size, item size range, robot, camera, etc. according to the actual industrial scenario.

[0088] During the unordered palletizing method, a camera can be used to capture color and depth images of the palletizing pallet, and the current state space information of the palletizing pallet can be extracted through image processing algorithms; alternatively, the current state space information of the palletizing pallet can be determined based on the preset pallet size and palletizing process.

[0089] Step 702: Determine the posture information of the items to be stacked based on the collected images of the items to be stacked.

[0090] Optionally, an RGB-D camera can be used to acquire color and depth images of the items to be stacked in real time, and image processing algorithms can be used to extract the posture information (such as three-dimensional dimensions) of the items to be stacked.

[0091] Step 703: Input the current state space information of the palletizing pallet and the posture information of the items to be palletized into the trained unordered palletizing model to obtain the palletizing action of the items to be palletized.

[0092] The trained unordered palletizing model will be based on the Actor network and the Critic network, combining palletizing state information (current state space information of the palletizing pallet and pose information of the items to be palletized) and action search space optimization to output palletizing actions of the items to be palletized that meet the requirements of the actual environment and stability.

[0093] Step 704: The palletizing action is performed on the items to be palletized by the palletizing execution mechanism.

[0094] In this embodiment of the application, the palletizing execution mechanism can be a robot, including a robotic arm and grippers.

[0095] The robotic arm and grippers are controlled to perform grasping, moving, and placing operations based on the stacking action of the items to be stacked.

[0096] In this embodiment, the trained unordered palletizing model can acquire the current state space information of the palletizing pallet and the posture information of the items to be palletized in real time, determine its palletizing strategy and execute it. It can be applied to a variety of different industrial scenarios, improving adaptability. The reinforcement learning-based unordered palletizing method in this embodiment is suitable for real-time object recognition, grasping and stacking tasks in dynamic environments in logistics, warehousing and intelligent manufacturing, and is especially suitable for scenarios where the object posture, stacking space and arrival order change randomly.

[0097] The unordered palletizing model, training method, and unordered palletizing method proposed in this application enable robots to make autonomous decisions in dynamic environments through reinforcement learning algorithms. This improves environmental adaptability (no need to pre-set object positions and stacking order; adapts to changes in object posture and stacking space in real time; through reinforcement learning strategies, the robot can dynamically adjust palletizing strategies to adapt to changes in object posture, stacking space, and arrival order); enhances online capabilities (palletizing is performed immediately upon item arrival, without buffering or readjustment); and ensures physical stability and improves palletizing efficiency (stack stability and efficiency are ensured through constraint modeling and reward function design; physical constraint modeling and reward function design effectively prevent stack tipping and collisions; action search space optimization strategy reduces computation time; combined with the reward function, efficient space utilization and maximum space utilization are achieved).

[0098] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0099] This application provides an electronic device. Figure 8 This is a schematic diagram of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, it includes: at least one processor 801; and a memory 802 communicatively connected to at least one processor 801; wherein the memory 802 stores instructions executable by at least one processor 801, the instructions being executed by at least one processor 801 to enable at least one processor 801 to run the trained unordered stacking model in the above embodiments and to execute the training method or unordered stacking method in the above embodiments.

[0100] The memory 802 and the processor 801 are connected by a bus. The bus can include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 801 and memory 802 together.

[0101] In this embodiment of the application, the electronic device can be a control chip in a robot, such as a microcontroller unit (MCU).

[0102] This application relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the method embodiments described above.

[0103] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0104] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A trained unordered stacking model, characterized in that, include: Action decision module and palletizing status reward expectation module; The action decision module is used to receive externally input palletizing status information, perform palletizing action probability analysis on the item to be palletized based on the palletizing status information, and output the palletizing action of the item to be palletized to the palletizing execution mechanism; wherein, the palletizing status information includes: state space information of the palletizing pallet and posture information of the item to be palletized; The palletizing status reward expectation module is used to receive the palletizing status information, perform reward analysis on the palletizing status information, and obtain the expected reward of the palletizing status information. The action decision module is also used to perform self-iterative updates based on the expected reward of the palletizing status information, the palletizing action of the item to be palletized, and the palletizing status information. The palletizing status reporting expectation module is also used to perform self-iterative updates based on the palletizing action of the items to be palletized.

2. The unordered stacking model trained according to claim 1, characterized in that, The action decision module includes: an Actor network, an action search space optimization unit, and an action determination unit; The Actor network is used to receive the palletizing status information and determine the action probability distribution of the palletizing status information. The action search space optimization unit is used to filter the action probability distribution of the palletizing status information based on preset rules, and determine the action probability distribution of the palletizing status information. The action determination unit is used to determine the stacking action of the item to be stacked from the possible actions in the probability distribution of possible actions.

3. The trained unordered stacking model according to claim 2, characterized in that, The action search space optimization unit is specifically used for: Heuristic rule filtering is performed on the action probability distribution of the palletizing status information to obtain an initial action probability distribution; The initial feasible action probability distribution is constrained by support area and centroid to determine the feasible action probability distribution of the stacking state information.

4. The unordered stacking model trained according to claim 2, characterized in that, The Actor network is also used for: The palletizing value function is determined based on the palletizing action of the items to be palletized and the palletizing status information. Based on the expected return of the palletizing value function and the palletizing state information, the Actor network is updated using the policy gradient method.

5. The trained unordered palletizing model according to claim 1, characterized in that, The palletizing status report expectation module includes: a Critic network.

6. The trained unordered palletizing model according to claim 5, characterized in that, The unordered stacking model also includes: a reward module; The reward module is used to determine the stacking reward value of the stacking action of the item to be stacked according to a preset reward function; The Critic network is specifically used to update the Critic network based on the stacking action of the item to be stacked and the stacking reward value of the stacking action.

7. The trained unordered stacking model according to claim 6, characterized in that, The preset method for the reward function is as follows: The reward function is determined based on the space utilization rate of the palletizing pallet, the safe space information of the palletizing pallet, and the stable value of the palletizing action of the item to be palletized.

8. The trained unordered stacking model according to claim 6, characterized in that, The Critic network is specifically used for: The Critic network is updated based on the temporal difference method, the stacking action of the item to be stacked, and the stacking reward value of the stacking action.

9. A training method for an unordered stacking model, characterized in that, The method includes: The disordered palletizing model is trained iteratively multiple times to obtain the trained disordered palletizing model as described in any one of claims 1-8 above. In each training session, a group of items to be stacked is sequentially stacked onto the corresponding stacking pallet. Specifically, this includes: inputting the posture information of one item to be stacked in the group and the current state space information of the stacking pallet; based on the posture information of the item to be stacked and the current state space information of the stacking pallet, stacking the item to be stacked onto the stacking pallet until the training termination condition is met; wherein, the training termination condition is: all items in the group to be stacked have been placed in the stacking pallet, or the stacking pallet can no longer hold the items to be stacked.

10. A method for unordered stacking, characterized in that, The method includes: Obtain the current state space information of the palletized pallet; Based on the collected images of the items to be stacked, determine the posture information of the items to be stacked; The current state space information of the palletizing pallet and the posture information of the item to be palletized are input into the unordered palletizing model trained as described in any of claims 1-8 above, so as to obtain the palletizing action of the item to be palletized. The palletizing action is performed on the items to be palletized by the palletizing execution mechanism.

Images