Luggage dynamic stacking method based on point cloud segmentation and reinforcement learning, robot device and application thereof

Abstract

The invention discloses a luggage dynamic stacking method based on point cloud segmentation and reinforcement learning, a robot device and application thereof, and is suitable for complex scenes such as airports. The robot device comprises a three-dimensional sensing module, a calculation processing module, a robot execution module and a ULD loading module, and real-time data interaction is achieved through the industrial Ethernet. The core of the method is'perception-decision-execution 'closed loop: a three-dimensional sensing module captures three-dimensional point cloud data of an operation space, and a structured scene representation is generated through point cloud segmentation, OBB calculation and multi-channel 2D height map construction; based on a reinforcement learning model of an Actor-Critic framework, an optimal 6D placement pose is output through offline training and online execution; and the robot execution module accurately completes luggage grabbing and placing. According to the method, traditional off-line planning is abandoned, real-time sensing replaces ideal model dependence, reinforcement learning replaces fixed rules, the method can dynamically adapt to changes such as luggage deformation and displacement, and stacking robustness, space utilization rate and compatibility are improved.

Description

Technical Field

[0001] This invention relates to the technical field of automated palletizing, specifically to a dynamic baggage palletizing method, robotic device, and its application based on point cloud segmentation and reinforcement learning, which is particularly suitable for automated baggage sorting and loading operations in complex scenarios such as airports. Background Technology

[0002] As the global aviation industry transforms towards digitalization, intelligence, and high-efficiency operation, the Civil Aviation Administration of China (CAAC) and other regulatory bodies are actively promoting the construction of strategies such as "smart airports" and "intelligent logistics hubs." The demand for automated baggage palletizing systems in airport baggage sorting and loading is rapidly increasing. These systems primarily rely on offline planning algorithms based on fixed geometric models and heuristic rules. Offline planning algorithms typically calculate optimal positions within a pre-defined ideal model, but this approach has many limitations in real-world scenarios. First, the algorithm-planned system assumes that baggage carts or unit loading / unloading equipment are rectangular with fixed dimensions, and that the baggage to be placed is also a regular cuboid. However, reality is often far more complex than the model: placed baggage may undergo slight deformation or displacement due to gravity or external pressure; soft-pack baggage, due to its irregular shape, often occupies a space different from the model; during loading, baggage may tip over due to vehicle bumps or unstable placement; and manual intervention by on-site staff may further alter the existing layout. Furthermore, the algorithm lacks real-time perception and adaptation to the real physical world, exhibiting insufficient dynamic response to changes, further exacerbating placement failures and worsening space utilization.

[0003] The limitations of traditional algorithms are particularly evident when dealing with the diversity of baggage. Airport baggage is diverse, including hard-sided suitcases, soft travel bags, backpacks, and irregularly shaped luggage. The mixture and irregularity of these items make it difficult to accurately model any manually set rules. Traditional algorithms are mainly based on a set of hard-coded fixed heuristic rules, such as "larger items first, smaller items later," "heavier items first, lighter items later," "placing items near corners or edges as much as possible," and "keeping the stacking plane." While these rules are applicable to some common situations, they struggle to provide globally optimal solutions in complex stacking scenarios. For example, a seemingly irregular arrangement might create better space for larger items later, but fixed rules cannot learn this long-term, non-intuitive globally optimal strategy, resulting in maximizing overall space utilization. Although some advanced systems introduce visual feedback, this is usually used for "error correction" or "verification" after placement, i.e., checking whether the previous placement was successful. The core logic of the decision-making process itself does not rely on real-time visual input.

[0004] Therefore, existing technologies need to seek more efficient and flexible solutions to address these challenges in order to improve the overall performance and efficiency of airport baggage palletizing systems. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing palletizing robots, such as their inability to perceive the real physical environment, their inability to perform mechanized static palletizing, their poor compatibility with diverse luggage, and their low space utilization due to rigid decision-making strategies. This invention provides a robot device, a dynamic luggage palletizing method, and its application based on point cloud segmentation and reinforcement learning.

[0006] To achieve the above objectives, the solution of the present invention is: A robotic device based on point cloud segmentation and reinforcement learning includes a 3D sensing module, a computing and processing module, a robot execution module, and a ULD loading module. The 3D sensing module is mounted on a gantry structure above a luggage cart and is used to capture complete and high-precision 3D point cloud data of the luggage cart's interior space. The computing and processing module is equipped with a high-performance GPU to run point cloud segmentation algorithms, reinforcement learning models, and data processing programs to achieve scene analysis, state representation, and intelligent decision-making. The robot execution module includes a six-axis palletizing robot and an end effector to perform luggage grasping and placement actions according to the instructions from the computing and processing module. The ULD loading module is used to carry the palletized luggage and realize luggage transfer. The 3D sensing module, robot execution module, and computing and processing module interact with each other via industrial Ethernet for high-bandwidth, low-latency data exchange, forming a closed-loop system of "perception-decision-execution".

[0007] Furthermore, the three-dimensional sensing module is a depth camera and / or lidar, and the scanning field of view of the three-dimensional sensing module completely covers the working space of the ULD loading module; the end effector is a flexible gripper or a vacuum suction cup array to adapt to luggage of different shapes and materials.

[0008] The present invention can also employ the following solutions: A dynamic baggage palletizing method based on the aforementioned point cloud segmentation and reinforcement learning-based robotic device includes the following steps: S1. Real-time environmental perception: The internal working space of the ULD loading module is scanned by the three-dimensional sensing module to obtain the original three-dimensional point cloud data; S2. Scene Analysis and State Representation: The original 3D point cloud data is processed and transformed into a structured 2.5D state representation that can be understood by the reinforcement learning model; the specific process is as follows: S21. Point cloud instance segmentation: The original 3D point cloud data is processed using a point cloud instance segmentation model based on the Transformer architecture to segment each independent baggage point cloud cluster and assign a unique ID to each point cloud cluster. S22: Minimum bounding rectangle (OBB) calculation. After preprocessing the point cloud of each baggage point cloud cluster by point cloud filtering and plane fitting, OBB geometric analysis is performed to output the 6D pose, three-dimensional dimensions and surface normal vectors of each baggage. S23: Construct a multi-channel 2D height map, divide the top view plane into discrete grids, and generate a 2.5D state representation containing height channels and occupancy channels. The height channels store the maximum height value of grid points along the Z-axis, and the occupancy channels use binarization to mark whether a grid is occupied. S3, Intelligent Decision-Making: Input the structured 2.5D state representation into the trained reinforcement learning model and output the 6D pose of the optimal placement position of the target. S4. Robot Execution: The robot execution module plans the path according to the 6D pose instructions and performs baggage grabbing and placement actions. After completing one palletizing cycle, return to step S1 and enter the next "perception-decision-execution" cycle until the palletizing task is completed.

[0009] Furthermore, the reinforcement learning model learns and applies the optimal palletizing strategy through two stages: "offline training" and "online execution." The offline training stage is conducted in a simulation environment, updating the Critic value network and Actor policy network using temporal differential error. The online execution stage inputs a 2D height map of the real environment generated after step S2 processing, along with the size information of the luggage to be placed. The state is then input into the pre-trained Actor policy network for inference and decision-making. The Actor policy network quickly calculates the probability distribution map of the optimal placement position, selecting the position with the highest probability and resolving it as a 6D target placement pose (x, y, z, roll, pitch, yaw), where x, y, and z are three-dimensional spatial coordinates, and roll, pitch, and yaw are the Euler angles of the luggage's rotational attitude in three-dimensional space, used to determine whether the luggage's placement orientation meets the palletizing requirements.

[0010] Furthermore, in step S22, the plane fitting preprocessing uses the RANSAC algorithm, which iteratively selects three points to fit a candidate plane model and calculates the distance from the points to the plane based on the plane equation. The plane equation is expressed as: ,in( a,b,c ) is the normal vector of the plane. , d It is the distance from the origin to the plane, for a given point. Its distance to the plane is:

[0011] in, For three-dimensional points The vertical distance to the target plane; The first in the point cloud cluster i A three-dimensional point instance; For the i-th 3D point 3D coordinates; a,b,c It is the normal vector of the plane. The three components; normal vector of the plane The modulus length; By maximizing the distance less than the threshold The optimal plane is found by determining the number of interior points.

[0012] Furthermore, the reinforcement learning model is built based on the Actor-Critic framework, and the training process of the reinforcement learning model includes the following steps: T1. Initialize the trainable parameters of the Actor policy network and the Critic value network; T2. Start the simulation training loop and obtain the 2D height map of the current simulation environment as the state. S ; T3, Actor Strategy Network Reception Status S Then, output the action probability distribution of all candidate placement positions. P Select target placement action A And execute it in the simulation environment; T4. Obtain the action to be executed. A The new state after S' and corresponding instant rewards R The instant reward R The calculation formula is:

[0013] in, It is a space utilization reward, which is positively correlated with the increase in the volume of used space. It is a stability bonus; if the luggage is stable after placement, it is positive, and if it is not, it is negative. It's a reward for completing the task. These are weighting coefficients; T5 and Critic value networks for the new state S' Perform an evaluation and output the state value. ; T6, Based on Instant Rewards R State value and the current status S value Calculate timing difference error The calculation formula is:

[0014] in, Indicates the first t The timing difference error of each cycle; Indicates execution of the first t The immediate reward obtained when entering state t+1 after the action; This is the discount factor, and its value ranges from (0,1]. This represents the output of the Critic value network, i.e., the state. S The expected cumulative reward value; Represents the (t+1)th state The expected cumulative reward value; Indicates the first t each state The expected cumulative reward value; T7. Utilizing timing difference error Update the parameters of the Critic value network using the loss function. The loss function is:

[0015] in, For all trainable parameters in the Critic value network; N The number of samples in batch training, i.e., the parameters in the first-order value network loss function. The updated formula is:

[0016] in It is the learning rate of the Critic value network. express Critic value network parameters The gradient; The parameter assignment operator indicates that the calculation result on the right side will be updated in the parameter on the left side; T8, Based on timing difference error And policy gradient, update the parameters of the Actor policy network. The updated formula is:

[0017] in Let be the learning rate of the Actor policy network; For all trainable parameters of the Actor policy network; The log-pair parameter represents the output probability of the Actor policy network. The gradient; The output of the Actor policy network, i.e., the state Select action The probability of; Indicates the first t The stacking action executed in each loop; Indicates the first t A loop of scene states; T9. Repeat steps T2-T8 until the decision accuracy of the reinforcement learning model meets the preset threshold, and the training is complete.

[0018] Furthermore, the intelligent decision-making process in step S3 is modeled as a Markov decision process, and its tuples are defined as ( S,A,P, R,γ ),in: S This represents the 2.5D state of the scene. A The posture and movement of luggage to be placed. P Let be the state transition probability. R For instant rewards, γ This is the discount factor.

[0019] Furthermore, in step S4, the specific process executed by the robot is as follows: The calculation and processing module encodes the target pose finally selected in step S3 and sends it to the controller of the robot execution module C; After receiving the instruction, the robot controller calls the movelt2 package in ROS2 to perform path planning and collision detection, and automatically plans and outputs the path. The palletizing robot controls the end effector to grab luggage from the waiting area and stack it into the target location along a planned path.

[0020] Furthermore, in step S23, the resolution of the discrete grid is... N×M Each cell in the grid ( ，j ), All include both height channels and occupied channels; The height channel stores the maximum height value of each grid point on the Z-axis, using the formula:

[0021] in, In a 2D heightmap, the first... The maximum height value of a grid cell, if the cell Cell If empty, then This is the height of the base. Indicates the row and column index of the grid cell; In a 2D heightmap, the first... A discrete grid cell; This represents the first point in the original 3D point cloud data. k points z Axis coordinates; This represents the first point in the original 3D point cloud data. k points shaft and Axis coordinates; The occupied channel is marked with a binary value to indicate whether each grid is occupied by an OBB of a certain baggage, represented as:

[0022] in, In a 2D heightmap, the first... The occupancy status of each grid cell; 1 indicates that the grid cell is occupied, meaning there is an OBB box for luggage in that grid; 0 indicates that the grid cell is not occupied, meaning there is no luggage in that grid and there is remaining available space. These two channels together constitute the minimum and most essential state information required for decision-making. .

[0023] An application of a robot device based on point cloud segmentation and reinforcement learning in airport baggage sorting and loading is disclosed. The robot device based on point cloud segmentation and reinforcement learning, along with the baggage dynamic palletizing method, enables automated dynamic palletizing of different types of baggage in complex airport scenarios.

[0024] By adopting the above solution, the present invention has the following beneficial effects compared with the prior art: Strong dynamic adaptability and robustness: This invention constructs a closed-loop system of "perception-decision-execution", which performs real-time and complete perception of the physical environment before each decision cycle. It can respond instantly to dynamic changes such as luggage deformation, displacement, tipping and manual intervention, and operate stably in a real semi-structured working environment, solving the problem of poor robustness of existing technologies.

[0025] Extensive baggage compatibility: By employing advanced point cloud instance segmentation technology (such as OneFormer3D) and OBB geometric analysis, the actual occupancy form of baggage can be directly analyzed from point cloud data without relying on preset models. It can effectively handle various objects such as hard cases, soft bags, and irregularly shaped baggage, greatly expanding the applicability of automated palletizing technology.

[0026] Globally optimal decision-making capability: By adopting a reinforcement learning model based on the Actor-Critic framework, a long-term, global loading strategy is learned through massive simulation training. This can dynamically balance the stability of the current placement with the availability of future space, achieving higher loading density and solving the problem of low space utilization caused by traditional fixed heuristic rules.

[0027] Highly efficient real-time response capability: Each module achieves high-bandwidth, low-latency data interaction through industrial Ethernet, combined with GPU-accelerated deep learning models, ensuring that each decision loop is completed within seconds, which can meet the high-efficiency operation requirements of airport baggage palletizing.

[0028] This invention can fundamentally solve the core pain points of existing airport baggage automated palletizing systems, significantly improve the level of automation, operational efficiency, stability and space utilization of palletizing, and has important practical application value and commercial prospects. Attached Figure Description

[0029] Figure 1 This is a schematic block diagram of the robot device of the present invention.

[0030] Figure 2 This is a simplified flowchart of the present invention.

[0031] Figure 3 This is a flowchart of the scenario analysis and state representation of the present invention.

[0032] Figure 4 This is a flowchart of the model training phase of the present invention.

[0033] Figure 5 This is a flowchart of the execution phase of the present invention. Detailed Implementation

[0034] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0035] This invention discloses a robotic device and a dynamic baggage palletizing method based on point cloud segmentation and reinforcement learning. The core idea is to construct a real-time closed-loop robotic device, abandoning the traditional offline "plan once, execute to the end" approach and instead mimicking the "observation-thinking-action" model of a human operator. Before deciding to place each piece of baggage, the robotic device uses 3D vision to "re-examine" the current real physical world. Based on this updated, fully realistic scene, it uses a reinforcement learning "brain" to make the optimal decision, finally sending the placement coordinates to the robot actuator for path planning and execution. This process can be summarized as a continuously looping "perception-decision-execution" intelligent closed loop. The essence of this invention is to replace the reliance on idealized models with real-time, high-precision 3D environmental perception; and to replace fixed, locally optimal human experience rules with reinforcement learning, which possesses self-learning and global optimization capabilities. In this way, the palletizing robot can instantly respond to and adapt to various dynamic changes occurring during the operation—whether it's the slight deformation and displacement of baggage due to gravity, the stacking uncertainty caused by irregular objects such as soft bags, or even temporary intervention by on-site personnel. This extreme dynamic adaptability is the fundamental guarantee for the invention to achieve high robustness, high space utilization and strong compatibility.

[0036] The specific implementation of this invention follows a closed-loop process of "perception-decision-execution" and relies on the close collaboration of three core hardware modules.

[0037] like Figure 1 As shown, the robot device based on point cloud segmentation and reinforcement learning of the present invention includes a three-dimensional sensing module A, a computing and processing module B, a robot execution module C, and a ULD loading module D. The three-dimensional sensing module A, computing and processing module B, and robot execution module C achieve high-bandwidth, low-latency data interaction through industrial Ethernet to ensure the real-time performance of the entire closed-loop system.

[0038] The 3D sensing module A acts as the "eyes" of the robot, capturing complete and high-precision 3D point cloud data of the interior space of the luggage cart. It typically uses an industrial-grade depth camera or LiDAR, or a combination of both, mounted on the gantry structure above the luggage cart to ensure the scanning field of view completely covers the working space.

[0039] The computational processing module B, acting as the "brain" of the robotic device, is responsible for running all the core algorithms of this invention. It is typically an industrial PC or server equipped with a high-performance GPU. The GPU is primarily used to accelerate the parallel computation of deep learning models (point cloud segmentation, reinforcement learning).

[0040] The robot execution module C acts as the "hands" of the robot device, its function being to precisely execute the grasping and placement of luggage according to the instructions issued by the computing module. The robot execution module C can consist of a six-axis industrial palletizing robot and an end effector (such as a flexible gripper or vacuum suction cup array) that can adapt to different luggage shapes and materials.

[0041] ULD (Unit Load Device) loading modules are containers or pallets used in air transport for loading, consolidating, and transporting goods. They are standard containers or pallets for air transport. The main function of ULD loading modules is to aggregate cargo, baggage, and mail for efficient, safe, and rapid loading, unloading, and transport, maximizing the use of aircraft cargo hold space and protecting goods from damage during transit.

[0042] like Figure 2 As shown, this invention also discloses a dynamic baggage palletizing method based on the aforementioned point cloud segmentation and reinforcement learning-based robotic device. Its core algorithm defines each specific step in the "perception-decision-execution" loop. The dynamic baggage palletizing method specifically includes the following steps: S1, Real-time Environmental Awareness Objective: To capture an accurate "snapshot" of the physical world inside the luggage cart at the current moment.

[0043] Implementation: At the beginning of each palletizing cycle, control the 3D sensing module A to scan the work space once.

[0044] Output: A raw 3D point cloud dataset containing millions of 3D coordinate points. This raw 3D point cloud dataset serves as the sole and reliable basis for all subsequent calculations, ensuring that decisions are made based on facts.

[0045] S2: Scene Analysis and State Representation Objective: To transform raw, unordered initial 3D point cloud data into a structured, information-rich scene representation that can be understood by a reinforcement learning model. Scene parsing and state representation are key bridges connecting "perception" and "decision-making."

[0046] Implementation details: such as Figure 3 As shown, scene analysis and state representation specifically include the following steps: S21. Point Cloud Instance Segmentation: In the computation processing module B, an advanced point cloud instance segmentation model based on the Transformer architecture (such as an improved variant of OneFormer3D) is run. The advanced point cloud instance segmentation model can operate directly in 3D space, and even in complex scenes where luggage is stacked and occluded, it can accurately segment each independent luggage instance and assign a unique ID to each instance.

[0047] S22. Minimum Bounding Rectangle (OBB) Calculation: For each baggage point cloud cluster segmented in the previous step, a series of preprocessing steps are performed, including point cloud filtering and plane fitting. Then, OBB (Oriented Bounding Box) geometric analysis is performed on the baggage instances. Plane fitting typically uses the RANSAC (Random Sample Consensus) algorithm to robustly identify the main planes in the point cloud cluster to determine its stable supporting surface. The plane fitting process iteratively selects the minimum subset of points (3 points for a plane) to fit a candidate plane model. The plane equation can be expressed as:

[0048] in (a,b,c) It is the normal vector of the plane. , d It is the distance from the origin to the plane.

[0049] For a point Its distance to the plane is:

[0050] in, For three-dimensional points The vertical distance to the target plane; The first in the point cloud cluster i A three-dimensional point instance; For the first i Three-dimensional points 3D coordinates; a,b,c It is the normal vector of the plane. The three components; normal vector of the plane The length of the module.

[0051] The RANSAC algorithm maximizes distances less than a threshold. The optimal plane is found by counting the number of interior points. This step not only outputs the precise 6D pose (position x, y, z and attitude roll, pitch, yaw) and 3D dimensions of each bag in its current orientation, but more importantly, it calculates the normal vectors of each surface using the OBB's pose matrix. These normal vectors are crucial for determining whether a surface is flat enough to serve as a subsequent baggage support platform.

[0052] S23. Constructing a multi-channel 2D height map: This is the core step in information fusion and dimensionality reduction. Logically divide the overhead view of the luggage cart into a fine grid (e.g., 100x100). Then, generate a 2.5D image containing multiple information channels for this grid as the final state representation. Let the overhead view be a discrete grid.G Its resolution is N×M For each cell in the grid (i, j) , Define the following channels: HeightChannel: Stores the maximum height value of each grid point on the Z-axis. Represents the overall "terrain" of the current stacked scene.

[0053]

[0054] in, In a 2D heightmap, the first... The maximum height value of a grid cell, if the cell Cell If empty, then This is the height of the base. Represents the row and column indices of a grid cell, satisfying i∈[1,N] , j∈ [1，M] , N×M For grid resolution; In a 2D heightmap, the first... A discrete grid cell; This represents the first point in the original 3D point cloud data. k The z-axis coordinates of each point; This represents the k-th point in the original 3D point cloud data. shaft and Axis coordinates.

[0055] Occupancy Channel: Each grid cell is marked with a binary value (0 or 1) to indicate whether it is occupied by an OBB of a certain baggage. Subsequent decisions in this invention are based on this simple and easy-to-learn state representation.

[0056]

[0057] in, In a 2D heightmap, the first... The occupancy status of each grid cell; 1 indicates that the grid cell is occupied, meaning there is an OBB box for luggage in that grid; 0 indicates that the grid cell is not occupied, meaning there is no luggage in that grid and there is remaining available space.

[0058] These two channels together constitute the minimum and most essential state information required for decision-making.

[0059] S3, Intelligent Decision-Making Objective: Based on the current scene state (i.e., the 2.5D heightmap generated in the previous step) and the information of the luggage to be placed, make the optimal placement decision. The entire decision-making process is modeled as a Markov Decision Process (MDP), consisting of a tuple. The definition, explained in detail below: state S 2.5D state representation of the scene ,in, Indicates the first t The scenario state of a palletizing cycle, i.e. the "state" in the Markov decision process; t The time-series index representing the palletizing cycle; Indicates the first t The height channel data of each cycle; Indicates the first t The data of the occupied channels in each cycle.

[0060] Action A: Place the luggage to be placed on the grid. (i,j) A certain position or movement on the body.

[0061] Transition probability In state Execute action Later transitioned to a new state The probability of this transition is deterministic in the simulation environment.

[0062] reward function The immediate reward received after performing an action. A well-designed reward function is key to the model's success.

[0063] in, Indicates the first t The total instant reward obtained after performing the placement action in each loop; It is a space utilization reward, which is positively correlated with the increase in the volume of used space. It is a stability bonus; if the luggage is stable after placement, it is positive, and if it is not, it is negative. It is a reward for completing the task; It is the weighting coefficient.

[0064] As a discount factor, The value is between (0,1] and is used to balance the importance of immediate rewards and future rewards.

[0065] This invention employs a reinforcement learning model based on the Actor-Critic framework. The reinforcement learning model learns and applies the optimal palletizing strategy through two phases: "offline training" and "online execution."

[0066] like Figure 4 As shown, the offline training phase takes place in a simulation environment, which is the process by which the reinforcement learning model understands the environment and learns strategies. The training process of a reinforcement learning model includes the following steps: Environment and State: The baggage stacking process is simulated in a highly realistic physical simulation environment. The scene at each moment is represented as a 2D height map (state S) consistent with the real world.

[0067] Actor Policy Network : Receive current state S Its output is a probability distribution map across all candidate placement locations. This probability distribution map represents the "recommendation" of placing the luggage in each location under the current state.

[0068] Critic Value Network : Receive current state S Its output is a scalar value score. This score assesses the "goodness" or "badness" of the current situation, that is, how high the total reward can be expected in the future from the current state.

[0069] Learning feedback (reward signal): After performing a placement action in the simulation environment, a reward value is calculated based on the result. (Reward, R) The reward function is carefully designed to balance multiple objectives, such as: space utilization reward: a positive reward is given if the placement increases the volume of used space. Stability reward: a positive reward is given if the stability of the luggage is high after placement (calculated by the physics engine); a large negative reward (penalty) is given if it tipps over. The reinforcement learning model learns to determine stability on its own without external input. Task completion reward: a small positive reward is given for successfully loading one piece of luggage, and a large positive reward is given for completing the entire palletizing task.

[0070] Updates and optimizations: The core of reinforcement learning models lies in utilizing temporal difference errors. TD-Error To simultaneously optimize the Actor and Critic value networks. Temporal difference error. TD-Error The calculation formula is as follows:

[0071] in, Indicates the first t The timing difference error of each cycle; Indicates execution of the first t After the next action, enter t+1 The instant reward obtained in each state; This is the discount factor, and its value ranges from (0,1]. This represents the output of the Critic value network, i.e., the state.S The expected cumulative reward value; Indicates the first t+1 each state The expected cumulative reward value; Indicates the first t each state The expected cumulative reward value.

[0072] Critic Value Network Updates: The Critic Value Network learns to predict future total rewards more accurately. If the value of a state is underestimated, TD-Error drives the Critic Value Network to increase the prediction of that state's value, and vice versa. The loss function of the Critic Value Network. for:

[0073] in, For all trainable parameters in the Critic value network; N The number of samples in batch training, i.e., the parameters in the first-order value network loss function. The updated formula is:

[0074] in It is the learning rate of the Critic value network. express Critic value network parameters The gradient; The parameter assignment symbol indicates that the calculation result on the right side will be updated to the parameter on the left side.

[0075] The Actor policy network adjusts its policy based on the "guidance" of the Critic value network. If, after performing an action, the Critic value network finds that it has entered a better-than-expected state (i.e., the TD-Error is positive), then the Actor policy network will increase the probability of choosing that action in the future; conversely, it will decrease the probability. Parameters are updated using the policy gradient method.

[0076]

[0077] in Let be the learning rate of the Actor policy network; For all trainable parameters of the Actor policy network; The log-pair parameter represents the output probability of the Actor policy network. The gradient; The output of the Actor policy network, i.e., the state Select action The probability of; Indicates the first t The stacking action performed in each loop involves placing the luggage to be placed onto the grid. A certain pose; Indicates the first t A loop of scene states.

[0078] In this way, the Actor policy network learns to make "visionary" decisions that can bring high long-term returns.

[0079] like Figure 5 As shown, the online execution phase, i.e., running in the real physical world, is the process by which the reinforcement learning model applies policies in the real world. Specifically, it includes: Input: Obtain a 2D height map of the real environment after processing in step S2, and the size information of the luggage to be placed.

[0080] Decision-making: Input the state into the pre-trained Actor policy network to perform reasoning and decision-making; Output: The Actor policy network quickly calculates the probability distribution of the optimal placement position. The calculation and processing module selects the position with the highest probability (or fine-tunes it by combining other constraints) and resolves it into a precise 6D pose (x, y, z, roll, pitch, yaw). x, y, and z are three-dimensional spatial coordinates, clearly defining the "landing point" of the luggage in the three-dimensional working environment. This is the basis for the robot's path planning. The robot needs to first locate the horizontal position using x and y, and then locate the height using z to achieve precise grasping and placement. Roll (roll angle), pitch (pitch angle), and yaw (yaw angle) are Euler angles describing the rotational attitude of the luggage in three-dimensional space. They rotate around the x, y, and z axes respectively, with the geometric center of the luggage as the origin of rotation, and are used to determine whether the placement direction of the luggage meets the palletizing requirements. The three parameters roll, pitch, and yaw together constitute "attitude information," solving the problem of traditional palletizing focusing only on "position" and ignoring "angle"—for irregular objects such as soft bags and irregularly shaped luggage, their placement angle directly affects stacking stability and space utilization.

[0081] S4, Robot Execution Objective: To translate the optimal placement decision generated in the previous step into real and precise actions in the physical world.

[0082] Implementation: The calculation and processing module encodes the final selected target pose (a 6D coordinate system containing x, y, z, roll, pitch, and yaw) and sends it to the controller of the robot execution module C.

[0083] After receiving the instruction, the robot controller calls the movelt2 package in ROS2 to perform path planning and collision detection, and automatically finds a reasonable path to output.

[0084] The palletizing robot drives six or more of its joints to control the end gripper, grabbing luggage from the waiting area and placing it precisely and smoothly into the target location along a planned path.

[0085] After completing this placement, immediately enter the next palletizing cycle, return to S1, and begin a brand new "perception-decision-execution" process for the next piece of luggage to arrive, until the entire loading task is completed.

[0086] The core innovation of this invention lies in providing a robot dynamic palletizing method and its overall architecture based on real-time perception. It abandons the offline planning of existing technologies that rely on historical information or ideal models, and establishes a closed-loop method of "perception-decision-execution" that fully perceives the real physical world before each decision cycle. This enables the robot device to adapt to and handle various dynamic changes (such as luggage tipping and human intervention) in real time, and has extremely high robustness.

[0087] This invention also innovatively proposes a data processing and information fusion method for transforming 3D point clouds into structured 2.5D state representations. Its innovation lies in how to "prepare input" for reinforcement learning models, which includes a novel end-to-end processing flow: a) using an advanced point cloud instance segmentation algorithm (based on a OneFormer3D variant of Transformer) to process complex stacked scenes; b) performing minimum bounding rectangle analysis on the segmented irregular point cloud set to obtain accurate pose and surface flatness information; c) fusing the above multi-source information into a multi-channel 2D height map, creatively transforming the 3D problem into an information-rich 2.5D image problem.

[0088] The global optimal placement decision model based on reinforcement learning, i.e., the reinforcement learning model, is also the innovative decision-making method of this invention. It replaces the traditional, fixed heuristic rules based on human experience with a reinforcement learning model (Actor-Critic framework) trained on massive simulations. This reinforcement learning model can learn long-term, non-intuitive, globally optimal loading strategies from high-dimensional state representations, dynamically balancing the two objectives of space utilization and stability.

[0089] The system-level application of this invention in the airport baggage palletizing scenario: combining the above-mentioned complete method, which includes three major innovations, with actual hardware to construct an automated robotic device that can be applied to the specific, complex, and commercially valuable scenario of airport baggage loading.

[0090] The above embodiments and figures are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

[0091] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A robot device based on point cloud segmentation and reinforcement learning, characterized by, Comprise: A three-dimensional sensing module mounted on the gantry structure above the luggage cart, used to capture complete and high-precision three-dimensional point cloud data of the interior space of the luggage cart; A computing processing module configured with a high-performance GPU, used to run point cloud segmentation algorithms, reinforcement learning models, and data processing programs to realize scene analysis, state representation, and intelligent decision-making; A robot execution module, including a six-axis palletizing robot and an end effector, used to perform luggage grabbing and placing actions according to the instructions of the computing processing module; A ULD loading module for carrying the palletized luggage and realizing luggage transfer; Wherein, the three-dimensional sensing module and the robot execution module interact with the computing processing module through industrial Ethernet for high-bandwidth, low-latency data interaction, forming a closed-loop system of "perception-decision-execution".

2. The point cloud segmentation and reinforcement learning based robot device of claim 1, wherein: The three-dimensional sensing module is a depth camera and / or a laser radar, and the scanning field of view of the three-dimensional sensing module completely covers the working space of the ULD loading module; The end effector is a flexible gripper or a vacuum suction cup array to adapt to luggage of different shapes and materials.

3. A luggage dynamic stacking method based on the point cloud segmentation and reinforcement learning robot device according to claim 1 or 2, characterized in that, Comprise the following steps: S1, real-time environment perception: scan the interior working space of the ULD loading module through the three-dimensional sensing module to obtain raw three-dimensional point cloud data; S2, scene analysis and state representation: process the raw three-dimensional point cloud data to convert it into a structured 2.5D state representation that the reinforcement learning model can understand; the specific process is: S21, point cloud instance segmentation: use a point cloud instance segmentation model based on the Transformer architecture to process the raw three-dimensional point cloud data, segment each independent luggage point cloud cluster, and assign a unique ID to each point cloud cluster; S22: Minimum Enclosing Rectangle OBB Calculation, after point cloud filtering and plane fitting preprocessing for each luggage point cloud cluster, perform OBB geometric analysis to output the 6D pose, three-dimensional size, and surface normal vector of each luggage; S23: Construct a multi-channel 2D height map, divide the top-down plane into discrete grids, and generate a 2.5D state representation containing height channels and occupancy channels, where the height channel stores the maximum height value of the grid point Z-axis, and the occupancy channel marks whether the grid is occupied with binary; S3, intelligent decision-making: input the structured 2.5D state representation into the trained reinforcement learning model to output the 6D pose of the target optimal stacking position; S4, robot execution: the robot execution module plans the path according to the 6D pose instruction and performs the luggage grabbing and placing action; After completing a stacking cycle, return to step S1 and enter the next "perception-decision-execution" cycle until the stacking task is completed.

4. The luggage dynamic palletizing method of claim 3, wherein: The reinforcement learning model learns and applies the optimal stacking strategy through the "offline training" and "online execution" two stages; The offline training stage is carried out in a simulation environment, and the Critic value network and Actor policy network are updated through the time difference error; The online execution stage inputs the 2D height map generated after step S2 processing in the real environment, as well as the size information of the luggage to be placed; The state is input into the trained Actor policy network for inference decision; the Actor policy network quickly calculates the probability distribution of the optimal placement position, and selects the position with the highest probability as the 6D target placement pose (x, y, z, roll, pitch, yaw), x, y and z are three-dimensional space coordinates, and roll, pitch and yaw are Euler angles of the rotation attitude of the luggage in the three-dimensional space, which are used to determine whether the placement direction of the luggage meets the stacking requirements.

5. The luggage dynamic palletizing method of claim 3, wherein: In step S22, the plane fitting preprocessing adopts the RANSAC algorithm, selects three points to fit a candidate plane model through iteration, and calculates the distance from the point to the plane based on the plane equation, The equation of a plane is given by: where (a, b, c) is the normal vector to the plane a,b,c d and d is the distance from the origin to the plane, for a point whose distance to the plane is:​​ wherein is a three-dimensional point to the target plane; is the i-th three-dimensional point instance in the point cloud cluster; i is the i-th three-dimensional point instance in the point cloud cluster; is the i-th three-dimensional coordinate of the i-th three-dimensional point is the i-th three-dimensional coordinate of the i-th three-dimensional point a,b,c is the three components of the normal vector of the plane; is the length of the normal vector of the plane; The best plane is found by maximizing the number of inliers with a distance less than a threshold .

6. The luggage dynamic palletizing method of claim 3 or 4, wherein: The reinforcement learning model is constructed based on an Actor-Critic framework, and the training process of the reinforcement learning model includes the following steps: T1, initialize the trainable parameters of the Actor policy network and the Critic value network; T2, start the simulation training cycle, get the 2D height map of the current simulation environment as the state S ; T3, Actor policy network receives state S After, output action probability distribution for all candidate bin locations P , select target bin action A and execute in simulation environment; T4, acquire execution action A new state after S' and the corresponding immediate reward R , the calculation formula of the immediate reward R ​ wherein, is a space utilization reward, positively correlated with the increment of used space volume; is a stability reward, positive if the luggage is stable after being placed, otherwise negative; is a task completion reward, is a weight coefficient; T5, Critic value network outputs state value S' for evaluation ; T6, Based on Instant Rewards R State value and the current status S value Calculate timing difference error The calculation formula is: in, Indicates the first t The timing difference error of each cycle; Indicates execution of the first t The immediate reward obtained when entering state t+1 after the action; This is the discount factor, and its value ranges from (0,1]. This represents the output of the Critic value network, i.e., the state. S The expected cumulative reward value; Represents the (t+1)th state The expected cumulative reward value; Indicates the first t each state The expected cumulative reward value; T7, utilize timing difference error updating the parameters of the Critic value network by a loss function the loss function is where, are all trainable parameters in the Critic value network; N is the number of samples in a batch, i.e., the number of parameters in the Critic value network loss function The update formula is: wherein is the learning rate of the Critic value network, denotes the gradient of the Critic value network parameters ; is the parameter assignment symbol, indicating the update of the right-hand side calculation result to the left-hand side parameters. T8, timing difference error and policy gradient, update the parameters of the actor policy network , the update formula is: in Let be the learning rate of the Actor policy network; For all trainable parameters of the Actor policy network; The log-pair parameter represents the output probability of the Actor policy network. The gradient; The output of the Actor policy network, i.e., the state Select action The probability of; Indicates the first t The stacking action executed in each loop; Indicates the first t A cycle of scene states; T9, repeat steps T2-T8 until the decision accuracy of the reinforcement learning model meets the preset threshold, and complete the training.

7. The luggage dynamic palletizing method of claim 3, wherein: The intelligent decision process of step S3 is modeled as a Markov decision process, whose tuple is defined as ( S,A,P,R,γ ), where: S is a 2.5D state representation of the scene, A is a pose action to place the luggage, P is a state transition probability, R is an immediate reward, γ is a discount factor.

8. The luggage dynamic palletizing method of claim 3, wherein: In step S4, the specific process executed by the robot is as follows: The calculation processing module encodes and sends the final selected target pose in step S3 to the controller of the robot execution module C; After receiving the instruction, the robot controller calls the movelt2 package in ros2, performs path planning and collision detection, and automatically plans the path output; The stacking robot controls the end gripper to grab the luggage from the waiting area and stacks the luggage to the target position along the planned path.

9. The luggage dynamic palletizing method of claim 3, wherein: In step S23, the resolution of the discrete grid is N×M Each cell in the grid ( ，j ), All include both height channels and occupied channels; The height channel stores the maximum height value of each grid point on the Z axis, and the formula is: wherein, represents the maximum height value of the th discrete grid cell in the 2D height map, if the cell Cell is empty, then is the base height; represents the row and column index of the grid cell; represents the th discrete grid cell in the 2D height map; represents the k th point in the original 3D point cloud data z axis coordinate; represents the k th point in the original 3D point cloud data axis and axis coordinate; The occupancy channel uses a binary value to mark whether each grid is occupied by the OBB of a certain luggage, which is represented as: in, In a 2D heightmap, the first... The occupancy status of each grid cell; 1 indicates that the grid cell is occupied, meaning there is an OBB box for luggage in that grid; 0 indicates that the grid cell is not occupied, meaning there is no luggage in that grid and there is remaining available space. These two channels together constitute the minimum, most essential state information needed for decision making .

10. Use of a robotic device in airport baggage sorting and loading, characterized in that: The robot device of claim 1 or 2, and the luggage dynamic stacking method based on point cloud segmentation and reinforcement learning of any one of claims 3 to 9, realizes the automatic dynamic stacking operation of different types of luggage in complex airport scenes.

Images