A cargo loading method, device and equipment based on deep reinforcement learning

Abstract

The application discloses a cargo loading method, device and equipment based on deep reinforcement learning, comprising the following steps: scanning a loading space through a multi-modal sensor to obtain three-dimensional point cloud data, constructing based on the three-dimensional point cloud data to obtain a three-dimensional container model corresponding to the loading space; identifying and 6D pose estimating a cargo to be loaded to obtain size and attitude information corresponding to the cargo to be loaded; generating a plurality of candidate placement poses according to the three-dimensional container model and the size and attitude information; determining an optimal placement pose in the candidate placement poses by using a pre-constructed deep reinforcement decision network, and performing a placement operation of the cargo to be loaded at the optimal placement pose by controlling a mechanical arm; after the placement operation is completed, re-scanning the loading space to obtain a current three-dimensional model, calculating an execution error, and updating the three-dimensional container model and network parameters of the deep reinforcement decision network by using the execution error.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent cargo loading, and in particular to a cargo loading method, device and equipment based on deep reinforcement learning. BACKGROUND

[0002] With the rapid development of intelligent manufacturing and smart logistics, automated loading technology has become a key link to improve operation efficiency and reduce labor costs. In highly structured industrial scenarios, such as palletizing operations on standardized production lines, traditional robotic systems can achieve good loading results through pre-programming and fixed trajectory planning. However, when the demand for automated loading is extended to more extensive fields such as smart logistics, air freight, e-commerce warehousing, etc., the operating environment faced is transformed into an unstructured scenario. In such scenarios, the cargo to be loaded (such as airport luggage, express packages) often has significant differences in size, shape, material and weight, and lacks uniform standards; at the same time, the internal space of the loading container (such as a truck compartment, an airplane luggage compartment, a container) is often irregular and complex, which poses higher requirements on automated loading systems.

[0003] Specifically, the existing technology mainly has the following limitations, including: in terms of environmental perception, most systems rely on two-dimensional vision or simple depth information, making it difficult to achieve high-precision three-dimensional reconstruction and dynamic updating of the loading space, resulting in the system being unable to accurately perceive the constantly changing loading state. In terms of object recognition and positioning, traditional methods often need to rely on precise pre-defined models (such as CAD models) of the objects to be operated, and it is difficult to achieve stable recognition and accurate six-dimensional pose estimation for non-standard shaped cargo, making the basis for grasping and placing operations unstable. In terms of placement decision-making, existing solutions mostly use heuristic algorithms based on geometric rules or single objectives (such as maximizing space filling rate), lacking the ability to comprehensively weigh multiple objectives, resulting in decision-making results often not being optimal. In terms of execution control, most systems use open-loop control strategies, lacking real-time monitoring and feedback adjustment mechanisms for execution results, making it easy for cumulative errors to lead to loading failures or space waste in actual operations, limiting the widespread application of automated systems in complex real-world environments. SUMMARY

[0004] Therefore, the present application aims to provide a cargo loading method, device, equipment and computer program product based on deep reinforcement learning, which can overcome the problems of poor adaptability, low intelligence level and lack of self-learning ability of existing automated loading technology in unstructured environments, thereby improving the autonomous, intelligent, efficient and robust loading operations of unknown cargo in irregular spaces.

[0005] To achieve the above object, the application provides a cargo loading method based on deep reinforcement learning, which comprises the following steps:

[0006] The loading space is scanned by a multi-modal sensor to obtain three-dimensional point cloud data, and a three-dimensional container model corresponding to the loading space is constructed based on the three-dimensional point cloud data;

[0007] The to-be-loaded cargo is identified and 6D pose estimation is performed to obtain size and attitude information of the to-be-loaded cargo;

[0008] According to the three-dimensional container model and the size and attitude information, a plurality of candidate placement poses are generated;

[0009] The optimal placement pose in the candidate placement poses is determined by using a pre-constructed deep reinforcement decision network, and a placement operation of the to-be-loaded cargo in the optimal placement pose is performed by controlling a mechanical arm;

[0010] After the placement operation is completed, the loading space is rescanned to obtain a current three-dimensional model, and an execution error is calculated, and the three-dimensional container model and the network parameters of the deep reinforcement decision network are updated using the execution error.

[0011] Preferably, the three-dimensional point cloud data is obtained by scanning the loading space by a multi-modal sensor, and the three-dimensional container model corresponding to the loading space is constructed based on the three-dimensional point cloud data, which comprises:

[0012] A depth image of the loading space is obtained by an RGB-D camera, and the depth image is converted into three-dimensional point cloud data;

[0013] The three-dimensional point cloud data is preprocessed including filtering and noise reduction operations, and the preprocessed three-dimensional point cloud data is constructed or updated by using a voxel-based truncated signed distance function fusion algorithm to obtain the three-dimensional container model; wherein,

[0014] The loading space is divided into a plurality of three-dimensional voxel grids, the truncated signed distance value of each voxel to the nearest object surface is calculated, and the truncated signed distance values of continuous multiple frames of depth images are fused into the voxels by weighted average method;

[0015] The truncated signed distance value and the corresponding weight of each voxel are updated in real time to obtain the three-dimensional container model.

[0016] Preferably, the truncated signed distance value of each voxel to the nearest object surface is calculated, and the truncated signed distance values of continuous multiple frames of depth images are fused into the voxels by weighted average method, which comprises:

[0017] Each voxel is projected into the current image frame to obtain the corresponding pixel coordinates and projection depth value;

[0018] Based on the projected depth value and the measured depth value of the corresponding pixel coordinate in the depth image, a signed distance function value of the current image frame is calculated, and the formula is expressed as: , wherein sdf k (x) represents the signed distance function value, z represents the projected depth value, d k (u) represents the measured depth value;

[0019] The signed distance function value is truncated within a preset threshold range to obtain a truncated signed distance value corresponding to the current image frame, and the formula is expressed as: , wherein tsdf k (x) represents the truncated signed distance value of the current image frame, and μ represents the preset threshold;

[0020] For each voxel, the updated truncated signed distance value and weight are calculated by weighted average according to the truncated signed distance value and weight of the previous image frame and the truncated signed distance value and weight of the current image frame, and the formula is expressed as: , , wherein and respectively represent the truncated signed distance value and weight of the voxel x in the previous image frame, tsdf k (x) and respectively represent the truncated signed distance value and weight of the current image frame.

[0021] Preferably, the identification of the to-be-loaded cargo and the 6D pose estimation obtain size and pose information of the to-be-loaded cargo, including:

[0022] An RGB image of the to-be-loaded cargo is obtained, and a pixel-level mask of the to-be-loaded cargo is extracted from the RGB image by using a preset instance segmentation network;

[0023] Based on the pixel-level mask and the point cloud data corresponding to the pixel-level mask, a three-dimensional bounding box size of the to-be-loaded cargo and a 6D pose including translation and rotation information relative to a camera coordinate system are regressed by a preset 6D pose estimation network to obtain the size and pose information.

[0024] Preferably, the generation of a plurality of candidate placement poses according to the three-dimensional container model and the size and pose information includes:

[0025] The three-dimensional container model is geometrically analyzed to identify all available free areas, support surfaces are identified in the free areas, and discrete three-dimensional coordinate points are generated on the support surfaces by using a surface sampling strategy;

[0026] For each of the three-dimensional coordinate points, a candidate placement point set including a plurality of pose-position combinations is generated in combination with the size and pose information and six basic rotation poses of the to-be-loaded cargo;

[0027] Verification is performed on each of the pose-position combinations including collision detection and validity verification, and all of the pose-position combinations that pass the verification are determined as valid candidate placement poses.

[0028] Preferably, the construction process of the deep reinforcement decision network comprises:

[0029] A network architecture including a policy network and a value network is constructed, wherein,

[0030] The state feature vector obtained by transforming the current environment state including the three-dimensional container model and the size and pose information is received by the policy network as input, and a placement action is output.

[0031] The value network performs value evaluation according to the current environment state and the placement action, outputs a corresponding state-action value evaluation, and uses the state-action value evaluation to guide the optimization of the policy network;

[0032] Based on the network architecture, the deep reinforcement learning network is trained by a deep reinforcement learning algorithm and based on a reward signal, wherein the reward signal is a scalar value calculated based on a pre-set multi-objective utility function after the placement action is executed, and the formula of the multi-objective utility function is:

[0033] U=w s *f s +w v *f v +w e *f e , wherein w s , w v , w e represent corresponding weight coefficients, f s represents a stability evaluation value, f v represents a space utilization evaluation value, and f e represents an execution efficiency evaluation value.

[0034] Preferably, the process of transforming the current environment state to obtain the state feature vector comprises:

[0035] A three-dimensional convolutional neural network is used to encode the three-dimensional container model to obtain an environment feature vector;

[0036] A multi-layer perceptron is used to encode the size and pose information to obtain a cargo feature vector.

[0037] The environment feature vector and the cargo feature vector are spliced to obtain the state feature vector.

[0038] Preferably, after the placement operation is completed, the loaded space is rescanned to obtain a current three-dimensional model, an execution error is calculated, and the execution error is used to update the three-dimensional container model and network parameters of the deep reinforcement decision network, including:

[0039] The loaded space is rescanned by the multi-modal sensor to obtain three-dimensional point cloud data after the placement operation, and the current three-dimensional model corresponding to the current real scene is generated;

[0040] The current three-dimensional model is compared with an expected three-dimensional model simulated in the three-dimensional container model before the placement operation to obtain the execution error;

[0041] The voxel data of the three-dimensional container model is corrected using the execution error to obtain an updated three-dimensional container model;

[0042] The execution error is used as a component of a reward signal and input into a training process of the deep reinforcement decision network to update network parameters of a policy network and a value network.

[0043] To achieve the above object, the application further provides a cargo loading device based on deep reinforcement learning, which comprises:

[0044] An environment perception unit is configured to scan the loaded space by a multi-modal sensor to obtain three-dimensional point cloud data, and construct a three-dimensional container model corresponding to the loaded space based on the three-dimensional point cloud data;

[0045] A cargo recognition unit is configured to recognize and estimate a 6D pose of a cargo to be loaded to obtain size and attitude information of the cargo to be loaded;

[0046] A pose generation unit is configured to generate a plurality of candidate placement poses according to the three-dimensional container model and the size and attitude information;

[0047] An intelligent decision unit is configured to determine an optimal placement pose from the candidate placement poses by using a pre-constructed deep reinforcement decision network, and perform a placement operation of the cargo to be loaded in the optimal placement pose by controlling a mechanical arm;

[0048] A closed-loop feedback unit is configured to rescan the loaded space to obtain a current three-dimensional model after the placement operation is completed, calculate an execution error, and use the execution error to update the three-dimensional container model and network parameters of the deep reinforcement decision network.

[0049] To achieve the above object, the application further provides a cargo loading device based on deep reinforcement learning, comprising a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of the cargo loading method based on deep reinforcement learning as described in the above embodiment.

[0050] To achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the cargo loading method based on deep reinforcement learning as described in the above embodiment.

[0051] To achieve the above object, the application further provides a computer program product, comprising computer programs / instructions, wherein the computer programs / instructions are executed by a processor to implement the steps of the cargo loading method based on deep reinforcement learning as described in the above embodiment.

[0052] Advantages:

[0053] The above scheme constructs a three-dimensional container model through multi-modal sensor scanning, identifies the 6D pose of the cargo, generates candidate placement poses, selects the optimal pose using a deep reinforcement decision network and controls the robot arm to execute, and finally updates the model and network parameters through closed-loop feedback. It constructs an intelligent closed loop of "perception-decision-execution-feedback", and realizes highly adaptive intelligent loading capability in unstructured environment. Through the introduction of deep reinforcement learning, the system can adaptively handle unknown cargo and dynamically changing environment, significantly improving the intelligent level and robustness of the loading operation. At the same time, the closed-loop feedback mechanism ensures that the system can continuously learn and optimize from the actual execution results, effectively overcoming the error accumulation problem of traditional open-loop systems, and improving the stability and efficiency of long-term operation.

[0054] The voxel-based truncated signed distance function (TSDF) fusion algorithm is adopted, which divides the loading space into a voxel grid and fuses multiple frames of depth image data, and can construct and update a high-precision three-dimensional environment model in real time. The weighted average fusion method of TSDF algorithm effectively suppresses sensor noise and measurement error, and provides a stable and reliable environment representation; the real-time updating mechanism enables the model to dynamically adapt to the changes of the internal state of the container during the loading process, providing an accurate environment basis for subsequent placement planning. Thus, the perception ability of the system to complex unstructured environment is improved, and reliable spatial information support is provided for intelligent decision-making.

[0055] By combining instance segmentation of RGB images with pose estimation of point cloud data, accurate identification and localization of non-standard goods were achieved. Multimodal data fusion fully utilizes color texture and geometric structure information, enhancing the generalization ability to identify unknown goods. The use of deep learning networks eliminates the need for predefined CAD models of objects, enabling the processing of goods with diverse shapes and significantly improving the system's generalization ability. Precise 6D pose estimation provides crucial posture information for subsequent grasping and placement operations, ensuring the accuracy and reliability of the operations and effectively overcoming the limitations of traditional methods in processing non-standard goods.

[0056] By identifying idle regions and supporting surfaces through geometric analysis, and combining surface sampling with various rotational attitudes to generate a set of candidate poses, rigorous collision detection and validity verification ensure the comprehensiveness and feasibility of the candidate poses, avoiding the omission of optimal solutions. Collision detection prevents interference with the environment and placed goods, while validity verification ensures the robot's accessibility and operational safety. Through a systematic pose generation and verification mechanism, high-quality and diverse input choices are provided for the deep reinforcement decision network, significantly improving the optimization space of the final placement strategy through increased execution efficiency and success rate.

[0057] Intelligent placement decisions are achieved through a collaborative architecture of policy and value networks, combined with reward signals driven by multi-objective utility functions. The multi-objective utility function comprehensively considers multiple key indicators such as stability, space utilization, and execution efficiency, enabling the system to balance conflicts between different objectives and generate a globally optimal placement strategy. The encoding method of state feature vectors effectively integrates environmental models and cargo information, providing rich contextual information for network decision-making. The deep reinforcement learning training mechanism allows the network to autonomously learn and optimize strategies from a large number of trials and errors, adapting to various complex scenarios and significantly improving the intelligence and generalization ability of decision-making.

[0058] By rescanning to obtain the actual 3D model and comparing it with the expected model to calculate the execution error, the environmental model is updated and the decision network optimized. This allows the system to detect deviations between the actual execution results and expectations, and promptly correct inaccuracies in the environmental model. Simultaneously, the execution error is used as part of the reward signal for network parameter updates, enabling the system to learn from both successes and failures and continuously improve its decision-making strategy. This effectively solves the error accumulation problem of traditional open-loop systems, improves the system's adaptability and robustness in long-term operation, and achieves true self-optimization and continuous improvement. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart illustrating a cargo loading method based on deep reinforcement learning, as provided in an embodiment of the present invention.

[0061] Figure 2 This is a schematic diagram of an autonomous cargo loading process based on multimodal and deep reinforcement learning, provided as an embodiment of the present invention.

[0062] Figure 3 This is a schematic diagram of the decision-making process of a deep reinforcement decision network provided in an embodiment of the present invention.

[0063] Figure 4 This is a schematic diagram of a cargo loading device based on deep reinforcement learning, provided as an embodiment of the present invention.

[0064] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] The present invention will be described in detail below with reference to the embodiments.

[0067] Reference Figure 1 The diagram shown is a flowchart of a cargo loading method based on deep reinforcement learning provided in an embodiment of the present invention.

[0068] In this embodiment, the method includes:

[0069] S11, the loading space is scanned by a multimodal sensor to obtain three-dimensional point cloud data, and a three-dimensional container model corresponding to the loading space is constructed based on the three-dimensional point cloud data.

[0070] Further, in step S11, the scanning of the loading space using a multimodal sensor to obtain three-dimensional point cloud data, and the construction of a three-dimensional container model corresponding to the loading space based on the three-dimensional point cloud data, includes:

[0071] S11-1 acquires depth images of the loading space using an RGB-D camera and converts the depth images into 3D point cloud data;

[0072] S11-2, preprocessing the 3D point cloud data, including filtering and noise reduction, and then using a voxel-based truncated symbolic distance function fusion algorithm to construct or update the preprocessed 3D point cloud data to obtain the 3D container model; wherein,

[0073] S11-2-1, the loading space is divided into multiple three-dimensional voxel grids, the truncation symbol distance value from each voxel to the nearest object surface is calculated, and the truncation symbol distance values ​​of multiple consecutive depth images are fused into the voxel by weighted averaging.

[0074] S11-2-2, update the truncation symbol distance value and corresponding weight of each voxel in real time to obtain the three-dimensional container model.

[0075] Further, in step S11-2-1, the calculation of the truncation symbol distance value from each voxel to the nearest object surface, and the fusion of the truncation symbol distance values ​​of multiple consecutive depth images into the voxel using a weighted average method, includes:

[0076] Project each voxel onto the current image frame to obtain the corresponding pixel coordinates and projection depth value;

[0077] Based on the projected depth value and the measured depth value of the corresponding pixel coordinates in the depth image, the symbolic distance function value of the current image frame is calculated, expressed by the formula: In the formula, sdf k (x) represents the sign distance function value, z represents the projection depth value, and d k (u) represents the measured depth value;

[0078] The symbolic distance function value is truncated within a preset threshold range to obtain the truncated symbolic distance value corresponding to the current image frame, expressed by the formula: In the formula, tsdf k (x) represents the truncation symbol distance value of the current image frame, and μ represents the preset threshold;

[0079] For each voxel, the updated truncated symbolic distance and weight are calculated by weighted averaging based on the truncation symbolic distance and weight of the previous image frame and the truncation symbolic distance and weight of the current image frame. The formula is as follows: , In the formula, and These represent the truncation symbol distance and weight of voxel x in the previous image frame, respectively. k (x) and These represent the truncation symbol distance value and weight of the current image frame, respectively.

[0080] Reference Figure 2 As shown. In this embodiment, the application scenarios of this method include smart logistics (such as automatic loading of parcels on trucks, vans, and containers), aviation hubs (such as automated loading and unloading of aircraft baggage compartments), e-commerce warehouses (such as automated packaging and boxing of order parcels), and automated moving services (such as loading irregular items such as furniture and boxes). The system uses a high-resolution, high-precision industrial-grade RGB-D camera (such as ToF or structured light principle) as the core sensor, which is fixedly installed in a position that can completely cover the target loading space (such as above the baggage compartment entrance). The camera scans the loading space (such as the baggage compartment) and simultaneously acquires color (RGB) images and depth images of the scene. Using the intrinsic and extrinsic parameters obtained from camera calibration, the depth image is converted into three-dimensional point cloud data representing the geometric information of the loading space surface. To improve data quality, the original three-dimensional point cloud data is preprocessed, including filtering and noise reduction. After obtaining high-quality point cloud data through preprocessing, the RANSAC (Random Sample Consensus) algorithm is used to segment the main planes in the point cloud data, thereby identifying the bottom and side boundaries of the container space. Based on this, a voxel-based truncated symbolic distance function (TSDF) fusion algorithm is used to construct a 3D container model. This 3D container model, as a core component of the digital twin model, not only reflects the precise state of the current loading space in real time, but also allows for further precision tuning of the 3D container model using an RGB-D camera, providing a high-fidelity environmental context for subsequent intelligent decision-making. Specifically, the loading space is divided into a uniform 3D voxel grid, with each voxel x storing a truncated symbolic distance value D(x) and a weight W(x). For a given camera pose T... x For each acquired depth map frame, the system updates it in the following manner:

[0081] For each voxel x, project it onto the current camera frame to obtain its coordinates u and projection depth value z on the image. Then, compare the projection depth value z with the measured depth value d of the corresponding pixel coordinates in the depth map. k (u) is compared, and the symbolic distance function value of the current frame is calculated, i.e. To suppress the influence of noise, the symbol distance function value is truncated within a preset threshold ±μ range to obtain the TSDF value of the current frame, i.e. Subsequently, the new TSDF values ​​are incorporated into the model using a weighted average method, and the update formula is as follows:

[0082] , In the formula, and These represent the truncation symbol distance and weight of voxel x in the previous image frame, respectively. k (x) and These represent the truncation symbol distance value and weight of the current image frame, respectively. The 3D container model constructed and updated in the above manner will be synchronized to the digital twin model in real time, enabling the continuous depth image sequence to be robustly fused into a globally consistent 3D model in real time, and dynamically updated based on new sensor data after each operation.

[0083] S12, identify and estimate the 6D pose of the cargo to be loaded to obtain the size and pose information of the cargo to be loaded.

[0084] Furthermore, in step S12, the identification and 6D pose estimation of the cargo to be loaded to obtain the corresponding size and pose information of the cargo includes:

[0085] S12-1, Obtain an RGB image of the goods to be loaded, and extract the pixel-level mask of the goods to be loaded from the RGB image using a preset instance segmentation network;

[0086] S12-2, Based on the pixel-level mask and the point cloud data corresponding to the pixel-level mask, the three-dimensional bounding box size of the cargo to be loaded and the 6D pose relative to the camera coordinate system, including translation and rotation information, are regressed through a preset 6D pose estimation network to obtain the size and pose information.

[0087] In this embodiment, when goods to be loaded (such as luggage) enter the gripping station of the robotic arm's working area via a conveyor belt, an RGB-D camera simultaneously acquires RGB images and depth information of the goods. The RGB images are then input into a pre-trained instance segmentation network (such as using a YOLO-seg architecture), which can accurately identify luggage-like objects in complex backgrounds and output a pixel-level mask for each detected goods instance in the image. This mask precisely defines the outline of the goods in the two-dimensional image, thereby separating it from the background and other possible distractions.

[0088] Based on the obtained pixel-level mask, it is mapped to the corresponding point cloud data of the depth image, thereby extracting the 3D point cloud set belonging to the luggage. The 3D point cloud set and the corresponding pixel-level mask of the RGB image are input into a preset 6D pose estimation network. This network adopts a hybrid architecture based on PointNet++ and DenseFusion, which can process geometric information and RGB texture information simultaneously. This network does not rely on the CAD model of the cargo, but directly regresses the complete pose information of the cargo in the camera coordinate system through deep learning. Its output includes: the 3D bounding box size (x, y, z) defining its position, and the 6D pose of the cargo relative to the camera coordinate system, which includes 3 translational degrees of freedom (i.e., the 3D coordinates [x, y, z] of the cargo's center point) and 3 rotational degrees of freedom (usually represented in quaternion form, defining the cargo's roll, pitch, and yaw angles in space). The architecture of the 6D pose estimation network is based on architectures such as PointNet++ and DenseFusion. Its training process uses a luggage-specific dataset for training and fine-tuning to achieve robust estimation of luggage of various shapes, sizes and appearances.

[0089] S13, Based on the three-dimensional container model and the size and orientation information, generate multiple candidate placement poses.

[0090] Furthermore, in step S13, generating multiple candidate placement poses based on the three-dimensional container model and the size and pose information includes:

[0091] S13-1, Perform geometric analysis on the three-dimensional container model to identify all available free areas, identify supporting surfaces within the free areas, and generate discrete three-dimensional coordinate points on the supporting surfaces by employing a surface sampling strategy.

[0092] S13-2, For each of the three-dimensional coordinate points, a set of candidate placement points including multiple posture-position combinations is generated by combining the size and posture information with the six basic rotation postures of the cargo to be loaded.

[0093] S13-3, perform a verification including collision detection and validity verification on each of the posture-position combinations, and determine all the posture-position combinations that pass the verification as valid candidate placement poses.

[0094] Furthermore, the collision detection involves performing a geometric interference check between the 3D model of the cargo to be loaded under the posture-position combination and the container boundary and the cargo already placed in the 3D container model.

[0095] The validity verification includes workspace verification and stability verification. Workspace verification verifies whether the posture-position combination is within the reachable workspace of the robotic arm. Stability verification verifies whether the projection of the center of gravity of the cargo to be loaded onto the support surface falls within the stable area.

[0096] In this embodiment, a geometric analysis is performed on the 3D container model. By identifying the top surface of already placed luggage, as well as the container's bottom and sidewalls, all available 3D free areas (i.e., negative spaces) are precisely segmented. These areas are unoccupied spaces that may accommodate new cargo. Within these free areas, horizontal or near-horizontal support surfaces (e.g., the flat bottom of the container or the stable top surface of already placed cargo) are further identified for placing new luggage. A surface sampling strategy is used to generate a series of discrete 3D coordinate points on these support surfaces through uniform mesh division as potential placement locations. These points represent potential support locations where new cargo may be placed, ensuring the comprehensiveness and uniformity of the sampling coverage.

[0097] For each 3D coordinate point, combined with the 3D dimensions and orientation information of the goods to be placed, the system iterates through all six basic rotational orientations (i.e., the stable placement orientations when the length, width, and height of the goods are facing downwards). For each rotational orientation, the system aligns the bottom center or main support surface of the goods with the 3D coordinate point, thereby generating a posture-position combination, i.e., a candidate 6D pose (including 3D position and 3D rotation). This method can generate a set of candidate placement points containing multiple posture-position combinations, ensuring that all possible placement methods are taken into account to maximize the potential of space utilization.

[0098] Each pose-position combination in the candidate placement point set is validated to filter out valid candidate placement poses. Validation includes collision detection: checking whether the bounding box of the goods to be placed interferes with or overlaps with the boundary of the 3D container model or the surface of other placed goods in that pose; and validity verification: assessing the feasibility of the placement pose, such as checking whether the pose is within the reachable workspace of the robotic arm and whether it meets basic stability requirements after placement (e.g., the support surface is large enough). Only pose combinations that pass both collision detection and validity verification are retained and ultimately determined as valid candidate placement poses. These valid candidate placement poses constitute a high-quality feasible solution set, which is then fed into a subsequent deep reinforcement decision network for optimal selection.

[0099] S14, a pre-constructed deep reinforcement decision network is used to determine the optimal placement pose among the candidate placement poses, and the robotic arm is controlled to perform the placement operation of the cargo to be loaded in the optimal placement pose.

[0100] Furthermore, in step S14, the construction process of the deep reinforcement decision network includes:

[0101] Construct a network architecture that includes a policy network and a value network, wherein,

[0102] The policy network receives a state feature vector as input and outputs a placement action. The state feature vector is obtained by transforming the current environmental state, which includes the three-dimensional container model and the size and pose information.

[0103] The value network performs a value assessment based on the current environmental state and the placement action, outputs a corresponding state-action value assessment, and uses the state-action value assessment to guide the optimization of the policy network.

[0104] Based on the aforementioned network architecture, the deep reinforcement learning network is obtained through training using a deep reinforcement learning algorithm and based on the reward signal. The reward signal is a scalar value calculated after the placement action is performed based on a preset multi-objective utility function, the formula of which is:

[0105] U=w s *f s +w v *f v +w e *f e In the formula, w s w v w e f represents the corresponding weighting coefficient. s f represents the stability assessment value. v f represents the space utilization assessment value. e This represents the performance evaluation value.

[0106] Furthermore, the stability assessment value is calculated by taking the total potential energy of all placed goods after placement; the lower the potential energy, the more stable the goods.

[0107] The space utilization assessment value is calculated by weighting the volume occupied by new cargo with the volume of the largest inscribed cuboid in the remaining space, in order to encourage the generation of large, usable, continuous remaining space, so as to reserve more usable space for subsequent cargo.

[0108] The execution efficiency evaluation value is obtained by verifying the accessibility of the target placement pose, and when the pose is accessible, it is taken as the reciprocal of the motion path cost.

[0109] Furthermore, the state feature vector, through the process of transforming the current environmental state, includes:

[0110] The three-dimensional container model is encoded using a three-dimensional convolutional neural network to obtain an environmental feature vector;

[0111] The size and orientation information is encoded using a multilayer perceptron to obtain a cargo feature vector;

[0112] The environmental feature vector and the cargo feature vector are concatenated to obtain the state feature vector.

[0113] In this embodiment, determining the optimal placement pose relies on a deep reinforcement learning (DRL) decision network pre-trained using a deep reinforcement learning (DRL) algorithm. This network is built on an actor-critic architecture, where the actor network (policy network) is responsible for generating placement actions, and the critic network (value network) is responsible for evaluating the value of the actions to guide optimization. It models the placement decision problem as a Markov decision process (MDP), referring to... Figure 3 As shown. Here, State (S) is defined as the three-dimensional container model V including the current cargo hold. grid (such as the TSDF voxel grid in the baggage compartment) and the physical property characteristics of the cargo to be placed. feat (Including size and orientation information); Action (A) is a six-dimensional vector that defines the target placement pose of the goods to be placed; Reward (R) is a scalar value calculated by the multi-objective utility function U after the placement action is executed, and its calculation formula is R=U=w s *f s +w v *f v +w e *f e In the formula, It is a weighting coefficient that can be dynamically adjusted according to different loading strategies (such as "stability", "speed" or "fullness").

[0114] Among them, f s This is a stability assessment value, representing the physical stability of the entire system after quantification. It is calculated based on either potential energy or the supporting polygons; where, based on potential energy, it calculates the total potential energy E of all placed goods (N in total). p The lower the potential energy, the more stable the system. This can be expressed by the formula: In the formula, m i Let g be the mass of cargo i, g be the acceleration due to gravity, and h be the mass of cargo i. i Its centroid height; based on the supporting polygon, it is for each non-bottom layer cargo. Calculate the projection P of its centroid onto the supporting plane. i With support area S i Convenient minimum distance d i (Stability margin), its stability can be determined by the least stable item, i.e. .

[0115] f v This is a space utilization assessment value, representing the degree to which space is effectively utilized and penalizing placement behaviors that lead to space fragmentation; it is calculated by determining the volume V occupied by new goods. new The volume V of the largest cuboid that can be accommodated in the remaining space hole,max The weighted sum, i.e. f v = V new + α * V hole,max To quantify and encourage the generation of large-capacity continuous spaces, where α is a weighting coefficient. This means that it is obtained by searching for the largest inscribed cuboid in the remaining point cloud data.

[0116] f e The efficiency evaluation value represents the cost and feasibility of the robot performing the placement action; it is achieved by verifying the target pose. Is it within the robot's reachable workspace? If reachable, then plan a collision-free path. And calculate its overall cost, such as path length. and execution time The efficiency function is inversely proportional to cost, and its formula is as follows: In the formula, The cost weight is used; if the path is unreachable or there is a collision, then... .

[0117] To input the current environmental state, including the model and cargo attributes, into the network, it needs to be converted into a fixed-size tensor, i.e., the state S is represented as... In the state representation stage, a three-dimensional convolutional neural network (3D-CNN) is used as the world model encoder E. w Voxelized 3D container model V grid (e.g., TSDF raster) is encoded as an environmental feature vector h w =E w (V grid Simultaneously, a multilayer perceptron (MLP) is used as the object encoder E. i The physical attributes of the goods, such as size and weight. feat Encoded as cargo feature vector h i =E i (I feat These two feature vectors are concatenated to form the complete state feature vector h=[h w ,h i [ ] serves as the input for a deeply reinforced decision network.

[0118] The policy network (Actor Network) π is a multilayer perceptron that receives the state feature vector h and directly outputs the deterministic placement action A (6D pose), i.e., A = π(S) = MLP actor (h). The Critic Network (Q) is another multilayer perceptron that receives the state feature vector h and the placement action A output by the policy network, and outputs the Q-value of the state-action pair, i.e., Q(S,A)=MLP. critic ([h,A]) is used to evaluate the value of the placement action and guide the optimization and updating of the policy network.

[0119] In actual deployment, the system encodes the current environmental state and inputs it into a pre-trained policy network. The policy network directly outputs a six-dimensional placement action (i.e., the optimal placement pose) through forward computation. This action comprehensively balances stability, space utilization, and execution efficiency. After determining the optimal placement pose, the system uses a motion planning module (such as employing the Fast Extended Random Tree algorithm, RRT) to generate a collision-free, smooth trajectory for the robotic arm and controls the robotic arm to perform grasping and placement operations, ensuring smooth and accurate movements and achieving intelligent loading decisions. The entire process realizes an efficient closed loop from intelligent decision-making to physical execution, improving the adaptability and robustness of the loading system.

[0120] S15, after the placement operation is completed, the loading space is rescanned to obtain the current three-dimensional model, and the execution error is calculated. The execution error is used to update the three-dimensional container model and the network parameters of the deep reinforcement decision network.

[0121] Furthermore, in step S15, after the placement operation is completed, the loading space is rescanned to obtain the current 3D model, and the execution error is calculated. The execution error is then used to update the 3D container model and the network parameters of the deep reinforcement decision network, including:

[0122] S15-1, The loading space is re-scanned using the multimodal sensor to obtain the three-dimensional point cloud data after the placement operation, and the current three-dimensional model corresponding to the current real scene is generated;

[0123] S15-2, compare the current three-dimensional model with the expected three-dimensional model simulated and generated in the three-dimensional container model before the placement operation to obtain the execution error;

[0124] S15-3, use the execution error to correct the voxel data of the three-dimensional container model to obtain the updated three-dimensional container model;

[0125] S15-4, The execution error is used as a component of the reward signal and input into the training process of the deep reinforcement decision network to update the network parameters of the policy network and the value network.

[0126] In this embodiment, after the robotic arm completes the cargo placement operation, it rescans the loading space using a multimodal sensor (such as an RGB-D camera) to acquire depth image data after the placement operation and converts it into a 3D point cloud representing the current real scene. Similar to the initial scan, the system performs preprocessing operations such as filtering and noise reduction on the acquired point cloud data, and then uses a voxel-based truncated symbolic distance function (TSDF) fusion algorithm to generate a current 3D model that accurately reflects the actual loading state.

[0127] The current 3D model is precisely compared with the expected 3D model simulated in the corresponding 3D container model of the digital twin model before the placement operation. For example, the execution error is obtained by calculating the voxel differences between the two models at corresponding spatial positions (calculating the difference between the actual pose (including 3D coordinates and rotation angles) of the cargo to be loaded and the expected pose). This error is specifically quantified as translation error and rotation error, comprehensively reflecting the degree of deviation between the actual execution effect and the expected plan. The calculated execution error is used to correct the voxel data of the 3D container model. Specifically, by adjusting the TSDF values ​​and weights of the corresponding voxels in the 3D container model, the model is corrected in real time to more accurately reflect the actual loading state, providing a reliable environmental basis for subsequent placement planning.

[0128] Simultaneously, execution errors are transformed into components of the reinforcement learning reward signal. Positive errors (actual results exceeding expectations) generate positive rewards, reinforcing the behavioral strategy that leads to success; negative errors (actual results falling short of expectations) generate negative rewards, prompting the network to adjust its strategy to avoid similar errors. This reward signal is input into the training process of the deep reinforcement decision network to update the parameters of the policy and value networks. Through this continuous learning mechanism, the system can learn from each practical operation, continuously optimize its decision-making strategy, and gradually improve its performance and robustness in complex unstructured environments, achieving the system's self-evolutionary capability.

[0129] Reference Figure 4 The diagram shown is a structural schematic of a cargo loading device based on deep reinforcement learning according to an embodiment of the present invention.

[0130] In this embodiment, the device 20 includes:

[0131] The environmental perception unit 21 is used to scan the loading space through a multimodal sensor to obtain three-dimensional point cloud data, and to construct a three-dimensional container model corresponding to the loading space based on the three-dimensional point cloud data.

[0132] The cargo identification unit 22 is used to identify the cargo to be loaded and estimate its 6D pose to obtain the size and pose information of the cargo to be loaded.

[0133] Pose generation unit 23 is used to generate multiple candidate placement poses based on the three-dimensional container model and the size and pose information;

[0134] The intelligent decision-making unit 24 is used to determine the optimal placement pose among the candidate placement poses using a pre-built deep reinforcement decision network, and to control the robotic arm to perform the placement operation of the cargo to be loaded in the optimal placement pose.

[0135] The closed-loop feedback unit 25 is used to rescan the loading space to obtain the current three-dimensional model after the placement operation is completed, calculate the execution error, and use the execution error to update the three-dimensional container model and the network parameters of the deep reinforcement decision network.

[0136] Each unit module of the device 20 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0137] This invention also provides a cargo loading device based on deep reinforcement learning, which includes the cargo loading apparatus based on deep reinforcement learning as described above. The cargo loading apparatus based on deep reinforcement learning can employ... Figure 4 The structure of the embodiment, correspondingly, can be executed Figure 1 The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0138] The device includes: a mobile phone, digital camera, or tablet computer, or other device with a camera function; or a device with an image processing function; or a device with an image display function. The device may include components such as a memory, processor, input unit, display unit, and power supply.

[0139] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0140] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0141] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0142] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The illustrated cargo loading method is based on deep reinforcement learning. The computer-readable storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0143] This invention also provides a computer program product, including a computer program / instructions, which are loaded and executed by a processor to implement... Figure 1 This illustrates a cargo loading method based on deep reinforcement learning.

[0144] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0145] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0146] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A cargo loading method based on deep reinforcement learning, characterized in that, The method includes: The loading space is scanned by a multimodal sensor to obtain three-dimensional point cloud data. Based on the three-dimensional point cloud data, a three-dimensional container model corresponding to the loading space is constructed. The cargo to be loaded is identified and its 6D pose is estimated to obtain the size and orientation information of the cargo to be loaded. Based on the 3D container model and the size and orientation information, multiple candidate placement poses are generated, including: Geometric analysis is performed on the three-dimensional container model to identify all available free areas. Support surfaces are identified within the free areas, and discrete three-dimensional coordinate points are generated on the support surfaces using a surface sampling strategy. For each of the three-dimensional coordinate points, a set of candidate placement points is generated by combining the size and orientation information with the six basic rotational orientations of the cargo to be loaded, including multiple orientation-position combinations. Each of the pose-position combinations is validated, including collision detection and validity verification, and all pose-position combinations that pass the validation are determined as valid candidate placement poses; A pre-constructed deep reinforcement decision network is used to determine the optimal placement pose among the candidate placement poses, and the robotic arm is controlled to perform the placement operation of the cargo to be loaded in the optimal placement pose. After the placement operation is completed, the loading space is rescanned to obtain the current 3D model, and the execution error is calculated. The execution error is then used to update the 3D container model and the network parameters of the deep reinforcement decision network.

2. The cargo loading method based on deep reinforcement learning according to claim 1, characterized in that, The step of scanning the loading space using a multimodal sensor to obtain three-dimensional point cloud data, and constructing a three-dimensional container model corresponding to the loading space based on the three-dimensional point cloud data, includes: The depth image of the loading space is acquired using an RGB-D camera, and the depth image is converted into 3D point cloud data. The 3D point cloud data undergoes preprocessing including filtering and noise reduction. A voxel-based truncated symbolic distance function fusion algorithm is then used to construct or update the preprocessed 3D point cloud data, resulting in the 3D container model. The loading space is divided into multiple three-dimensional voxel grids. The truncation symbol distance value from each voxel to the nearest object surface is calculated. The truncation symbol distance values ​​of multiple consecutive depth images are fused into the voxel by weighted averaging. The truncation symbol distance value and corresponding weight of each voxel are updated in real time to obtain the three-dimensional container model.

3. The cargo loading method based on deep reinforcement learning according to claim 2, characterized in that, The calculation of the truncation symbol distance value from each voxel to the nearest object surface, and the fusion of the truncation symbol distance values ​​from multiple consecutive depth images into the voxel using a weighted average method, includes: Project each voxel onto the current image frame to obtain the corresponding pixel coordinates and projection depth value; Based on the projected depth value and the measured depth value of the corresponding pixel coordinates in the depth image, the symbolic distance function value of the current image frame is calculated, expressed by the formula: In the formula, sdf k (x) represents the sign distance function value, z represents the projection depth value, and d k (u) represents the measured depth value; The symbolic distance function value is truncated within a preset threshold range to obtain the truncated symbolic distance value corresponding to the current image frame, expressed by the formula: In the formula, tsdf k (x) represents the truncation symbol distance value of the current image frame, and μ represents the preset threshold; For each voxel, the updated truncated symbolic distance and weight are calculated by weighted averaging based on the truncation symbolic distance and weight of the previous image frame and the truncation symbolic distance and weight of the current image frame. The formula is as follows: , In the formula, and These represent the truncation symbol distance and weight of voxel x in the previous image frame, respectively. k (x) and These represent the truncation symbol distance value and weight of the current image frame, respectively.

4. The cargo loading method based on deep reinforcement learning according to claim 1, characterized in that, The process of identifying and estimating the 6D pose of the cargo to be loaded, to obtain the corresponding size and pose information of the cargo, includes: Obtain an RGB image of the goods to be loaded, and use a preset instance segmentation network to extract the pixel-level mask of the goods to be loaded from the RGB image; Based on the pixel-level mask and the point cloud data corresponding to the pixel-level mask, the three-dimensional bounding box size of the cargo to be loaded and the 6D pose relative to the camera coordinate system, including translation and rotation information, are regressed through a preset 6D pose estimation network to obtain the size and pose information.

5. The cargo loading method based on deep reinforcement learning according to claim 1, characterized in that, The construction process of the deep reinforcement decision network includes: Construct a network architecture that includes a policy network and a value network, wherein, The policy network receives a state feature vector as input and outputs a placement action. The state feature vector is obtained by transforming the current environmental state, which includes the three-dimensional container model and the size and pose information. The value network performs a value assessment based on the current environmental state and the placement action, outputs a corresponding state-action value assessment, and uses the state-action value assessment to guide the optimization of the policy network. Based on the aforementioned network architecture, the deep reinforcement learning network is obtained through training using a deep reinforcement learning algorithm and based on the reward signal. The reward signal is a scalar value calculated after the placement action is performed based on a preset multi-objective utility function, the formula of which is: U=w s *f s +w v *f v +w e *f e In the formula, w s w v w e f represents the corresponding weighting coefficient. s f represents the stability assessment value. v f represents the space utilization assessment value. e This represents the performance evaluation value.

6. The cargo loading method based on deep reinforcement learning according to claim 5, characterized in that, The state feature vector is obtained through a process of transforming the current environmental state, including: The three-dimensional container model is encoded using a three-dimensional convolutional neural network to obtain an environmental feature vector; The size and orientation information is encoded using a multilayer perceptron to obtain a cargo feature vector; The environmental feature vector and the cargo feature vector are concatenated to obtain the state feature vector.

7. The cargo loading method based on deep reinforcement learning according to claim 1, characterized in that, After the placement operation is completed, the loading space is rescanned to obtain the current 3D model, and the execution error is calculated. The execution error is then used to update the 3D container model and the network parameters of the deep reinforcement decision network, including: The loading space is rescanned using the multimodal sensor to obtain 3D point cloud data after the placement operation, and the current 3D model corresponding to the current real scene is generated. The execution error is obtained by comparing the current 3D model with the expected 3D model simulated and generated in the 3D container model before the placement operation. The voxel data of the three-dimensional container model are corrected using the execution error to obtain the updated three-dimensional container model; The execution error is used as a component of the reward signal and input into the training process of the deep reinforcement decision network to update the network parameters of the policy network and the value network.

8. A cargo loading device based on deep reinforcement learning, characterized in that, The device includes: An environmental perception unit is used to scan the loading space using a multimodal sensor to obtain three-dimensional point cloud data, and to construct a three-dimensional container model corresponding to the loading space based on the three-dimensional point cloud data. The cargo identification unit is used to identify the cargo to be loaded and estimate its 6D pose to obtain the size and pose information of the cargo to be loaded. The pose generation unit is configured to generate multiple candidate placement poses based on the 3D container model and the size and pose information; the pose generation unit is further configured to: Geometric analysis is performed on the three-dimensional container model to identify all available free areas. Support surfaces are identified within the free areas, and discrete three-dimensional coordinate points are generated on the support surfaces using a surface sampling strategy. For each of the three-dimensional coordinate points, a set of candidate placement points is generated by combining the size and orientation information with the six basic rotational orientations of the cargo to be loaded, including multiple orientation-position combinations. Each of the pose-position combinations is validated, including collision detection and validity verification, and all pose-position combinations that pass the validation are determined as valid candidate placement poses; The intelligent decision-making unit is used to determine the optimal placement pose among the candidate placement poses using a pre-built deep reinforcement decision network, and to control the robotic arm to perform the placement operation of the cargo to be loaded in the optimal placement pose. The closed-loop feedback unit is used to rescan the loading space to obtain the current 3D model after the placement operation is completed, calculate the execution error, and use the execution error to update the 3D container model and the network parameters of the deep reinforcement decision network.

9. A cargo loading device based on deep reinforcement learning, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the steps of a cargo loading method based on deep reinforcement learning as described in any one of claims 1 to 7.

Images