A visual guidance robot disordered grasping planning method based on deep reinforcement learning

Abstract

This invention, entitled "A Visual Guidance Robot Unordered Grasping Planning Method Based on Deep Reinforcement Learning," belongs to the field of robot grasping planning technology. Addressing the problems of existing methods relying on local optimum scoring in complex scenarios, which easily leads to frequent misgrasping, and the high dimensionality of end-to-end learning exploration, this invention proposes: acquiring sensor data to extract foreground point clouds, generating a set of grasping candidates filtered by physical feasibility; constructing reinforcement learning state vectors and inputting them into a policy network, with discrete branches outputting target candidate indices and continuous branches outputting pose and execution parameter residuals; obtaining refined pose parameters after action calculation, and then executing the grasping operation. This invention compresses the decision space to the candidate neighborhood, balancing exploration efficiency and long-term continuous box clearing benefits, and is mainly used for grasping unordered scattered objects in industrial warehousing and logistics sorting.

Description

Technical Field

[0001] This invention relates to the field of robot grasping planning technology, and in particular to a visually guided robot unordered grasping planning method based on deep reinforcement learning. Background Technology

[0002] In industrial warehousing and logistics sorting, objects in bins or containers are often piled up in a disordered and scattered state. Robots need to rely on visual perception systems to identify graspable targets in the scene and plan safe and efficient grasping actions. In recent years, grasping candidate generation technology based on color-depth vision has made significant progress. Related methods can predict a large number of six-DOF grasping pose candidates from scene point clouds, and continuously improve the accuracy and candidate coverage of single-frame detection driven by large-scale labeled datasets. Some methods further improve the efficiency and quality of candidate generation by introducing grasping priors to screen graspable regions in the point cloud in the early stage. Based on this, existing systems have reported high single-grab success rates and picking speeds in bin emptying experiments in parallel gripper scenarios.

[0003] However, the optimization objectives of the aforementioned methods are mostly focused on single-shot grasp detection or single-step grasp scoring, i.e., finding the locally optimal grasp pose under the current observation and executing it immediately. In actual industrial operations, there are complex occlusion, support, and boundary interference relationships between objects within the bin. The candidate with the highest score in the current frame does not necessarily correspond to the globally optimal next action. If the grasping system makes decisions based solely on the instantaneous score without explicitly considering the execution time of the action, collision risk, retry cost after failure, and impact on subsequent grasping efficiency, frequent high-scoring false grasps, collision backoffs, and repeated selection of failed candidates can easily occur in complex scenarios with high occlusion and high stacking, leading to a decrease in the overall throughput of continuous box clearing operations. Meanwhile, although deep reinforcement learning has shown potential in solving long-term decision-making problems in the field of robot operation, if the policy network is directly made to search for and grasp trajectories in the complete six-degree-of-freedom continuous pose space from the original visual observation, it faces difficulties such as excessive exploration dimensionality, huge training sample requirements, slow convergence speed, and significant distribution gap between the simulation environment and the real environment. This makes it difficult for pure end-to-end deep reinforcement learning solutions to be stably implemented in real industrial environments.

[0004] Therefore, how to balance grasping success rate, motion safety, execution efficiency, and continuous box clearing benefits with limited training samples and sensor noise, based on the high-quality action subset provided by the grasping candidate generation, and through reasonable action space design and long-term value optimization, is a technical problem that urgently needs to be solved in the field of unordered grasping planning. Summary of the Invention

[0005] The purpose of this invention is to provide a visually guided robot unordered grasping planning method based on deep reinforcement learning, in order to solve the technical problems of existing unordered grasping schemes in complex industrial scenarios, such as frequent misgrabbing and collision backtracking due to relying only on local optimal scores, and the high exploration dimensionality and difficulty in stable implementation faced by pure end-to-end deep reinforcement learning. The invention aims to balance grasping success rate, motion safety and long-term continuous box clearing benefits under limited samples.

[0006] To achieve the above objectives, this invention provides a method for unordered grasping planning of a vision-guided robot based on deep reinforcement learning, comprising the following steps: Acquire visual sensor data of the target scene and the robot's current state information; Based on the visual sensor data, a foreground point cloud is extracted, and a set of grabbing candidates filtered by physical feasibility and kinematic reachability is generated on the foreground point cloud. A reinforcement learning state vector is constructed based on the visual sensor data, the grasping candidate set, the robot's current state information, and historical execution information. The reinforcement learning state vector is input into the deep reinforcement learning policy network. The discrete action branch of the deep reinforcement learning policy network outputs the discrete index of the target grabbing candidate in the grabbing candidate set, and the pose residual and execution parameter residual for the target grabbing candidate are output through the continuous action branch. Based on the initial pose and parameters of the target grabbing candidate, the motion is calculated by combining the pose residual and the execution parameter residual to obtain the refined target pose and refined execution parameters. Motion planning and grasping execution operations are performed based on the refined target pose and the refined execution parameters.

[0007] As a preferred embodiment of the present invention, the extraction of foreground point clouds based on the visual sensor data includes: Obtain the color image and raw depth image at the current moment; The color image and the original depth image are jointly input into the depth correction sub-network, and the corrected depth image and the pixel-by-pixel depth uncertainty map are output. The depth uncertainty map is used to characterize the reliability of the depth measurement at the corresponding pixel position. Using camera intrinsic parameters and extrinsic parameters obtained from hand-eye calibration, the pixels corresponding to the corrected depth image are converted to the robotic arm's base coordinate system to generate point cloud data. The point cloud data is then preprocessed with bottom surface removal, downsampling, and outlier filtering to extract the foreground point cloud. This scheme provides a reliable geometric basis for the policy network to avoid low-confidence regions by explicitly modeling depth uncertainty.

[0008] As a preferred embodiment of the present invention, the step of generating a crawling candidate set filtered by physical feasibility and kinematic reachability on the foreground point cloud includes: An initial candidate set is generated in the foreground point cloud, and each candidate in the initial candidate set is subjected to gripper sweep body collision check and robot arm inverse kinematics solution. Candidates with collision interference or no effective inverse kinematics solution are eliminated. For each remaining candidate, geometric and physical features including local contact normal, collision margin, operability index, grasping score, depth uncertainty and occlusion are extracted. Based on the weighted calculation of the geometric and physical features, an initial comprehensive score is calculated for each candidate to be retained. Candidates are then sorted from highest to lowest based on their initial comprehensive scores, and a predetermined number of top candidates are retained. The candidate set for the current decision-making cycle consists of several candidates. This scheme compresses the continuous action domain into a high-value discrete subset, significantly reducing the exploration dimensionality and data requirements of the policy network.

[0009] As a preferred embodiment of the present invention, the reinforcement learning state vector is constructed based on the visual sensor data, the grasping candidate set, the robot's current state information, and historical execution information. The constructed reinforcement learning state vector includes the following five types of information components: Global scene features are extracted from the global depth-height map, foreground mask, and depth uncertainty map encoding. The candidate feature tensor contains the pose parameters, geometric features, and scoring information of each candidate in the capture candidate set; Robot status, including current joint angles, joint angular velocities, end effector pose, gripper opening width, and current task cumulative timer; Historical status includes a preset number of failure type codes for recent crawling attempts, the corresponding reason labels for candidate rejection, and the cumulative number of retries; The constraint information includes the effective operating boundaries of the bin, the range of the safe zone, and the failure codes returned by the motion planning module. This scheme enables the policy network to comprehensively consider real-time visual features, kinematic states, and historical error-proofing experience when making decisions.

[0010] As a preferred embodiment of the present invention, the pose residual includes the translation residual in the candidate local coordinate system. and rotation angle residual The execution parameter residuals include the gripper opening width residuals. and end approach velocity residual ; The step of calculating the action based on the initial pose and parameters of the target grabbing candidate, combined with the pose residual and the execution parameter residual, includes: Multiply the initial rotation matrix of the target grabbing candidate by the rotation transformation matrix transformed from the rotation angle residual to obtain the refined grabbing rotation matrix. The translation residual defined in the local coordinate system is transformed to the robot arm base coordinate system and then superimposed on the initial translation vector of the target grasping candidate to obtain the refined translation vector; The residuals of the gripper opening width and the end-effector approach velocity are superimposed on the initial gripper opening width and the baseline approach velocity, respectively. After truncation by upper and lower thresholds, a refined gripper opening width and actual end-effector approach velocity are obtained. This scheme enables the strategy to possess both global candidate comparison capabilities and the ability to perform local fine-tuning beyond the discrete accuracy of the candidate generator.

[0011] As a preferred embodiment of the present invention, the motion planning and grasping execution operation based on the refined target pose and the refined execution parameters includes: Based on the refined grasping rotation matrix, refined translation vector, and actual end-effector approach velocity, generate the pre-grab pose, grasp pose, lift pose, and release pose; The joint space trajectories from the current joint configuration to the pre-grasp pose, grasp pose, lift pose, and release pose are solved sequentially. The trajectory time parameterization of the segment from the pre-grasp pose to the grasp pose is constrained by the actual end-effector approach velocity. Perform discrete collision detection and continuous collision detection on each trajectory segment; If any trajectory planning fails, the collision margin is lower than the threshold, or the estimated execution time exceeds the upper limit, the current target capture candidate is marked as unexecutable, the planning failure code is written into the historical state vector, and the system is returned to the policy network to re-output the discrete index. If each trajectory segment is successfully planned and the collision margin meets the requirements, the robotic arm is controlled to sequentially execute the approach, closing, lifting, dwell verification, and placement processes. This scheme separates the grasping feasibility determination from the trajectory feasibility verification in its system architecture, using this as an execution gating mechanism to ensure the safety and reliability of physical motion.

[0012] As a preferred embodiment of the present invention, the controlled robotic arm sequentially executes the processes of approaching, closing, lifting, stopping verification, and placing, employing a multi-signal fusion strategy to determine successful grasping; a successful grasp is determined if and only if the following conditions are simultaneously met: The robotic arm has reached the preset height threshold. The opening width indicator after the grippers close indicates a partially closed state. Wrist force sensor readings or drive current proxy signals indicate the presence of an effective load at the end; After the lifting is completed, the acquired image is compared with the image before the lifting, and differential detection confirms that the object has been removed from the captured area. This scheme effectively overcomes the susceptibility of a single sensor to interference and improves the robustness of detection.

[0013] As a preferred embodiment of the present invention, if a successful capture is determined to be a failure, the failure type is classified into one of the following according to the feedback signal of the execution phase: empty capture, slippage, path unreachable, execution collision, evacuation obstruction, and timeout, and a conditional retry strategy is triggered: If the failure type is empty grab, the robotic arm applies a random residual perturbation to the current target grab candidate in the pose neighborhood and then re-executes the action calculation and execution. If the failure type is slippage, increase the clamping force setting of the gripper closing action and decrease the acceleration and speed parameters during the lifting phase during the retry. If the failure type is path unreachable, execution collision, or evacuation blocked, the current target grabbing candidate is removed from the grabbing candidate set, and the process is switched to other candidates in the set. When the cumulative number of retries exceeds the preset limit or multiple different types of failures occur consecutively, a re-sensing process is triggered to re-acquire visual sensor data for the target scene. This solution is tailored to the specific cause of failure, reducing the frequency of manual intervention and the risk of repeated failures.

[0014] As a preferred embodiment of the present invention, the reward function used by the deep reinforcement learning policy network during the training phase Defined as: in, This is a function to indicate successful fetching. Capture candidate stability quality items for the selected target; The increment of the potential function for clearing the frame is calculated based on the decrease in the number of remaining objects in the bin before and after the action is performed, the increase in the free volume in the bin, and the decrease in the scene occlusion. The execution time for a single fetch; , , , , as well as These are the corresponding penalty indicator functions; This is a retry round; to The reward component is assigned a weighting coefficient. This scheme enables the strategy to not only evaluate immediate actions but also proactively optimize the cumulative throughput of continuous clearing operations.

[0015] As a preferred embodiment of the present invention, the training and deployment of the deep reinforcement learning policy network adopts a four-stage process: Phase 1: Supervised or semi-supervised pre-training of the front-end perception network and the crawling candidate generation network using a synthetic crawling dataset; The second stage involves performing reinforcement learning training on the deep reinforcement learning policy network in a parallel environment within a physical simulation platform, based on a hybrid action flexible actor-commentator framework. The third stage involves introducing progressive domain randomization into the physical simulation platform, sequentially performing randomization operations on object geometry and friction coefficient, camera noise model and object texture, as well as controlling communication delay and systematic defects of the depth sensor. The fourth stage involves online fine-tuning and deployment on a real robot system. When conditions are met—namely, being in a non-critical production time window, having reached the minimum batch size for newly collected samples, and having no recent collision alarm events—the parameters of the deep reinforcement learning policy network are updated online using a mixed batch of real-world and simulated experience data. This approach establishes an efficient simulation-to-real-world transfer link, minimizing deployment performance gaps within a controllable budget.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention combines candidate set constraints with hybrid action deep reinforcement learning to compress the exploration scope of the policy network from the complete six-DOF continuous pose space to a candidate neighborhood pre-screened by physical feasibility and kinematic reachability. Discrete branches are responsible for global value ranking within the candidate set, while continuous branches are responsible for outputting small-range pose residuals and adjusting execution parameters. This synergy allows the policy to significantly reduce the exploration dimensionality while retaining the ability to finely correct candidate poses, thereby reducing the amount of simulation interaction and real robot data consumption required for training and improving the stability of training convergence. The reward function incorporates a box-clearing potential function increment, execution time cost, and multiple types of failure penalties. This ensures that the policy network, at each decision, not only evaluates the immediate grasping quality of the current candidate but also considers the cumulative impact of the action on subsequent box-clearing efficiency, collision risk, and retry cost. This reduces high-scoring misgrasping and invalid repetitions in continuous box-clearing operations, thereby improving overall picking throughput. The motion planning and collision detection module, acting as a feasibility gate in the policy backend, intercepts unreachable or collision-risk actions before execution and feeds back failure information to the state space for the policy network to reference in subsequent decisions. Combined with a conditional retry mechanism based on failure type classification, this reduces the collision downtime rate and the frequency of manual intervention during execution. The simulation-to-real-world transfer link, combining explicit modeling of deep uncertainty with progressive domain randomization and minimal online fine-tuning, narrows the performance gap between simulation training strategies and real deployment environments, reducing the amount of data and debugging cycle required for real-world retraining. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the unordered grasping planning method for a vision-guided robot based on deep reinforcement learning, provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Other embodiments obtained by those skilled in the art based on this invention without inventive effort are within the scope of protection of this invention.

[0019] like Figure 1 As shown, this invention provides a visually guided robot unordered grasping planning method based on deep reinforcement learning. The overall technical concept of this method is as follows: instead of directly outputting a complete six-degree-of-freedom grasping trajectory from the original image end-to-end using the reinforcement learning policy, the continuous action domain is first compressed into a set of physically feasible candidate subsets by the visual perception and grasping candidate generation stages. Then, the deep reinforcement learning policy network performs long-term value ranking and local residual correction on this candidate subset, thereby achieving coordinated optimization of grasping success rate, motion reachability, collision safety, execution time, failure retry cost, and continuous box clearing efficiency while ensuring exploration efficiency. The overall architecture of this method covers visual perception and deep geometric reconstruction, foreground point cloud preprocessing, grasping candidate generation and screening, reinforcement learning state construction and policy reasoning, action calculation, motion planning and collision detection, grasping execution and online monitoring, failure detection and conditional retry, and online fine-tuning. These stages are coupled in a closed-loop manner, jointly forming a complete grasping planning chain for unordered and scattered object scenarios in industrial warehousing.

[0020] In terms of hardware configuration, this embodiment employs at least one color-depth vision sensor, at least one multi-degree-of-freedom robotic arm, and at least one controllable end effector. As a preferred hardware implementation, this embodiment uses a fixed, top-view mounted color-depth camera, positioned approximately 0.7 to 1.2 meters directly above the material bin, ensuring the bin's opening area is completely within the sensor's field of view. This camera features a color imaging channel and an active binocular depth channel, with a working distance of approximately 0.3 to 3 meters, and a field of view sufficient for top-view coverage of conventional industrial material bins. The robotic arm is a six-degree-of-freedom serial industrial collaborative arm with a system update frequency of at least 500 Hz and sub-millimeter repeatability. The end effector is a parallel two-finger gripper with a maximum stroke of 85 mm and opening width feedback capability. In an alternative implementation, the end effector can be replaced with a suction cup or a combined gripper / suction gripper; in this case, only the parametric model and contact force modeling method of the gripping candidate need to be adjusted accordingly, while the overall planning link structure remains unchanged. The software system runs on an industrial personal computer or a workstation equipped with a graphics processing unit, requiring a minimum of 24 gigabytes of video memory, a minimum of 64 gigabytes of system memory, and a minimum of 16 processor cores.

[0021] Hardware calibration and environment initialization are performed upon system startup. The robot arm's base coordinate system is defined as follows: The camera coordinate system is The rigid body transformation from the camera coordinate system to the robot arm base coordinate system is obtained through a hand-eye calibration process. This transformation can be decomposed into a rotation matrix. With translation vector Simultaneously, the 3D boundary model of the hopper, the restricted area for the robotic arm's movement, and the geometric information of the hopper's bottom and side walls are imported into the motion planning scene for use in subsequent collision detection. The system reads and stores the camera intrinsic parameter matrix. Maximum opening width of the gripper Minimum opening width Pre-grab offset distance Evacuation and elevation and benchmark approach speed In this embodiment, the pre-grab offset distance is 50 to 80 mm, and the withdrawal lifting height is 80 to 120 mm. These parameters can be adjusted during operation according to the actual bin size and object characteristics.

[0022] See Figure 1 In each decision cycle of the planning process, the system executes the following steps step by step.

[0023] The system acquires color image frames, depth image frames, and robot joint and gripper state information at the current moment. For any pixel coordinate in the depth image... Depth value at The coordinates of the corresponding 3D spatial point in the camera coordinate system are obtained as follows: The point is then transformed to the robotic arm's base coordinate system using the rotation matrix and translation vector obtained from hand-eye calibration. The system performs temporal median filtering on the original depth map to suppress random noise, performs spatial neighborhood-based hole filling on invalid depth pixels, and extracts the object region within the bin from the background through foreground segmentation. In a preferred embodiment, the system also introduces a depth correction sub-network, which takes the joint data of the color image and the original depth map as input, performs denoising correction on the depth values ​​through an encoder-decoder structure, and outputs a corrected depth measurement map. With pixel-by-pixel depth uncertainty map In the depth uncertainty map, the value of each pixel represents the reliability of the depth measurement at that location; a higher value indicates higher uncertainty. The engineering significance of this design lies in the fact that depth sensors often experience significantly increased measurement errors when facing reflective surfaces, transparent objects, or porous structures. Without explicit modeling of this error, the subsequent evaluation of candidate locations will lack the ability to suppress low-reliability areas, ultimately leading to the strategy repeatedly selecting unreliable candidate locations. During the acquisition phase, the system also detects the effective depth pixel ratio and image exposure status. If the effective depth pixel ratio is lower than a preset threshold... If the image is overexposed, underexposed, or otherwise abnormal, the image capture process will not proceed. Instead, the image data will be re-acquired after resampling or exposure parameter adjustment.

[0024] The foreground point cloud preprocessing stage further refines the geometric properties of the obtained point cloud data. The system uses a random sample-consensus algorithm to fit the bottom plane of the bin and removes points belonging to the bottom surface. The remaining foreground point clouds are downsampled using a voxel grid method with a grid size of 3 to 5 millimeters. After downsampling, the local normal vector of each retained point is estimated, and isolated noise points are removed based on a statistical outlier detection method. The resulting foreground point cloud set is then obtained. For transparent objects, highly reflective workpieces, or objects with large areas of missing depth, the system will generate a depth uncertainty map. The percentage of effective pixels in the depth region is used as a penalty in subsequent candidate selection evaluation, thereby guiding the policy network to actively avoid regions with severely unreliable depth information during decision-making. The starting point of this process is not to improve the visual appearance quality of the point cloud, but to provide a more geometrically stable input basis for subsequent candidate selection, policy decision-making, and collision verification.

[0025] In the candidate generation stage, the system does not directly search in the continuous six-degree-of-freedom pose space, but first constructs an initial candidate set from the foreground point cloud. The specific process is as follows: For each sampling point in the foreground point cloud, its local normal, local curvature, and local height difference relative to the bottom surface of the bin are estimated. In areas with high gripping feasibility, candidate gripping frames are generated using the parallel gripper geometry model as constraints. The pose of each candidate is described by the contact point position, contact normal, gripper approach direction, and estimated gripper opening width. For the assessment of gripping feasibility, the preferred implementation method adopts a joint screening approach of gripping prior and local collision / accessibility. That is, low-value areas in the foreground point cloud are first removed through gripping mapping. After generating candidates in the remaining high-value areas, gripper sweep collision checks and bin boundary interference checks are performed on each candidate. At the same time, inverse kinematics is used to determine whether the candidate pose is within the reachable workspace of the robotic arm. Candidates with no valid inverse kinematics solutions are directly eliminated. It should be noted that although the candidates retained through the pre-screening all possess at least one set of feasible inverse kinematic solutions, there are still significant differences in kinematic quality among the different candidates. Some candidate poses are located in the central region of the robot arm's dexterity workspace, with ample joint margin and high maneuverability, while others are near joint limit positions or singular configurations, and although inverse kinematics are solvable, their motion quality and robustness during execution are poor. Therefore, in the subsequent candidate scoring, the system further calculates continuous maneuverability indices as a kinematic quality measure for each retained candidate. In an alternative implementation, the candidate generation front-end here can also be replaced by a contact map-based six-DOF candidate generation network or other deep learning-based dense pose prediction models, as long as their output format meets the input interface of the subsequent policy network.

[0026] For each candidate that passed the initial screening The system extracts the following features: the rotation matrix of the candidate pose. With translation vector Candidate gripper opening width Local contact normal Collision margin Operability indicators Score for capturing Deep uncertainty Occlusion and local color-depth image features cropped with candidate center alignment. Collision margin Defined as the minimum clearance distance between the candidate gripper sweep body and scene obstacles; operability index This is the operability value of the Jacobian matrix at the inverse kinematic solution corresponding to the candidate pose. A larger value indicates that the configuration is further from a singular configuration and has more joint margin. The value is a continuous value ranging from zero to positive real numbers. The system calculates an initial comprehensive score for each candidate pose: In the formula, to These are empirical weighting coefficients, controlling the relative contributions of grabbing score, collision margin, operability, depth uncertainty, and occlusion to the overall score. This approach allows candidates located at the center of the dexterous workspace to receive higher scores, while candidates near joint limits or in unusual configurations, even if reachable, will be ranked lower due to lower operability. This effectively differentiates the kinematic performance quality of different candidates during the ranking phase. After ranking according to this initial score from highest to lowest, the top candidates are retained. One candidate, The typical value is 32 or 64, forming the current crawling candidate set. The key significance of this step in the overall method is that deep reinforcement learning strategies no longer need to explore feasible grasps from scratch in a vast space of continuous actions. Instead, they make value judgments and fine-tuning on a subset of candidates that have already undergone preliminary verification of physical feasibility and kinematic reachability. This significantly reduces the exploration dimensionality of the strategy and the requirements for training samples. If the total number of candidates that pass the screening in the current decision-making cycle is less than the preset minimum value... If the system determines that the observation information under the current perspective is insufficient or the scene occlusion is too severe, it will not enter the strategy reasoning stage but will enter the re-perception path, that is, re-collect image data and re-execute the above perception and candidate generation process.

[0027] In the reinforcement learning state space construction phase, the system encodes all the information required for the current decision into a state vector. This state vector contains five types of information: global scene features. The candidate feature tensor is extracted from the global depth-height map, foreground mask, and uncertainty map via a convolutional coding network. ,in For a single candidate, the feature dimension includes the pose parameters, geometric features, and scoring information of each candidate; robot state. This includes the current joint angle, joint angular velocity, end effector pose, gripper opening width, and current task cumulative timer; historical status. , including recent The failure type code of each crawling attempt, the corresponding reason label for the candidate being rejected, and the cumulative number of retries; constraint information. This includes the effective working boundary of the bin, the safe zone range, and the failure code returned by the motion planning module in the previous call. By jointly encoding the above five types of information into a unified state representation, the policy network can not only refer to the current visual geometry information and candidate quality when making grasping decisions, but also perceive the uncertainty level of depth measurement, the robot's current kinematic state, and the experience accumulated from previous failed attempts. This gives it an inherent ability to suppress situations such as "graspable but not worth grasping", "repeated failures", and "planned unreachability".

[0028] The policy network employs a hybrid action flexible actor-commentator framework. This framework decomposes the action space into two branches: a discrete part and a continuous part. Discrete actions... This indicates which grabbing candidate the strategy selects from the current candidate set; continuous actions. This represents the residual correction and execution parameter adjustment applied to the selected candidate, and its specific form is as follows: in , , These represent the translation residuals along the three coordinate axes in the candidate local coordinate system. , , The rotation angle residuals are the three axes of the candidate local coordinate system. This refers to the residual width of the gripper opening. This represents the terminal approach velocity residual. In this embodiment, the range of each translational residual is truncated within... Within millimeters, the range of each rotational residual is truncated. Within a certain range, the policy exploration is strictly limited to the local neighborhood of the candidate pose. The core advantage of this hybrid action space design is that the discrete branch is responsible for global selection in the candidate set, while the continuous branch is responsible for fine-tuning in the vicinity of the selected candidate. The two work together to give the policy both the ability to compare the candidate set globally and the ability to optimize locally beyond the discrete precision limit of the candidate generator. Moreover, the dimension of the entire search space is much smaller than that of exploring directly in a complete six-degree-of-freedom continuous space. In alternative implementations, the policy network can also be implemented using other reinforcement learning frameworks such as deep deterministic policy gradient, proximal policy optimization, or pure discrete policy. In this case, the parameterization of the action space needs to be adapted accordingly, but the basic architecture of candidate constraints plus local residual correction remains the same.

[0029] Policy network output candidate index Continuous residuals And an execution confidence score If the confidence score is lower than a preset threshold... If the confidence score is not lower than the threshold, the system will not directly execute the current decision, but will instead request a re-perception or switch to a more conservative decision-making strategy. If the confidence score is not lower than the threshold, the system will proceed to the action calculation stage.

[0030] The motion calculation stage refines the pose and calculates the execution parameters for the selected candidates based on the output of the policy network. For the selected candidates... Its candidate local coordinate system is determined by the candidate rotation matrix. By definition, the translation residuals output by the policy network are first transformed to the robot arm's base coordinate system and then superimposed onto the candidate translation vectors. The refined gripping rotation matrix, translation vector, gripper opening width, and approach speed are calculated as follows: in , This indicates that the matrix exponent is taken after transforming the rotation vector into an antisymmetric matrix. Translation residual. Defined in the candidate local coordinate system, by left multiplying by the candidate rotation matrix After transforming it to the robot arm's base coordinate system, it is added to the candidate's original translation vector, thereby ensuring that the physical semantics of the residual is always fine-tuned along the local direction of the candidate grasping frame itself, and is not affected by the candidate's posture in the base coordinate system. The gripper opening width and approach speed are respectively cut off within their respective effective ranges. The system's preset benchmark approach speed, and These are the minimum and maximum permissible approach speeds, respectively. This is the actual end-effector approach velocity adjusted by the policy network. This velocity parameter is directly used in subsequent execution phases to control the end-effector linear velocity command when the robotic arm moves at a constant speed from the pre-grasping pose towards the grasping pose along the approach direction. It is also transmitted to the motion planning module as a time parameterized constraint for this trajectory segment, allowing the planner to calculate the corresponding joint velocity curve. Through this calculation, the policy network can dynamically adjust the approach velocity based on the fragility of objects in the scene, their stacking density, and historical failure experiences, thereby achieving a balance between grasping efficiency and contact safety. Based on the refined grasping pose, the system generates a pre-grasping pose with the following translation components: In the formula This is the unit vector representing the candidate approach direction. The withdrawal pose is then lifted along the vertical direction of the world coordinate system or the candidate local normal direction after the grab closure is completed. If the target pose after residual correction exceeds the reachable workspace of the robotic arm, the calculated value of the gripper opening exceeds the physical limit, or the approach direction interferes with the wall of the hopper, the system determines that the current action is invalid, marks the candidate as unexecutable, and returns to the strategy reasoning stage to select a new candidate.

[0031] The motion planning and collision detection stages use the refined target pose sequence and its corresponding velocity parameters as input. This embodiment employs a sampling-based motion planning framework, loading the planning scene with a 3D geometric model or computer-aided design model of the bin, obstacle representations converted from real-time point clouds, and an object model already attached to the robot's end effector. The planner sequentially solves the joint space trajectories for each segment: from the current joint configuration to the pre-grasp pose, from the pre-grasp pose to the grasp pose, from the grasp pose to the lift pose, and from the lift pose to the deployment pose. The time parameterization of the trajectory segment from the pre-grasp pose to the grasp pose is influenced by the approach velocity output from the motion calculation stage. Constraints are defined, and the planner generates joint velocity curves that meet the end effector linear velocity requirements. For each trajectory segment, the system performs discrete and continuous collision detection to ensure that the minimum clearance between the robotic arm, end effector, and any obstacles in the scene is greater than a safety margin threshold throughout the entire motion. If the planning solution for any trajectory segment fails, the collision margin is lower than the threshold, or the estimated execution time exceeds the allowable upper limit, the planning segment returns the corresponding failure code. The system marks the candidate as unexecutable, writes the failure code into the historical state, and returns to the strategy reasoning stage to reselect candidates. This stage plays the role of execution feasibility gating in the overall method: the grasping decision output by the policy network is only sent to the robotic arm controller for execution after being verified as actually executable through motion planning and collision detection, thus separating the grasping feasibility determination from the trajectory feasibility determination at the system architecture level.

[0032] The grasping execution phase employs a five-stage process: approach, closing, lifting, pause verification, and placement. After the robotic arm moves to the pre-grasping pose according to the planned trajectory, it uses the approach velocity output from the motion calculation phase along the candidate approach direction. The robotic arm moves at a constant linear speed to the grasping position. Upon reaching the grasping position, the gripper performs a closing action using a preset force or speed pattern. After closing, the robotic arm rises along the evacuation direction to the lifting position. It pauses at the lifting peak for approximately 0.3 to 0.5 seconds, during which the system verifies successful grasping. If the grasp is successful, the robotic arm moves the object to the target delivery container according to the unloading path. During execution, the system monitors the wrist force / torque signal in real time. If the force or torque reading exceeds the safety threshold, the current movement is immediately stopped, the robotic arm is controlled to retreat to a safe position, and a failure label indicating a collision type is recorded.

[0033] Successful grasping is determined using a multi-signal fusion strategy. A grasp is considered successful when multiple conditions are met simultaneously: the robotic arm's lifting height reaches a preset threshold; the gripper's opening width after closing is not completely closed, indicating that an object is indeed being held between the grippers; the wrist force sensor or drive current proxy signal indicates the presence of an effective load at the end effector; and after lifting, image differences before and after lifting confirm that the object in the grasped area has been removed. This multi-signal fusion determination method offers higher reliability than relying solely on gripper opening or force sensors, effectively addressing minute opening differences caused by thin workpieces or signal deviations caused by sensor drift.

[0034] Failure detection and classification are crucial components of this method. This embodiment categorizes grasping failures into six types: empty grasp, slippage, path unreachable, execution collision, evacuation obstruction, and timeout. The criteria for an empty grasp are: the gripper opening width is nearly fully closed after closure and the load agent signal is below the minimum load threshold. The criteria for slippage are: a sudden drop in the load agent signal during lifting, an abnormal change in the gripper opening width, or visual detection of the grasped object falling off during transport. Path unreachable failures originate from planning failure codes returned by the motion planning module. Execution collision failures are triggered by an emergency stop due to real-time force / torque monitoring exceeding a safety threshold during execution. Evacuation obstruction failures are caused by motion planning failure during the lifting phase or abnormal resistance detected during lifting monitoring. Timeout failures are triggered when the total execution time of a single grasp exceeds the preset maximum allowable duration.

[0035] The system invokes different retry strategies based on the failure type, rather than simply repeating the same strategy for all failures. For air-grabbing failures, it prioritizes applying small-scale random residual perturbations to the current candidate within its pose neighborhood before re-execution to test whether minor deviations in the candidate's pose caused the grasp to fail to contact the object. For slippage failures, it prioritizes increasing the clamping force setting and reducing the acceleration and velocity during the lifting phase to improve clamping stability. For path unreachable or collision failures, the system directly removes the current candidate from the available set and switches to a candidate with a higher collision margin for replanning and execution. For evacuation obstruction failures, it similarly removes the current candidate and switches to a candidate with a smoother evacuation path. If multiple consecutive failures occur with different failure types, the system triggers a re-sensing process, re-acquiring image data and re-executing sensing and candidate generation. When the cumulative number of retries exceeds a preset limit... When the current grabbing round ends, the system ends the current grabbing round and records the complete log information for that round. As a subordinate implementation, a slight tossing or separating motion can be introduced during retries for unreachable paths or execution collisions to change the stacking state of objects in the bin, creating more favorable geometric conditions for subsequent grabbing.

[0036] The policy network is trained using a reward-shaping-based reinforcement learning method. The reward function is defined as: In the formula, This is a function to indicate successful capture; it takes a value of 1 if successful and 0 otherwise. The stability quality term for the selected candidate reflects the geometric and mechanical grasping quality of the candidate pose; The box clearing potential function increment measures the contribution of the current grabbing action to the efficiency of subsequent box clearing. This refers to the execution time of this capture; Indicator functions for executing collision-triggered or over-force-triggered actions; This is an indicator function for empty capture; This represents the cumulative number of retry rounds currently being captured; This is a timeout indicator function; This is an indicator function for slippage, which takes a value of 1 when an object is detected to have detached during the lifting process; This is an indicator function for when a path is unreachable; it takes the value 1 when the motion planning module returns a planning failure code. This is an indicator function for obstructed evacuation, and its value is 1 when a planning failure or abnormal resistance is detected during the lifting phase. to These are the weighting coefficients for each reward component. The penalty term in the reward function corresponds exactly to the six failure types identified in the failure detection stage, ensuring that the policy network receives a clear negative feedback signal for each failure mode during training, thereby learning differentiated avoidance behaviors for different failure causes. Specifically, the collision penalty weight... and path unreachable penalty weight The slippage penalty weight can be set to a larger value to enhance the strategy's sensitivity to collision risk and kinematic constraints. The value of should encourage the strategy to choose candidates with higher contact stability or appropriately reduce the approach speed and the penalty weight for obstructed withdrawal. The guidance strategy considers the accessibility of evacuation paths when selecting candidates. The clearing potential function is defined as: In the formula, This represents the total number of objects remaining in the bin, including both visible and occluded objects. This is an estimate of the free volume inside the hopper; This is a measure of the degree of scene occlusion. to These are the corresponding weights, all of which are positive. Clear the box potential function increment. This is equal to the difference between the clearing potential function after the current grasping action and the value before the action. When an object is successfully removed from the bin... Decrease makes Increase the free volume of the hopper As it increases, the degree of scene occlusion also increases. This usually decreases as well, and the three components change in the same direction, making... A positive value is used here to generate positive incentives in the reward function. Instead of simply counting visible objects, this approach addresses the issue that in heavily occluded scenes, many objects may be invisible but still exist within the bin. Using only the visible object count as a potential function component would generate falsely high potential values ​​in such scenarios, misleading the strategy into believing the bin clearing status is favorable. By introducing this potential function increment, the reward function not only encourages the strategy to complete the current grasp but also guides it to select actions that make the layout of the remaining objects in the bin easier to grasp—that is, pursuing a long-term goal where the post-grab scene is better than the pre-grab scene. This design is one of the key features that distinguishes this method from existing schemes that only optimize single-step grasping scores.

[0037] The policy network employs a dual-commenter structure to suppress overestimation of the value function. Specifically, the system initializes two parameter-independent commenter networks. and and the corresponding target network and The policy objective function is: in This is a discrete candidate selection strategy. For continuous residual strategy, and These are the entropy regularization temperature coefficients for the discrete and continuous parts, respectively. and For two commentator networks, This serves as an experience replay buffer. The target value is calculated using the smaller of the two target commentator network outputs: In the formula As a discount factor, For the immediate reward at the current moment, For the state at the next moment, and For the next time-instance strategy in the state Action sampling below, and There are two target commenter networks, whose parameters are derived from the corresponding online commenter networks. and Synchronization is performed periodically using a soft update method. This embodiment employs training techniques such as prioritizing experience replay and reweighting of real samples. Typical hyperparameter settings include: discount factor. Target network soft update coefficient Learning rate The training batch size is 256.

[0038] The training process is divided into four phases. In the first phase, supervised or semi-supervised pre-training of the front-end perception network and the grasping candidate generation network is performed using a large-scale synthetic grasping dataset and synthetic data from cluttered scenes. This enables the candidate generator to output high-quality candidates under various cluttered scenarios. In the second phase, a bin simulation environment is established on a physical simulation platform. Massive parallel environment acceleration technology is used to train the hybrid action flexible actor-commentator strategy, with approximately five million to twenty million simulation interactions and a training duration of approximately 20 to 40 hours. In the third phase, a progressive domain randomization strategy is introduced. Randomization proceeds by first randomizing the object geometry and friction coefficient, then randomizing the camera noise model and object texture, and finally randomizing control communication delay and systematic defects of the depth sensor. This gradually increases the distribution difference between the simulation and the real environment to improve the strategy's generalization ability to real-world changes. In the fourth phase, a small amount of interaction data is collected on a real robot. Deployment and adaptation are performed using a combination of simulation-guided exploration and offline-to-online policy updates. The number of grasping attempts required for real-world online fine-tuning is approximately 500 to 3000. The design of this four-stage training process follows the principle of division of labor: simulation is responsible for learning the planning logic, and real data is responsible for correcting sensing and contact deviations. This allows for the achievement of deployable policy performance within a limited budget of real robot data.

[0039] A safety protection mechanism is implemented during the online fine-tuning phase. The system will only perform online strategy parameter updates if all of the following conditions are met: the system is currently within a non-production critical time window; the number of newly collected real transfer samples has reached the minimum batch requirement; and no high-risk collision alarm events have occurred recently. If any of the above conditions are not met, the system will only write the newly collected samples to the experience buffer and log them, without modifying the parameters of the current online strategy. During online updates, the system mixes samples from the simulation experience pool and the real experience pool at a preset ratio to form a training batch, with the sampling weight of real samples set to 3.0 to enhance the strategy's adaptation to the distribution characteristics of the real environment. This controlled update mechanism ensures the safety and controllability of the system during engineering deployment.

[0040] The complete execution process of this method is illustrated using an industrial warehouse bin picking scenario as an example. Twenty storage items of varying shapes are randomly stacked inside the bin, including regular boxes, cylindrical parts, sheet parts, and partially reflective workpieces. After system startup, calibration and environmental initialization are completed first. In each grasping cycle, the system acquires the current color-depth image, generates a foreground point cloud after depth correction and foreground extraction, and generates a candidate set on the point cloud through grasping guidance and collision / reachability filtering. The policy network infers from this candidate set by combining global scene features, candidate features, robot state, and historical failure information, selects a candidate, and outputs pose and velocity residuals. The motion calculation module superimposes the residuals onto the candidate poses and generates a complete pre-grasp-grasp-retreat pose sequence and approach velocity parameters. The motion planning module solves the joint trajectories of each segment in a planning scenario with a bin model and real-time point cloud obstacles, and verifies them through collision detection. After verification, the robotic arm sequentially executes the approach, close, lift, and stop verification process. If the judgment is successful, the object is moved to the target container; if the judgment fails, the corresponding retry strategy is executed according to the failure type. This process is repeated until all objects in the bin are picked out or the termination condition is met. As the number of objects in the bin gradually decreases, the clearing potential function guides the strategy to select an action sequence that is conducive to subsequent grasping, thereby maintaining high grasping efficiency throughout the entire clearing cycle.

[0041] Based on the above embodiments, those skilled in the art can make reasonable modifications according to specific application scenarios and engineering conditions. Regarding sensor configuration, a second vision sensor can be added above the material bin to form multi-view fusion observation, or a secondary camera can be installed on the wrist of the robotic arm for close-range precise positioning. Regarding the end effector, when the target object is a soft package or an irregularly shaped workpiece, it can be replaced with a suction cup-type end effector. In this case, the contact model in the parameterized model of the grasping candidate changes from the symmetrical contact of the gripper's two fingers to the contact of the suction cup's adsorption surface, and the opening width parameter in the candidate features is replaced with the suction cup diameter parameter, while the rest of the planned link structure remains consistent. When it is necessary to process both suction and gripping modes simultaneously, a combined gripping-suction end effector can be used, and the discrete action branches of the policy network are correspondingly expanded to a joint selection of candidate indices and grasping modes. Regarding the specific implementation of the policy network, the encoding of global scene features can use convolutional neural networks, visual transformers, or other network structures suitable for point cloud / image feature extraction; the aggregation of candidate features can use an attention mechanism to model cross-candidate relationships on the candidate set. Regarding the training platform, the simulation environment can also be replaced with other physical simulation engines that support parallel reinforcement learning training. The range of domain randomization parameters and the progressive pacing during training can be adjusted according to the actual characteristics of the target deployment environment. All the above modifications are equivalent substitutions or natural extensions within the scope of the core technical concept of this invention, and do not depart from the spirit and essence of this invention.

[0042] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for unordered grasping planning of a vision-guided robot based on deep reinforcement learning, characterized in that, Includes the following steps: Acquire visual sensor data of the target scene and the robot's current state information; Based on the visual sensor data, a foreground point cloud is extracted, and a set of grabbing candidates filtered by physical feasibility and kinematic reachability is generated on the foreground point cloud. A reinforcement learning state vector is constructed based on the visual sensor data, the grasping candidate set, the robot's current state information, and historical execution information. The reinforcement learning state vector is input into the deep reinforcement learning policy network. The discrete action branch of the deep reinforcement learning policy network outputs the discrete index of the target grabbing candidate in the grabbing candidate set, and the pose residual and execution parameter residual for the target grabbing candidate are output through the continuous action branch. Based on the initial pose and parameters of the target grabbing candidate, the motion is calculated by combining the pose residual and the execution parameter residual to obtain the refined target pose and refined execution parameters. Motion planning and grasping execution operations are performed based on the refined target pose and the refined execution parameters.

2. The method according to claim 1, characterized in that, The extraction of the foreground point cloud based on the visual sensor data includes: Obtain the color image and raw depth image at the current moment; The color image and the original depth image are jointly input into the depth correction sub-network, and the corrected depth image and the pixel-by-pixel depth uncertainty map are output. The depth uncertainty map is used to characterize the reliability of the depth measurement at the corresponding pixel position. Using the camera's intrinsic parameters and the extrinsic parameters obtained from hand-eye calibration, the pixels corresponding to the corrected depth image are converted to the robotic arm's base coordinate system to generate point cloud data. The point cloud data is then preprocessed by removing the bottom surface, downsampling, and filtering out outliers to extract the foreground point cloud.

3. The method according to claim 2, characterized in that, The process of generating a crawling candidate set on the foreground cloud, filtered by physical feasibility and kinematic reachability, includes: An initial candidate set is generated in the foreground point cloud, and each candidate in the initial candidate set is subjected to gripper sweep body collision check and robot arm inverse kinematics solution. Candidates with collision interference or no effective inverse kinematics solution are eliminated. For each remaining candidate, geometric and physical features including local contact normal, collision margin, operability index, grasping score, depth uncertainty and occlusion are extracted. Based on the weighted calculation of the geometric and physical features, an initial comprehensive score is calculated for each candidate to be retained. Candidates are then sorted from highest to lowest based on their initial comprehensive scores, and a predetermined number of top candidates are retained. The candidate set for the current decision-making cycle consists of several candidates.

4. The method according to claim 2, characterized in that, The reinforcement learning state vector is constructed based on the visual sensor data, the grasping candidate set, the robot's current state information, and historical execution information. The constructed reinforcement learning state vector contains the following five types of information components: Global scene features are extracted from the global depth-height map, foreground mask, and depth uncertainty map encoding. The candidate feature tensor contains the pose parameters, geometric features, and scoring information of each candidate in the capture candidate set; Robot status, including current joint angles, joint angular velocities, end effector pose, gripper opening width, and current task cumulative timer; Historical status includes a preset number of failure type codes for recent crawling attempts, the corresponding reason labels for candidate rejection, and the cumulative number of retries; Constraint information includes the effective operating boundaries of the bin, the safe zone range, and the failure codes returned by the motion planning module.

5. The method according to claim 1, characterized in that, The pose residual includes the translation residual in the candidate local coordinate system. and rotation angle residual The execution parameter residuals include the gripper opening width residuals. and end approach velocity residual ; The step of calculating the action based on the initial pose and parameters of the target grabbing candidate, combined with the pose residual and the execution parameter residual, includes: Multiply the initial rotation matrix of the target grabbing candidate by the rotation transformation matrix transformed from the rotation angle residual to obtain the refined grabbing rotation matrix. The translation residual defined in the local coordinate system is transformed to the robot arm base coordinate system and then superimposed on the initial translation vector of the target grasping candidate to obtain the refined translation vector; The residual difference in the gripper opening width and the residual difference in the end-effector approach speed are respectively superimposed on the initial gripper opening width and the reference approach speed. After being truncated by upper and lower limit thresholds, the refined gripper opening width and the actual end-effector approach speed are obtained.

6. The method according to claim 5, characterized in that, The motion planning and grasping execution operation based on the refined target pose and the refined execution parameters includes: Based on the refined grasping rotation matrix, refined translation vector, and actual end-effector approach velocity, generate the pre-grab pose, grasp pose, lift pose, and release pose; The joint space trajectories from the current joint configuration to the pre-grasp pose, grasp pose, lift pose, and release pose are solved sequentially. The trajectory time parameterization of the segment from the pre-grasp pose to the grasp pose is constrained by the actual end-effector approach velocity. Perform discrete collision detection and continuous collision detection on each trajectory segment; If any trajectory planning fails, the collision margin is lower than the threshold, or the estimated execution time exceeds the upper limit, the current target capture candidate is marked as unexecutable, the planning failure code is written into the historical state vector, and the system is returned to the policy network to re-output the discrete index. If the trajectory planning for each segment is successful and the collision margin meets the requirements, the robotic arm is controlled to sequentially execute the approach, closing, lifting, dwell verification, and placement processes.

7. The method according to claim 6, characterized in that, The controlled robotic arm sequentially executes the approach, closing, lifting, pause verification, and placement processes, employing a multi-signal fusion strategy to determine successful grasping; a grasp is considered successful if and only if the following conditions are met simultaneously: The robotic arm has reached the preset height threshold. The opening width indicator after the grippers close indicates a partially closed state. Wrist force sensor readings or drive current proxy signals indicate the presence of an effective load at the end; After the lifting is completed, the image acquired is compared with the image before the lifting, and differential detection confirms that the object has been removed from the captured area.

8. The method according to claim 7, characterized in that, If a successful capture is judged as a failure, the failure type is classified into one of the following based on the feedback signals during the execution phase: empty capture, slippage, path unreachable, execution collision, evacuation obstruction, and timeout, and a conditional retry strategy is triggered: If the failure type is empty grab, the robotic arm applies a random residual perturbation to the current target grab candidate in the pose neighborhood and then re-executes the action calculation and execution. If the failure type is slippage, increase the clamping force setting of the gripper closing action and decrease the acceleration and speed parameters during the lifting phase during the retry. If the failure type is path unreachable, execution collision, or evacuation blocked, the current target grabbing candidate is removed from the grabbing candidate set, and the process is switched to other candidates in the set. When the cumulative number of retries exceeds the preset limit or multiple different types of failures occur consecutively, the re-sensing process is triggered to re-acquire visual sensor data of the target scene.

9. The method according to any one of claims 1 to 8, characterized in that, The reward function used by the deep reinforcement learning policy network during the training phase Defined as: in, This is a function to indicate successful fetching. Capture candidate stability quality items for the selected target; The increment of the potential function for clearing the frame is calculated based on the decrease in the number of remaining objects in the bin before and after the action is performed, the increase in the free volume in the bin, and the decrease in the scene occlusion. The execution time for a single fetch; , , , , as well as These are the corresponding penalty indicator functions for execution collision, empty grab, timeout, slippage, path unreachable, and evacuation obstruction; This is a retry round; to This refers to the weighting coefficient corresponding to the reward component.

10. The method according to claim 1, characterized in that, The training and deployment of the deep reinforcement learning policy network adopts a four-stage process: Phase 1: Supervised or semi-supervised pre-training of the front-end perception network and the crawling candidate generation network using a synthetic crawling dataset; The second stage involves performing reinforcement learning training on the deep reinforcement learning policy network in a parallel environment within a physical simulation platform, based on a hybrid action flexible actor-commentator framework. The third stage involves introducing progressive domain randomization into the physical simulation platform, sequentially performing randomization operations on object geometry and friction coefficient, camera noise model and object texture, as well as controlling communication delay and systematic defects of the depth sensor. Phase 4: Online fine-tuning and deployment in a real robot system. When the conditions are met—that it is in a non-critical production time window, the number of newly collected samples reaches the minimum batch size, and there are no recent collision alarm events—the parameters of the deep reinforcement learning policy network are updated online using a mixed batch of real experience data and simulation experience data.

Images