Disordered workpiece grabbing method based on three-dimensional visual perception and intelligent decision

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a disordered workpiece grabbing method based on three-dimensional visual perception and intelligent decision, and the method comprises the steps: collecting the color image data and depth image data of a scene, and generating the three-dimensional point cloud data of the scene based on the internal reference of a camera; performing target detection and instance segmentation processing based on the color image data to generate target segmentation mask data, calculating an outlier degree score and a top degree score of a target in combination with depth data, and selecting an optimal capture target according to a comprehensive score; through an intelligent target selection strategy based on YOLOv11-SAMV3 fusion, in combination with an outlier degree score and a top degree score, a grabbing target is comprehensively evaluated, an isolated workpiece located at the top of a stack is preferentially selected, the shielding and collision risks are reduced, intelligent and automatic grabbing target selection is guaranteed, and the grabbing efficiency and the system adaptability are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making. Background Technology

[0002] With the development of robotics technology, the application scenarios of existing unordered grasping technology are constantly expanding. Reinforcement learning methods that use grasping success rate as the training objective of the network cannot effectively meet the multi-indicator differentiated needs of different application scenarios for unordered grasping. Efficient multi-objective optimization of unordered grasping behavior of robots is of great practical significance for improving the customized working ability of robots and expanding the application scenarios of robots.

[0003] Currently, in the field of industrial automated sorting, existing technologies such as deep learning solutions like GraspNet have significant drawbacks in the method of grasping disordered workpieces based on 3D vision perception: the target selection strategy is not intelligent enough and cannot comprehensively evaluate the occlusion state, isolation degree and spatial position of the workpiece in real time, which makes it easy to make suboptimal grasping target selection in mixed loading scenarios, increasing the risk of collision and the probability of grasping failure.

[0004] Therefore, a method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making is proposed to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making, which solves the problems mentioned in the background technology, such as the tendency to select suboptimal grasping targets, increasing collision risks and the probability of grasping failure.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making, the method comprising the following steps: Collect color and depth image data of the scene, and generate 3D point cloud data of the scene based on camera intrinsic parameters; Target detection and instance segmentation are performed based on the color image data to generate target segmentation mask data. The outlier score and apex score of the target are calculated by combining the depth data. The best target to be captured is selected based on the comprehensive score. Based on the point cloud region corresponding to the optimal grasping target, candidate grasping pose data is generated, and the surface normal alignment constraint and centroid distance constraint are fused to perform pose scoring processing to generate the optimal grasping pose data. A closed-loop pose fine-tuning strategy is constructed, and the optimal grasping pose is subjected to real-time visual servo compensation processing through the hovering, alignment and grasping stages, and the robotic arm is controlled to perform the grasping action. The outlier score is characterized by calculating the minimum Euclidean distance between the center point of the target bounding box and all other target center points, and the apex score is characterized by the minimum value of the target region depth. The pose scoring process includes a weighted fusion of the original GraspNet score, surface normal alignment score, and centroid distance score. The real-time visual servo compensation process is based on the calculation of translation amount based on the deviation between the target center and the gripper center in the image coordinate system.

[0007] Preferably, the acquisition of color image data and depth image data of the scene includes the following steps: Color and depth images of the scene are acquired using a 3D structured light camera; Based on camera intrinsic parameters and depth images, 3D point cloud data of the generated scene is calculated using a coordinate transformation formula, where the coordinate transformation formula is: ; in To retrieve the depth value corresponding to a pixel (u,v) from a depth image, The column coordinates of the pixel in the image. Here are the row coordinates of the pixel in the image. , The principal point coordinates of the camera. , Let be the focal length of the camera in the x and y directions. Let x be the x-coordinate of the point cloud in the camera coordinate system. Let y be the point cloud coordinate in the camera coordinate system. Let z be the z-coordinate of the point cloud in the camera coordinate system; The generated 3D point cloud data is preprocessed by denoising and filtering to remove outliers and redundant data, and the resulting optimized point cloud data is used for subsequent processing.

[0008] Preferably, the target detection and instance segmentation process includes the following steps: S21. Input the color image data into the YOLOv11 network for target detection processing to generate a set of bounding boxes for potential targets. , The total number of elements in the set; Each detection box is used as a cue input to the SAM model for instance segmentation, generating a pixel-level segmentation mask for each target; Each segmentation mask is mapped onto the depth image, and the set of depth values ​​corresponding to the target region is extracted. The apex score of the target is calculated based on the depth value set, and the calculation formula is as follows: : in The apex degree score is given for the i-th target. This is the set of depth values ​​corresponding to the i-th target region; The outlier score of the target is calculated using the following formula: ; in The outlier score for the i-th target is... Let be the coordinates of the center point of the i-th target bounding box. Let J represent the coordinates of the center point of the bounding box of the j-th target. Calculate the overall score for each objective: ; in and These are the weighting coefficients. Let be the overall score of the i-th target; select the target with the highest overall score as the best target to capture.

[0009] Preferably, the extraction of the depth value set corresponding to the target region includes a step of calculating the proportion of effective depth points: The proportion of valid depth values ​​in the target area to the total number of points is counted. If the proportion is lower than the threshold, the target is excluded to ensure data integrity. The weighting coefficients w1 and w2 were calibrated experimentally to prioritize the dominant role of outlier scores.

[0010] Preferably, generating the optimal grasping pose data includes the following steps: The point cloud region corresponding to the best grasping target is input into the pre-trained GraspNet model to generate a set of candidate grasping poses. , The set G contains the total number of candidate capture poses, each pose including position, orientation, width and the original GraspNet score; For each candidate grasping pose, extract the neighborhood point cloud of its grasping point, fit the local surface through principal component analysis, and calculate the unit normal vector. The direction of the normal vector is uniformly corrected to point outward from the object. The formula for calculating the surface normal alignment score is as follows: ; in The surface normal alignment score has a value range of [-1, 1]. The angle between the grabber approach vector and the surface normal vector; The formula for calculating the centroid distance score is as follows: ; in The centroid distance score is given with a value range of [0,1]. To capture the Euclidean distance from the point to the 3D centroid of the target point cloud, These are normalization parameters; Calculate the overall score for candidate poses: ; in The overall score for the candidate capture pose. The weighting coefficients of the original GraspNet score. The weighting coefficients for surface normal alignment scores are as follows: The weighting coefficient for the centroid distance score. The original scores generated for the GraspNet model; The pose with the highest overall score is selected as the optimal grasping pose.

[0011] Preferably, the local surface fitting uses the RANSAC algorithm instead of principal component analysis; The weighting coefficients are determined through grid search optimization and satisfy the constraint condition α+β+γ=1.

[0012] Preferably, the construction of the closed-loop pose fine-tuning strategy includes the following steps: Control the robotic arm to move to a preset safe height position directly above the optimal grasping posture, and record it as the pre-grasping posture; At the pre-captured pose, the current color image and depth image are acquired in real time using a 3D camera; Run the YOLOv11 and SAM models again on the current image to detect the best grasping target and calculate its center coordinates in the image coordinate system, while extracting the center coordinates of the gripper. Based on the camera intrinsic parameters and the current depth value, the pixel deviation is converted into a translation in the robot's base coordinate system. The calculation formula is as follows: ; ;

[0013] in This represents the translation in the x-direction within the robot's base coordinate system. This represents the translation in the y-direction within the robot's base coordinate system. Let x be the x-coordinate of the center of the target in the image coordinate system. Let y be the center coordinate of the target in the image coordinate system. Let x be the x-coordinate of the center of the gripper in the image coordinate system. Let be the ordinate of the center of the gripper in the image coordinate system. This is the current depth value. Let be the focal length of the camera along the x-axis. Let be the focal length of the camera along the y-axis. Control the movement of the robotic arm to compensate for deviations, so that the end of the gripper is aligned with the center of the target; The robotic arm descends vertically along the Z-axis to the grasping height, continuously monitoring image deviations during the process, and repeating fine adjustments when the deviation exceeds a threshold. Finally, the crawling action is executed, and the crawling results are reported back to the system log.

[0014] Preferably, the preset safety height is adaptively adjusted according to the target size and the motion dynamics parameters of the robotic arm; The deviation threshold is set through calibration experiments, allowing for fine-tuning of accuracy without introducing oscillations.

[0015] Preferably, the method further includes: After grasping, the verification process confirms the success of the grasp based on force sensors and visual feedback. If it fails, the S2-S4 process is retried. The method also includes an integrated incremental learning mechanism, which uses the captured data to update the GraspNet model and weight parameters, thereby improving the system's adaptability.

[0016] Preferably, the incremental learning mechanism employs an online gradient descent algorithm, updating the model parameters after each N crawling tasks are completed. The method is based on the ROS framework to achieve modular integration, in which the target selection, pose scoring and fine-tuning modules run as independent nodes and data interaction is achieved through topic communication.

[0017] Compared with existing technologies, this invention provides a method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making, which has the following beneficial effects: 1. In this invention, an intelligent target selection strategy based on YOLOv11-SAMV3 fusion is used to comprehensively evaluate the grasping target by combining outlier score and top score. The strategy prioritizes isolated workpieces located at the top of the stack, reducing the risk of occlusion and collision, ensuring the intelligent and automated selection of grasping targets, and improving grasping efficiency and system adaptability.

[0018] 2. In this invention, by integrating surface normal alignment constraints and centroid distance constraints into a grasping pose scoring mechanism, the original GraspNet score is co-optimized with geometric and physical constraints. Grasping poses with both high stability and low collision risk are selected in real time to generate the optimal grasping decision, thereby reducing pose deviation and failure probability during the grasping process.

[0019] 3. In this invention, a closed-loop pose fine-tuning strategy of "hover-align-grab" is adopted to compensate for the cumulative errors of camera calibration and robotic arm movement in real time by using visual servoing. This enables the gripper end to be precisely aligned with the target center, suppresses the execution deviation of the grasping pose, ensures the reliability and robustness of the grasping action, and improves the grasping success rate of the system in complex scenarios. Attached Figure Description

[0020] Figure 1 This is a flowchart of a method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] For specific implementation examples, please refer to: Figure 1 A method for grasping disordered workpieces based on 3D visual perception and intelligent decision-making, the method includes the following steps: Collect color and depth image data of the scene, and generate 3D point cloud data of the scene based on camera intrinsic parameters; Target detection and instance segmentation are performed based on color image data to generate target segmentation mask data. The outlier score and apex score of the target are calculated by combining depth data. The best target to be captured is selected based on the comprehensive score. Based on the point cloud region corresponding to the best grasping target, candidate grasping pose data is generated, and the surface normal alignment constraint and centroid distance constraint are fused to perform pose scoring processing to generate the optimal grasping pose data. A closed-loop pose fine-tuning strategy is constructed, and the optimal grasping pose is processed in real time by visual servo compensation during the hovering, alignment and grasping stages, and the robotic arm is controlled to perform the grasping action. The outlier score is represented by calculating the minimum Euclidean distance between the center point of the target bounding box and the center points of all other targets, while the apex score is represented by the minimum value of the depth of the target region. The pose scoring process includes a weighted fusion of the original GraspNet score, surface normal alignment score, and centroid distance score. The real-time visual servo compensation processing calculates the translation amount based on the deviation between the target center and the gripper center in the image coordinate system.

[0023] Acquire color and depth image data of the scene, including the following steps: Color and depth images of the scene are acquired using a 3D structured light camera; Based on camera intrinsic parameters and depth images, 3D point cloud data of the generated scene is calculated using a coordinate transformation formula, where the coordinate transformation formula is: ; in To retrieve the depth value corresponding to a pixel (u,v) from a depth image, The column coordinates of the pixel in the image. Here are the row coordinates of the pixel in the image. , The principal point coordinates of the camera. , Let be the focal length of the camera in the x and y directions. Let x be the x-coordinate of the point cloud in the camera coordinate system. Let y be the point cloud coordinate in the camera coordinate system. Let z be the z-coordinate of the point cloud in the camera coordinate system; The generated 3D point cloud data is preprocessed by denoising and filtering to remove outliers and redundant data, and the optimized point cloud data is used for subsequent processing. First, a statistical outlier removal algorithm is used to process the point cloud data. This algorithm calculates the average distance between each point and its nearest neighbor, and removes points that deviate from the mean by more than a threshold based on the distance distribution. The threshold is usually set as a multiple of the standard deviation of the distance to the mean, 2 times the standard deviation, to remove noisy points. Secondly, voxel grid filtering is applied for downsampling. By dividing the point cloud space into a uniform voxel grid and retaining the centroid point within each voxel, the amount of data is reduced while maintaining the shape characteristics of the point cloud. The voxel size is set according to the point cloud density, usually from 0.005 meters to 0.01 meters. Finally, radius filtering is used to further smooth the point cloud and remove points with insufficient neighboring points within a specified radius to eliminate isolated noise. These steps are performed sequentially to ensure that the point cloud data is clean and compact, providing high-quality input for subsequent processing.

[0024] Object detection and instance segmentation processing includes the following steps: S21. Input the color image data into the YOLOv11 network for target detection processing to generate a set of bounding boxes for potential targets. , The total number of elements in the set; The YOLOv11 network is based on the optimized YOLO architecture, using Darknet-53 as the backbone network and introducing the path aggregation network PANet to enhance feature fusion. In implementation, the input color image is first preprocessed, including resizing the image to 640×640 pixels and normalizing pixel values ​​to the range [0,1]. Then, the image is input into the network, which outputs bounding box predictions. Each bounding box contains center coordinates, width, height, target confidence, and class probability. The bounding box decoding formula is: ; ; ; ; in The x-coordinate of the center point of the bounding box. Here is the y-coordinate of the center point of the bounding box. The width of the bounding box. The height of the bounding box. For the sigmoid function, This represents the original x-direction offset output by the network. This represents the original y-direction offset output by the network. This is the width offset of the network output. This is the height offset of the network output. This represents the x-axis offset of the grid cell. This represents the y-direction offset of the grid cell. The preset width of the prior bounding box. The preset height of the prior bounding box is used; after decoding, non-maximum suppression (NMS) is applied, and the overlap threshold is set to 0.5 to remove redundant detection boxes, and finally the set of bounding boxes is output; the entire process is implemented based on the PyTorch framework, and the model weights are obtained by pre-training on the COCO dataset; Each detection box is used as a cue input to the SAM model for instance segmentation, generating a pixel-level segmentation mask for each target; SAM is based on the Vision Transformer architecture and supports zero-shot instance segmentation. In implementation, each bounding box output by YOLOv11 is first converted into a cue format, including box coordinates and optional point cue. Then, the cue and color image are fed into the SAM model, and the model generates a segmentation mask through an encoder-decoder structure. The encoder extracts image features, and the decoder combines the cue information to output a binary mask, where a pixel value of 1 represents the target region and 0 represents the background. After the mask is generated, post-processing is performed, including morphological operations to smooth the edges and filtering small connected regions to remove noise. Finally, each target corresponds to an accurate pixel-level mask for subsequent depth information extraction. The processing uses a pre-trained version of SAM, which can adapt to various workpiece types without fine-tuning. Each segmentation mask is mapped onto the depth image, and the set of depth values ​​corresponding to the target region is extracted. The apex score of the target is calculated based on the depth value set, and the calculation formula is as follows: : in The apex degree score is given for the i-th target. This is the set of depth values ​​corresponding to the i-th target region; The outlier score of the target is calculated using the following formula: ; in The outlier score for the i-th target is... Let be the coordinates of the center point of the i-th target bounding box. Let J represent the coordinates of the center point of the bounding box of the j-th target. Calculate the overall score for each objective: ; in and These are the weighting coefficients. Let be the overall score of the i-th target; select the target with the highest overall score as the best target to capture.

[0025] Extracting the set of depth values ​​corresponding to the target region includes the steps for calculating the proportion of effective depth points: The proportion of valid depth values ​​in the target area to the total number of points is counted. If the proportion is lower than the threshold, the target is excluded to ensure data integrity. The weighting coefficients w1 and w2 were calibrated experimentally to prioritize the dominant role of outlier scores.

[0026] Generating optimal grasping pose data includes the following steps: The point cloud region corresponding to the best grasping target is input into the pre-trained GraspNet model to generate a set of candidate grasping poses. , The set G contains the total number of candidate capture poses, each pose including position, orientation, width and the original GraspNet score; For each candidate grasping pose, extract the neighborhood point cloud of its grasping point, fit the local surface through principal component analysis, and calculate the unit normal vector. The direction of the normal vector is uniformly corrected to point outward from the object. The formula for calculating the surface normal alignment score is as follows: ; in The surface normal alignment score has a value range of [-1, 1]. The angle between the grabber approach vector and the surface normal vector; The formula for calculating the centroid distance score is as follows: ; in The centroid distance score is given with a value range of [0,1]. To capture the Euclidean distance from the point to the 3D centroid of the target point cloud, These are normalization parameters; Calculate the overall score for candidate poses: ; in The overall score for the candidate capture pose. The weighting coefficients of the original GraspNet score. The weighting coefficients for surface normal alignment scores are as follows: The weighting coefficient for the centroid distance score. The original scores generated for the GraspNet model; The pose with the highest overall score is selected as the optimal grasping pose.

[0027] Local surface fitting uses the RANSAC algorithm instead of principal component analysis; For the RANSAC algorithm, it is applied to the neighborhood point cloud of candidate grasping points to fit a local planar surface. The algorithm randomly samples the minimum point set to calculate the planar model and counts the number of interior points; after multiple iterations, the model with the most interior points is selected as the final plane, and its unit normal vector is calculated; the direction of the normal vector is corrected by the centroid of the point cloud to ensure that it points outward from the object. The weighting coefficients are determined through grid search optimization and satisfy the constraint condition α+β+γ=1; For the grid search optimization of weight coefficients α, β, and γ, the parameter grid is defined as follows: α ranges from [0.1 to 0.5] with a step size of 0.1; β ranges from [0.2 to 0.6] with a step size of 0.1; γ ranges from [0.2 to 0.6] with a step size of 0.1, and satisfies the constraint α + β + γ = 1. The success rate of each parameter combination is evaluated on the validation set, and the optimal combination is selected. Cross-validation is used during the search process to ensure the robustness of the coefficients.

[0028] Constructing a closed-loop pose fine-tuning strategy includes the following steps: Control the robotic arm to move to a preset safe height position directly above the optimal grasping posture, and record it as the pre-grasping posture; At the pre-captured pose, the current color image and depth image are acquired in real time using a 3D camera; Run the YOLOv11 and SAM models again on the current image to detect the best grasping target and calculate its center coordinates in the image coordinate system, while extracting the center coordinates of the gripper. Based on the camera intrinsic parameters and the current depth value, the pixel deviation is converted into a translation in the robot's base coordinate system. The calculation formula is as follows: ; ;

[0029] in This represents the translation in the x-direction within the robot's base coordinate system. This represents the translation in the y-direction within the robot's base coordinate system. Let x be the x-coordinate of the center of the target in the image coordinate system. Let y be the center coordinate of the target in the image coordinate system. Let x be the x-coordinate of the center of the gripper in the image coordinate system. Let be the ordinate of the center of the gripper in the image coordinate system. This is the current depth value. Let be the focal length of the camera along the x-axis. Let be the focal length of the camera along the y-axis. Control the movement of the robotic arm to compensate for deviations, so that the end of the gripper is aligned with the center of the target; The robotic arm descends vertically along the Z-axis to the grasping height, continuously monitoring image deviations during the process, and repeating fine adjustments when the deviation exceeds a threshold. Finally, the crawling action is executed, and the crawling results are reported back to the system log.

[0030] The preset safety height is adaptively adjusted based on the target size and the robot arm's motion dynamics parameters; The preset safety height is dynamically calculated based on the target size and the robotic arm's motion dynamics parameters. During implementation, the 3D bounding box dimensions are first extracted from the target point cloud, and then the safety margin is calculated based on the robotic arm's maximum acceleration and response time. The formula is: ; in: Preset safe height; This is the scaling factor, with a default value of 1.2; Let be the maximum value of the target 3D bounding box size, where For length, For width, For height; This is the dynamic coefficient, with a default value of 0.5; This is the maximum acceleration of the robotic arm; This refers to the system response time. After calculation, Limit the height to a reasonable range of 0.1 to 0.5 meters to avoid being too high or too low. Adjustments should be made in real time to ensure the optimal safe height for each grabbing task. The deviation threshold is set through calibration experiments, allowing for fine-tuning of accuracy without introducing oscillations.

[0031] The method also includes: After grasping, the verification process confirms the success of the grasp based on force sensors and visual feedback. If it fails, the S2-S4 process is retried. The method also includes an integrated incremental learning mechanism, which uses the captured data to update the GraspNet model and weight parameters, thereby improving the system's adaptability. The incremental learning mechanism is triggered upon completion of the crawling task and is based on an online learning framework. First, it collects crawling result data, including success and failure labels, point cloud fragments, and crawling poses. Then, it updates the model parameters using an exponential moving average: for the GraspNet model weights θ, the update formula is: ; in For the updated model parameters, These are the model parameters before the update. Forgetting factor, The gradient of the loss function. The loss function; An update is performed after every N fetches, using mini-batch gradient descent with a learning rate of 0.001. The mechanism integrates a rollback function; if performance degrades, the old parameters are restored to ensure stability.

[0032] The incremental learning mechanism uses an online gradient descent algorithm to update the model parameters after every N crawling tasks are completed. The linear gradient descent algorithm is used for the incremental learning described above, and the parameter update formula is: ; in These are the model parameters for the next time step. Here are the model parameters at the current time step, and α is the learning rate. For the loss function in parameters gradient at, This serves as the index for time steps; the algorithm employs adaptive learning rate adjustment to improve convergence. The method as a whole is based on the ROS framework to achieve modular integration, in which the target selection, pose scoring and fine-tuning modules run as independent nodes and data interaction is achieved through topic communication; For ROS framework integration, the system is modularized into independent nodes: target selection node: runs YOLOv11-SAM; pose scoring node: runs GraspNet and scoring logic; fine adjustment point: handles visual servoing; nodes communicate with each other through ROS topics; synchronous calls are implemented using ROS services to ensure real-time performance; the framework is deployed on the Ubuntu system, and nodes are started through launch files.

[0033] The steps of this method are as follows: In the data acquisition and point cloud generation stage, color and depth images of the scene are first acquired using a 3D structured light camera. Based on the camera's intrinsic parameters, the depth data is converted into 3D point cloud data. This process includes coordinate transformation and point cloud preprocessing, denoising, and filtering to remove outliers and redundant data, generating an optimized point cloud for subsequent processing.

[0034] The next stage is target detection and intelligent selection, which involves target detection and instance segmentation based on color image data. A YOLOv11 network is used to generate bounding boxes for potential targets, and then the SAM model is used for instance segmentation to obtain pixel-level segmentation masks. Combined with depth data, outlier score and top-degree score are calculated for each target. The outlier score assesses the degree of isolation by the minimum Euclidean distance between the target's bounding box center and the center points of other targets, while the top-degree score assesses spatial location by the minimum depth value of the target region. Simultaneously, the proportion of effective depth points is considered to ensure data integrity, and a weighted fusion is used to calculate a comprehensive score, prioritizing workpieces with high outlier scores and located at the top as the best grasping targets.

[0035] The next stage is the grasping pose generation and scoring phase. The point cloud region corresponding to the optimal grasping target is input into the pre-trained GraspNet model to generate a set of candidate grasping poses, each containing position, orientation, and width information. Pose scoring is performed by fusing surface normal alignment constraints and centroid distance constraints. The surface normal alignment score evaluates the alignment between the grasper's proximity vector and the local surface normal vector, while the centroid distance score evaluates the distance from the grasping point to the target's centroid. Combining the original GraspNet scores, a weighted fusion score is calculated to select the optimal grasping pose. Local surface fitting can be performed using principal component analysis or the RANSAC algorithm, with weight coefficients determined through grid search optimization.

[0036] Finally, the closed-loop pose fine-tuning and execution phase involves constructing a closed-loop strategy for hovering, alignment, and grasping. The robotic arm first moves to a safe height directly above the optimal grasping pose. This preset safe height is adaptively adjusted based on the target size and the robotic arm's dynamic parameters. At the pre-grasping pose, images are acquired in real-time, and the target detection and segmentation model is run again to calculate the image coordinate deviation between the target center and the gripper center. Based on camera intrinsic parameters and depth values, the pixel deviation is converted into a translation in the robot's base coordinate system, controlling the robotic arm to compensate for the deviation and ensure the gripper's end is precisely aligned with the target center. During the robotic arm's descent along the Z-axis, the deviation is continuously monitored, and fine-tuning is performed when it exceeds a threshold, ultimately executing the grasping action. After grasping, verification is performed using force sensors and visual feedback; if failure occurs, the process is retried. The system's integrated incremental learning mechanism periodically updates model parameters to improve adaptive capabilities. The entire method is modularly integrated based on the ROS framework, with each module acting as an independent node and collaborating through topic communication.

[0037] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0038] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making, characterized in that: The method includes the following steps: Collect color and depth image data of the scene, and generate 3D point cloud data of the scene based on camera intrinsic parameters; Target detection and instance segmentation are performed based on the color image data to generate target segmentation mask data. The outlier score and apex score of the target are calculated by combining the depth data. The best target to be captured is selected based on the comprehensive score. Based on the point cloud region corresponding to the optimal grasping target, candidate grasping pose data is generated, and the surface normal alignment constraint and centroid distance constraint are fused to perform pose scoring processing to generate the optimal grasping pose data. A closed-loop pose fine-tuning strategy is constructed, and the optimal grasping pose is subjected to real-time visual servo compensation processing through the hovering, alignment and grasping stages, and the robotic arm is controlled to perform the grasping action. The outlier score is characterized by calculating the minimum Euclidean distance between the center point of the target bounding box and all other target center points, and the apex score is characterized by the minimum value of the target region depth. The pose scoring process includes a weighted fusion of the original GraspNet score, surface normal alignment score, and centroid distance score. The real-time visual servo compensation process calculates the translation amount based on the deviation between the target center and the gripper center in the image coordinate system.

2. The method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to claim 1, characterized in that: The acquisition of color image data and depth image data of the scene includes the following steps: Color and depth images of the scene are acquired using a 3D structured light camera; Based on camera intrinsic parameters and depth images, 3D point cloud data of the generated scene is calculated using a coordinate transformation formula, where the coordinate transformation formula is: ； in To retrieve the depth value corresponding to a pixel (u,v) from a depth image, The column coordinates of the pixel in the image. Here are the row coordinates of the pixel in the image. , The principal point coordinates of the camera. , Let be the focal length of the camera in the x and y directions. Let x be the x-coordinate of the point cloud in the camera coordinate system. Let y be the point cloud coordinate in the camera coordinate system. Let z be the z-coordinate of the point cloud in the camera coordinate system; The generated 3D point cloud data is preprocessed by denoising and filtering to remove outliers and redundant data, and the resulting optimized point cloud data is used for subsequent processing.

3. The method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to claim 1, characterized in that: The target detection and instance segmentation process includes the following steps: Color image data is input into the YOLOv11 network for object detection processing, generating a set of bounding boxes for potential targets. , The total number of elements in the set; Each detection box is used as a cue input to the SAM model for instance segmentation, generating a pixel-level segmentation mask for each target; Each segmentation mask is mapped onto the depth image, and the set of depth values ​​corresponding to the target region is extracted. The apex score of the target is calculated based on the depth value set, and the calculation formula is as follows: ： in The apex degree score is given for the i-th target. This is the set of depth values ​​corresponding to the i-th target region; The outlier score of the target is calculated using the following formula: ； in The outlier score for the i-th target is... Let be the coordinates of the center point of the i-th target bounding box. Let J represent the coordinates of the center point of the bounding box of the j-th target. Calculate the overall score for each objective: ； in and These are the weighting coefficients. Let be the overall score of the i-th target; select the target with the highest overall score as the best target to capture.

4. The method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to claim 3, characterized in that: The extraction of the depth value set corresponding to the target region includes the step of calculating the proportion of effective depth points: The proportion of valid depth values ​​in the target area to the total number of points is counted. If the proportion is lower than the threshold, the target is excluded to ensure data integrity. The weighting coefficients w1 and w2 were calibrated experimentally to prioritize the dominant role of outlier scores.

5. The method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to claim 1, characterized in that: The process of generating optimal grasping pose data includes the following steps: The point cloud region corresponding to the best grasping target is input into the pre-trained GraspNet model to generate a set of candidate grasping poses. , The set G contains the total number of candidate capture poses, each pose including position, orientation, width and the original GraspNet score; For each candidate grasping pose, extract the neighborhood point cloud of its grasping point, fit the local surface through principal component analysis, and calculate the unit normal vector. The direction of the normal vector is uniformly corrected to point outward from the object. The formula for calculating the surface normal alignment score is as follows: ； in The surface normal alignment score has a value range of [-1, 1]. The angle between the grabber approach vector and the surface normal vector; The formula for calculating the centroid distance score is as follows: ； in The centroid distance score is given with a value range of [0,1]. To capture the Euclidean distance from the point to the 3D centroid of the target point cloud, These are normalization parameters; Calculate the overall score for candidate poses: ； in The overall score for the candidate capture pose. The weighting coefficients of the original GraspNet score. The weighting coefficients for surface normal alignment scores are as follows: The weighting coefficient for the centroid distance score. The original scores generated for the GraspNet model; The pose with the highest overall score is selected as the optimal grasping pose.

6. The method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to claim 5, characterized in that: The local surface fitting uses the RANSAC algorithm instead of principal component analysis. The weighting coefficients are determined through grid search optimization and satisfy the constraint condition α+β+γ=1.

7. The method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to claim 1, characterized in that: The method for constructing a closed-loop pose fine-tuning strategy includes the following steps: Control the robotic arm to move to a preset safe height position directly above the optimal grasping posture, and record it as the pre-grasping posture; At the pre-captured pose, the current color image and depth image are acquired in real time using a 3D camera; Run the YOLOv11 and SAM models again on the current image to detect the best grasping target and calculate its center coordinates in the image coordinate system, while extracting the center coordinates of the gripper. Based on the camera intrinsic parameters and the current depth value, the pixel deviation is converted into a translation in the robot's base coordinate system. The calculation formula is as follows: ； ； in This represents the translation in the x-direction within the robot's base coordinate system. This represents the translation in the y-direction within the robot's base coordinate system. Let x be the x-coordinate of the center of the target in the image coordinate system. Let y be the center coordinate of the target in the image coordinate system. Let x be the x-coordinate of the center of the gripper in the image coordinate system. Let be the ordinate of the center of the gripper in the image coordinate system. This is the current depth value. Let be the focal length of the camera along the x-axis. Let be the focal length of the camera along the y-axis. Control the movement of the robotic arm to compensate for deviations, so that the end of the gripper is aligned with the center of the target; The robotic arm descends vertically along the Z-axis to the grasping height, continuously monitoring image deviations during the process, and repeating fine adjustments when the deviation exceeds a threshold. Finally, the crawling action is executed, and the crawling results are reported back to the system log.

8. The method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to claim 7, characterized in that: The preset safety height is adaptively adjusted according to the target size and the motion dynamics parameters of the robotic arm. The deviation threshold is set through calibration experiments, allowing for fine-tuning of accuracy without introducing oscillations.

9. The method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to claim 1, characterized in that: The method further includes: After grasping, the verification process confirms the success of the grasp based on force sensors and visual feedback. If it fails, the S2-S4 process is retried. The method also includes an integrated incremental learning mechanism, which uses the captured data to update the GraspNet model and weight parameters, thereby improving the system's adaptability.

10. A method for grasping disordered workpieces based on three-dimensional visual perception and intelligent decision-making according to claim 9, characterized in that: The incremental learning mechanism described above uses an online gradient descent algorithm to update the model parameters after every N crawling tasks are completed. The method is based on the ROS framework to achieve modular integration, in which the target selection, pose scoring and fine-tuning modules run as independent nodes and data interaction is achieved through topic communication.

Images