Mechanical arm grabbing method and system based on semantic point cloud top surface principal axis analysis

Abstract

The present application relates to the technical field of robot grabbing, and particularly relates to a mechanical arm grabbing method and system based on semantic point cloud top surface principal axis analysis, which comprises the following steps: acquiring a color image, a depth image and camera intrinsic parameters, inputting a text prompt, and generating a binary mask of a target through a multi-modal segmentation model; performing back projection on the depth image with the binary mask and depth validity as joint screening conditions to obtain target point cloud; taking the depth value of the nearest point plus a fixed layer thickness as a slice threshold to extract a top surface thin layer point cloud, wherein the top surface thin layer point cloud is a set of all three-dimensional points with depth values falling within the interval from the depth value of the nearest point to the depth value of the nearest point plus the fixed layer thickness; obtaining a three-dimensional arithmetic mean of all points of the top surface thin layer point cloud to obtain a grabbing center point in a camera coordinate system; constructing a two-dimensional covariance matrix after projecting the top surface thin layer point cloud to a horizontal plane and performing feature decomposition; and obtaining a gripper opening and closing direction angle after rotating and taking modulus normalization in the direction of the feature vector corresponding to the maximum eigenvalue.

Description

Technical Field

[0001] This invention relates to the field of robot grasping technology, and in particular to a robotic arm grasping method and system based on semantic point cloud top surface principal axis analysis. Background Technology

[0002] Visual grasping systems typically identify the target first and then solve for the grasping pose. Early recognition methods relied on fixed template matching, comparing pre-established object appearance templates pixel-by-pixel with real-time acquired images. This approach was poorly robust to changes in lighting and viewing angle, and a new template library needed to be built for each new object. With the rise of convolutional neural networks, closed-set detectors (such as the YOLO series and Faster R-CNN) and instance segmentation models (such as Mask R-CNN) achieved end-to-end object recognition and segmentation. However, these models could only recognize predefined object categories in the training set. Adding new object categories required re-collecting labeled data and fine-tuning the model, resulting in high maintenance costs in diverse industrial sorting scenarios. In recent years, large-scale vision-language models (such as CLIP, Grounding DINO, and Segment Anything) have aligned text semantics with image features through large-scale multimodal pre-training, achieving open-vocabulary object detection and segmentation. Operators only need to specify the target object with natural language text prompts, without needing to retrain or fine-tune the segmentation model to adapt to new objects, significantly reducing adaptation costs.

[0003] In terms of grasping pose determination, methods based on CAD model template matching require the pre-establishment of accurate 3D models for each type of object. After registering the real-time point cloud with the model, the pose is estimated. This method has poor generalization ability and a large modeling workload. End-to-end deep learning methods (such as GraspNet-1 Billion and GPD) directly use depth maps or 3D point clouds as input and grasping candidates as output for supervised training. They perform well on objects within the training set distribution, but the success rate drops significantly on objects outside the training set distribution, and the inference results lack interpretability. Methods based on point cloud geometric analysis extract geometric features from the point cloud through operations such as normal vector estimation and principal component analysis to determine the grasping pose without additional training data. However, they are sensitive to background points, desktop points, and point clouds of adjacent objects, and lack a mechanism to pass hard constraints of recognition semantic results into the geometric processing flow.

[0004] In existing technologies, a typical scheme based on the back projection of the detection frame uses the depth value at the center pixel of the detection frame and the camera intrinsic parameter matrix as the back projection to obtain the three-dimensional coordinates. These coordinates are used as the gripping center, and the gripper approaches the target in a preset fixed posture (usually vertically downward). This scheme is simple in process, but the gripping angle lacks adaptability to the shape of the object: for slender workpieces (such as screwdrivers or long strips), the misalignment of the gripper opening and closing direction with the main axis of the object can easily lead to gripping failure or the workpiece slipping out of the gripper; in addition, the depth value at the center of the detection frame may fall in the middle, side, or even the background of the object, resulting in a large positioning error.

[0005] Furthermore, the end-to-end point cloud solution for the entire scene backprojects the camera depth map into a full-scene point cloud and directly feeds it into the grasping network (such as GraspNet). The network predicts multiple candidate grasping poses for each point and outputs a score, taking the one with the highest score as the final grasping candidate. This solution does not distinguish between points in the background, desktop, and adjacent objects; all points participate in inference, and the computational cost is positively correlated with the density of the scene point cloud. When there are multiple adjacent objects in the working area, the network has difficulty accurately separating specific semantic targets from the background, and its generalization ability to objects outside the training distribution is limited, making it unsuitable for industrial scenarios where object types change frequently.

[0006] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0007] This invention provides a robotic arm grasping method and system based on semantic point cloud top surface principal axis analysis, thereby effectively solving the problems in the background technology.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: a robotic arm grasping method based on semantic point cloud top surface principal axis analysis, comprising the following steps:

[0009] Acquire time-synchronized color images, depth maps, and camera intrinsic parameters; input text prompts; and generate a binary mask of the target from the color images and text prompts using a multimodal segmentation model.

[0010] The depth map is back-projected using the binary mask and depth validity as joint filtering conditions, and only the 3D points corresponding to pixels that simultaneously satisfy the mask being true and the depth value being valid are retained to obtain the target point cloud.

[0011] Traverse all points in the target point cloud along the camera optical axis direction corresponding to the depth map, and extract the top surface thin layer point cloud using the depth value of the nearest point plus a fixed layer thickness as the slicing threshold. The top surface thin layer point cloud is a set of all three-dimensional points whose depth values ​​fall within the range of the depth value of the nearest point to the depth value of the nearest point plus a fixed layer thickness.

[0012] The three-dimensional arithmetic mean of all points in the thin point cloud on the top surface is used to obtain the grab center point in the camera coordinate system. After projecting the thin point cloud on the top surface onto the horizontal plane, a two-dimensional covariance matrix is ​​constructed and eigenvalue decomposition is performed. The gripper opening and closing direction angle is obtained by rotating and normalizing the eigenvector corresponding to the largest eigenvalue.

[0013] The gripping center point and the opening and closing direction angle of the gripper are transformed to the coordinate system of the robotic arm base according to the pre-calibrated homogeneous transformation matrix from the camera to the robotic arm base, thereby driving the robotic arm to perform a sequence of actions including approaching, gripping, lifting, and placing.

[0014] Furthermore, the multimodal segmentation model undergoes size adjustment and normalization processing before the color image is input;

[0015] When the multimodal segmentation model outputs a detection box, it expands the detection box outward by a preset number of pixels in the four directions of up, down, left, and right, then crops it to the image boundary and rasterizes it into a binary mask. The preset number of pixels ranges from 10 to 40 pixels.

[0016] Furthermore, the depth value is valid if it is within the camera's range and is not zero, or is not NaN or an invalid marker value;

[0017] The back projection uses the inverse of the camera intrinsic parameter matrix to convert pixel coordinates and depth values ​​into three-dimensional spatial coordinates, and the joint filtering of mask and depth is performed pixel by pixel.

[0018] Furthermore, the fixed layer thickness ranges from 5 mm to 100 mm, and the fixed layer thickness is related to the opening width of the two-finger gripper and the curvature of the top surface of the object;

[0019] The extraction of the thin point cloud on the top surface is based on the premise that the angle between the camera's optical axis and the direction of gravity is less than a preset angle threshold, and the installation is either top-down or near-top-down.

[0020] Furthermore, the opening and closing direction angle of the gripper is determined by the following steps:

[0021] The horizontal coordinates of each point are obtained by projecting the thin point cloud on the top surface onto a horizontal plane.

[0022] Construct a two-dimensional covariance matrix and perform eigenvalue decomposition;

[0023] Rotate the eigenvector corresponding to the largest eigenvalue by 90°.

[0024] The rotated orientation angle is normalized to the interval [0°, 180°) by taking the modulus of 180°.

[0025] Furthermore, the grasping center point and the gripper opening / closing direction angle share the same top surface thin-layer point cloud, and the two parameters are solved in a single point cloud traversal. The gripper opening / closing direction angle is obtained through closed-form interpretable principal component analysis.

[0026] Furthermore, the sequence of actions performed by the driven robotic arm, including approaching, grasping, lifting, and placing, includes:

[0027] The robotic arm moves to a proximity pose that is offset vertically by a preset proximity height directly above the grasping center point, and the gripper has opened to a preset width before moving to the proximity pose;

[0028] It then descends vertically to the gripping center point and, in accordance with the gripping posture of the opening and closing direction angle of the gripper, performs gripper closure to complete the gripping;

[0029] After clamping, the object is lifted vertically upwards to a preset lifting height, thus detaching it from the worktable.

[0030] Move to the preset target position and perform the gripper opening and placement action.

[0031] Furthermore, the method also includes:

[0032] When the binary mask is empty or its area is less than a preset threshold, skip the current frame and re-acquire the image;

[0033] When the number of points in the top surface thin-layer point cloud is less than a preset lower limit, the current target is abandoned and an alarm is triggered;

[0034] When the ratio of the largest eigenvalue to the second largest eigenvalue of the two-dimensional covariance matrix is ​​less than a preset threshold, the opening and closing direction angle of the gripper adopts a preset default direction or is manually specified by the operator.

[0035] Furthermore, when the text prompt is set to "object", the scattered workpieces on the workbench are picked up sequentially.

[0036] When the text prompt is set to a specific category, specific parts are sorted from mixed workpieces;

[0037] Changing the text prompt switches the sorting object.

[0038] The present invention also includes a robotic arm grasping system based on semantic point cloud top surface principal axis analysis, using the method described above, the system comprising:

[0039] The camera is used to acquire time-synchronized color images, depth images, and camera intrinsic parameters;

[0040] An input unit is used to input text prompts, and a binary mask of the target is generated from the color image and the text prompts using a multimodal segmentation model;

[0041] The target point cloud unit is used to back-project the depth map using the binary mask and depth validity as joint filtering conditions, and retain only the three-dimensional points corresponding to pixels that simultaneously satisfy the mask being true and the depth value being valid, to obtain the target point cloud.

[0042] The top surface thin-layer point cloud extraction unit is used to traverse all points in the target point cloud along the camera optical axis direction corresponding to the depth map, and extract the top surface thin-layer point cloud by using the depth value of the nearest point plus a fixed layer thickness as the slicing threshold. The top surface thin-layer point cloud is a set of all three-dimensional points whose depth values ​​fall within the range of the depth value of the nearest point to the depth value of the nearest point plus a fixed layer thickness.

[0043] The grasping calculation unit is used to calculate the three-dimensional arithmetic mean of all points in the top surface thin point cloud to obtain the grasping center point in the camera coordinate system. After projecting the top surface thin point cloud onto the horizontal plane, a two-dimensional covariance matrix is ​​constructed and eigenvalue decomposition is performed. The gripper opening and closing direction angle is obtained by rotating and normalizing the eigenvector direction corresponding to the largest eigenvalue.

[0044] The robotic arm and drive unit are used to transform the gripping center point and the opening and closing direction angle of the gripper to the coordinate system of the robotic arm base according to the pre-calibrated homogeneous transformation matrix from the camera to the robotic arm base, and drive the robotic arm to perform a sequence of approaching, gripping, lifting and placing actions.

[0045] The present invention also includes a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described above.

[0046] The present invention also includes a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0047] The beneficial effects of this invention are as follows: text prompts allow for changing the grasping target without retraining the model; mask and depth validity are combined logically and methodically as back-projection filtering conditions, allowing semantic results to directly enter geometric processing with hard constraints; the top surface thin-layer slice uses "nearest depth + fixed layer thickness" as the threshold, eliminating the need to know the absolute height of the object; the semantic mask hard constraints and the two-level geometric filtering of the top surface slice suppress interference from the background and adjacent objects; the gripper opening and closing direction angle is obtained by rotating the maximum eigenvector of the two-dimensional covariance matrix and taking the modulus, which is closed-form and interpretable; the grasping center and the gripper opening and closing direction angle share the same top surface thin-layer point cloud, which is completed in a single point cloud traversal, resulting in low computational overhead. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;

[0050] Figure 2 This is a schematic diagram of the system structure in Embodiment 1 of the present invention;

[0051] Figure 3 This is a flowchart of the method in Embodiment 2 of the present invention;

[0052] Figure 4 This is a schematic diagram of the top surface thin slice and the main axis direction in Embodiment 2 of the present invention;

[0053] Figure 5 This is a schematic diagram of the computer device in Embodiment 3 of the present invention. Detailed Implementation

[0054] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0055] Example 1:

[0056] like Figure 1 As shown: A robotic arm grasping method based on semantic point cloud top surface principal axis analysis includes the following steps:

[0057] Acquire time-synchronized color images, depth maps, and camera intrinsics; input text prompts; and generate a binary mask of the target from the color images and text prompts using a multimodal segmentation model.

[0058] The depth map is back-projected using binary mask and depth validity as joint filtering conditions, and only the 3D points corresponding to pixels that simultaneously satisfy the conditions of true mask and valid depth value are retained to obtain the target point cloud.

[0059] Traverse all points in the target point cloud along the camera optical axis direction corresponding to the depth map, and use the depth value of the nearest point plus a fixed layer thickness as the slicing threshold to extract the top thin layer point cloud. The top thin layer point cloud is a set of all three-dimensional points whose depth values ​​fall within the range of the depth value of the nearest point to the depth value of the nearest point plus a fixed layer thickness.

[0060] The three-dimensional arithmetic mean of all points in the top thin point cloud is used to obtain the grab center point in the camera coordinate system. After projecting the top thin point cloud onto the horizontal plane, a two-dimensional covariance matrix is ​​constructed and eigenvalue decomposition is performed. The gripper opening and closing direction angle is obtained by rotating and normalizing the eigenvector corresponding to the largest eigenvalue.

[0061] Based on the pre-calibrated homogeneous transformation matrix from the camera to the robotic arm base, the gripping center point and the opening and closing direction angle of the gripper are transformed to the coordinate system of the robotic arm base, driving the robotic arm to execute a sequence of approaching, gripping, lifting, and placing actions.

[0062] Text prompts allow for changing the grasping target without retraining the model. Masking and depth validity are combined logically and methodically as back-projection filtering conditions, allowing semantic results to directly enter geometric processing with hard constraints. The top surface thin-layer slice uses "nearest depth + fixed layer thickness" as the threshold, eliminating the need to know the absolute height of the object. The semantic mask hard constraints and the two-level geometric filtering of the top surface slice suppress interference from the background and adjacent objects. The gripper opening and closing direction angle is obtained by rotating the maximum eigenvector of the two-dimensional covariance matrix and taking the modulus, which is closed-form and interpretable. The grasping center and the gripper opening and closing direction angle share the same top surface thin-layer point cloud, which is completed in a single point cloud traversal, resulting in low computational overhead.

[0063] In this embodiment, the multimodal segmentation model undergoes size adjustment and normalization processing before the color image is input;

[0064] When the multimodal segmentation model outputs a detection box, it expands the detection box outward by a preset number of pixels in the four directions of up, down, left, and right, then crops it to the image boundary and rasterizes it into a binary mask. The preset number of pixels ranges from 10 to 40 pixels.

[0065] Among them, a valid depth value is defined as: a depth value that is within the camera's range and is not zero, not NaN, or an invalid marker value;

[0066] Back projection uses the inverse of the camera intrinsic matrix to convert pixel coordinates and depth values ​​into three-dimensional spatial coordinates, and the joint filtering of mask and depth is performed pixel by pixel.

[0067] As a preferred embodiment of the above, the fixed layer thickness ranges from 5 mm to 100 mm, and the fixed layer thickness is related to the opening width of the two-finger gripper and the curvature of the top surface of the object.

[0068] Extraction of thin-layer point clouds on the top surface is based on the premise that the angle between the camera's optical axis and the direction of gravity is less than a preset angle threshold, and the camera is installed in a top-down or near-top-down view.

[0069] The opening and closing direction angle of the gripper is determined by the following steps:

[0070] The horizontal coordinates of each point are obtained by projecting the thin point cloud on the top surface onto the horizontal plane.

[0071] Construct a two-dimensional covariance matrix and perform eigenvalue decomposition;

[0072] Rotate the eigenvector corresponding to the largest eigenvalue by 90°.

[0073] The rotated orientation angle is normalized to the interval [0°, 180°) by taking the modulus of 180°.

[0074] In this embodiment, the gripping center point and the gripper opening / closing direction angle share the same top surface thin layer point cloud. A single point cloud traversal completes the solution of both parameters, and the gripper opening / closing direction angle is obtained through closed-form interpretable principal component analysis.

[0075] The sequence of actions that drives the robotic arm to perform, including approaching, grasping, lifting, and placing, includes:

[0076] The robotic arm moves to a proximity pose that is offset vertically by a preset proximity height directly above the center point of the gripper, and the gripper has already opened to a preset width before moving to the proximity pose;

[0077] Then it descends vertically to the center point of the gripper and, in accordance with the gripper opening and closing direction angle, executes the gripper closing to complete the gripping;

[0078] After clamping, the object is lifted vertically upwards to a preset lifting height, thus detaching it from the worktable.

[0079] Move to the preset target position and perform the gripper opening and placement action.

[0080] As a preferred embodiment of the above, the method further includes:

[0081] When the binary mask is empty or its area is less than a preset threshold, skip the current frame and re-acquire the image;

[0082] When the number of points in the top thin-layer point cloud is less than the preset lower limit, abandon the current target and issue an alarm;

[0083] When the ratio of the largest eigenvalue to the second largest eigenvalue of the two-dimensional covariance matrix is ​​less than a preset threshold, the opening and closing direction angle of the gripper adopts a preset default direction or is manually specified by the operator.

[0084] When the text prompt is set to "object", the scattered workpieces on the workbench will be picked up sequentially.

[0085] When the text prompt is set to a specific category, it sorts out specific parts from mixed workpieces;

[0086] Changing the text prompt switches the sorting object.

[0087] like Figure 2 As shown, this embodiment also includes a robotic arm grasping system based on semantic point cloud top surface principal axis analysis, using the method described above. The system includes:

[0088] The camera is used to acquire time-synchronized color images, depth images, and camera intrinsic parameters;

[0089] The input unit is used to input text prompts, and a binary mask of the target is generated from the color image and text prompts through a multimodal segmentation model;

[0090] The target point cloud unit is used to back-project the depth map using a binary mask and depth validity as joint filtering conditions, and retain only the 3D points corresponding to pixels that simultaneously satisfy the mask being true and the depth value being valid, thus obtaining the target point cloud.

[0091] The top surface thin-layer point cloud extraction unit is used to traverse all points in the target point cloud along the camera optical axis direction corresponding to the depth map, and extract the top surface thin-layer point cloud by using the depth value of the nearest point plus a fixed layer thickness as the slicing threshold. The top surface thin-layer point cloud is a set of all three-dimensional points whose depth values ​​fall within the range of the depth value of the nearest point to the depth value of the nearest point plus a fixed layer thickness.

[0092] The grasping calculation unit is used to calculate the three-dimensional arithmetic mean of all points in the top thin point cloud to obtain the grasping center point in the camera coordinate system. After projecting the top thin point cloud onto the horizontal plane, a two-dimensional covariance matrix is ​​constructed and eigenvalue decomposition is performed. The gripper opening and closing direction angle is obtained by rotating and normalizing the eigenvector corresponding to the largest eigenvalue.

[0093] The robotic arm and drive unit are used to transform the gripping center point and the opening and closing direction angle of the gripper to the coordinate system of the robotic arm base according to the pre-calibrated homogeneous transformation matrix from the camera to the robotic arm base, and drive the robotic arm to perform a sequence of approaching, gripping, lifting and placing actions.

[0094] Example 2:

[0095] This embodiment presents a method that uses a text-driven semantic mask as a geometric gate for depth back projection and simultaneously solves for the gripping position (3D mean) and the gripper opening / closing direction angle (principal component analysis (PCA) principal axis rotated 90°) on a thin point cloud on the top surface. The method includes the following steps:

[0096] Acquire time-synchronized color images, depth maps, and camera intrinsics, and input a text prompt;

[0097] A multimodal segmentation model supporting text prompts is used to obtain detection boxes or binary masks for one or more targets from a color image and text prompts. When there are multiple candidates, one is selected according to a preset strategy (highest confidence, edge position, or specified index). When the model only provides detection boxes, the detection boxes are expanded outward by a preset number of pixels in the four directions (up, down, left, and right) and then cropped to the image boundary and rasterized into a binary mask. The preset number of pixels is usually between 10 and 40 pixels. Before inputting the segmentation model, the image needs to be resized to the model input size and normalized. A confidence threshold is used to filter low-confidence candidates.

[0098] The depth map is back-projected using camera intrinsic parameters, retaining only the 3D points corresponding to pixels that simultaneously satisfy both "mask value is true" and "depth value is valid", thus obtaining the target point cloud. The back-projection uses the inverse of the camera intrinsic parameter matrix to convert pixel coordinates and depth values ​​into 3D spatial coordinates, and the joint filtering of mask and depth is performed pixel by pixel. "Depth value is valid" means that the depth is within the camera's range and is not zero, not NaN, or an invalid value. Invalid depths are commonly found in occluded areas, highly reflective surfaces, and distant areas beyond the camera's range.

[0099] In the target point cloud, all points are traversed along the optical axis, and the depth z_min of the nearest point is taken. Using z_min + Δh as the slicing threshold, the top surface thin layer point cloud is extracted. The top surface thin layer point cloud refers to the set of all three-dimensional points whose depth values ​​fall within the interval [z_min, z_min + Δh]. The fixed layer thickness Δh ranges from 5 mm to 100 mm. The selection of Δh is related to the opening width of the two-finger gripper and the curvature of the top surface of the object. If Δh is too small, top surface points may be missed, and if it is too large, side surface points may be included. The extraction of the top surface thin layer point cloud is based on the premise that the camera is installed in a top-down or near-top-down view, and the angle between its optical axis and the direction of gravity is less than a preset angle threshold.

[0100] The three-dimensional arithmetic mean of all points in the thin point cloud on the top surface is used to obtain the capture center point (camera coordinate system).

[0101] Projecting the thin point cloud layer on the top surface onto the horizontal plane, constructing a two-dimensional covariance matrix and performing eigenvalue decomposition, taking the eigenvector corresponding to the largest eigenvalue, rotating it by 90°, and normalizing it to [0°, 180°) by taking the modulus of 180°, yields the gripper opening and closing direction angle. The projection operation is achieved by retaining the horizontal coordinates of each point when the camera is mounted from above. The eigenvector corresponding to the largest eigenvalue reflects the direction in which the point cloud is distributed longest on the horizontal plane. After rotating it by 90°, this becomes the opening and closing direction that the gripper should take. Since the two grippers completely overlap geometrically after rotating 180° around the vertical axis, taking the modulus of 180° can compress the direction angle to the smallest independent representation interval.

[0102] The gripping center point is transformed to the base coordinate system using a pre-calibrated homogeneous transformation matrix from the camera to the robotic arm base, and the opening and closing direction angle of the gripper is transformed to the base coordinate system according to the rotation component of the homogeneous transformation matrix. When there is an installation angle deviation between the camera optical axis and the vertical axis of the robotic arm base, the opening and closing direction angle of the gripper needs to be compensated by superimposing the projection angle of the rotation component on the horizontal plane.

[0103] The robotic arm executes a sequence of grasping actions based on the grasping center point, the gripper opening and closing angle, and preset approach / grasp / lift / placement offset parameters. The approach position is a preset approach height offset vertically above the grasping center point. The gripper opens to a preset width before moving to the approach position and then descends vertically to the grasping position to avoid collisions with adjacent objects during lateral insertion. The grasping position is the grasping center point combined with the gripper opening and closing angle; after the end effector reaches its position, the gripper closes to complete the grasping action. The lifting position is a preset lifting height offset vertically after grasping, allowing the object to detach from the worktable. The placement position is a preset target position combined with the release action.

[0104] Typical single-cycle grasping process: After the system powers on, it completes camera initialization and internal parameter loading, as well as loading the transformation matrix from camera to robotic arm base. The operator inputs text prompts through configuration files or the interactive interface. After the main loop starts, each loop first acquires a time-synchronized color image and depth image. After multimodal segmentation, candidate selection, logical AND gated back projection, top surface thin-layer slicing, 3D mean value calculation to find the grasping center, horizontal PCA calculation to find the gripper opening and closing direction angle, and coordinate transformation to the base system, it sequentially sends motion commands to the robotic arm, such as gripper opening, moving to the approach pose, moving to the grasping pose, gripper closing, lifting, transporting, moving to the placement pose, and gripper opening. Each motion command waits for the robotic arm to confirm before sending the next one. After placement is completed, it enters the next loop or waits for a new text prompt.

[0105] When the text prompt is set to "object", the system picks up scattered workpieces on the workbench sequentially; when a specific category (such as "screw") is used, it sorts out specific parts from the mixed workpieces. Changing the text prompt switches the sorting object, without requiring additional training of the multimodal segmentation model.

[0106] Boundary condition handling: If the mask obtained in the step is empty or the area is less than the preset threshold, it indicates that the target is not in the field of view or the segmentation has failed. In this case, the current frame is skipped and the image is reacquired. If the number of points in the top surface thin layer point cloud obtained in the step is less than the preset lower limit, it indicates that the object is too small, too far away, or the depth map quality is poor. In this case, the current target is abandoned and an alarm is triggered. If the ratio of the largest feature value to the second largest feature value in the step is less than the preset threshold, it indicates that the top surface of the object is approximately isotropic or the point cloud noise is too large. In this case, the gripper opening and closing direction angle adopts the preset default direction or is manually specified by the operator.

[0107] Figure 3 The diagram shows the overall flowchart of the grasping method in this embodiment. In the diagram, the three input nodes at the top are text prompts, a color image, a depth map, and camera intrinsic parameters, respectively. The text prompts and color image are fed into a multimodal segmentation model to obtain the target's bounding box or binary mask. After candidate selection and expansion, the mask is rasterized into a binary mask. The mask and depth validity are jointly gated and back-projected using a logical AND method to convert valid pixels in the depth map into a target 3D point cloud. The top surface thin-layer slicing module extracts the top surface thin-layer point cloud from the target point cloud using the nearest point depth plus a fixed layer thickness as a threshold. The top surface thin-layer point cloud is processed through two parallel branches to calculate the grasping center point (3D arithmetic mean) and the gripper opening / closing direction angle (90° rotation of the PCA principal axis after horizontal projection). The results from the two branches are transformed to the robotic arm base coordinate system by the hand-eye calibration matrix, driving the robotic arm to sequentially execute the approach, grasp, lift, and place action sequence. The depth map and camera intrinsic parameters are directly input to the gated back-projection node from the right side, without going through the segmentation and masking processing path.

[0108] Figure 4 The diagram shows a geometrical representation of the top surface thin-layer slice and its principal axis, consisting of a side view (a) and a top view (b). In the side view, the vertical axis is the camera optical axis direction Z, and the horizontal axis is the X direction; ● represents the point z_min with the minimum depth in the top surface thin layer; the light-colored area represents the top surface thin-layer point cloud that participates in subsequent calculations, located in the depth interval [z_min, z_min + Δh]; the dark-colored area represents the object side points excluded by the top surface slice threshold; the desktop point cloud is excluded by semantic masking and does not participate in back projection. In the top view, the elliptical area represents the projection of the top surface thin-layer point cloud onto the horizontal plane; ★ represents the grab center point, which is the projection position of the three-dimensional arithmetic mean of all top surface points onto the horizontal plane; the solid arrow indicates the principal axis direction of the horizontal distribution of the point cloud (PCA maximum eigenvector); the dashed arrow indicates the gripper opening and closing direction, obtained by rotating the principal axis direction by 90°.

[0109] Example 3:

[0110] Please see Figure 5 The diagram shows a structural schematic of a computer device provided in an embodiment of this application. An embodiment of this application provides a computer device 400, including a processor 410 and a memory 420. The memory 420 stores a computer program executable by the processor 410. When the computer program is executed by the processor 410, it performs the method described above.

[0111] This application embodiment also provides a storage medium 430, on which a computer program is stored, and the computer program is executed by a processor 410 to perform the above method.

[0112] The storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0113] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0114] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0115] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0116] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0118] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0119] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0120] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A robotic arm grasping method based on semantic point cloud top surface principal axis analysis, characterized in that, Includes the following steps: Acquire time-synchronized color images, depth maps, and camera intrinsic parameters; input text prompts; and generate a binary mask of the target from the color images and text prompts using a multimodal segmentation model. The depth map is back-projected using the binary mask and depth validity as joint filtering conditions, and only the 3D points corresponding to pixels that simultaneously satisfy the mask being true and the depth value being valid are retained to obtain the target point cloud. Traverse all points in the target point cloud along the camera optical axis direction corresponding to the depth map, and extract the top surface thin layer point cloud using the depth value of the nearest point plus a fixed layer thickness as the slicing threshold. The top surface thin layer point cloud is a set of all three-dimensional points whose depth values ​​fall within the range of the depth value of the nearest point to the depth value of the nearest point plus a fixed layer thickness. The three-dimensional arithmetic mean of all points in the thin point cloud on the top surface is used to obtain the grab center point in the camera coordinate system. After projecting the thin point cloud on the top surface onto the horizontal plane, a two-dimensional covariance matrix is ​​constructed and eigenvalue decomposition is performed. The gripper opening and closing direction angle is obtained by rotating and normalizing the eigenvector corresponding to the largest eigenvalue. The gripping center point and the opening and closing direction angle of the gripper are transformed to the coordinate system of the robotic arm base according to the pre-calibrated homogeneous transformation matrix from the camera to the robotic arm base, thereby driving the robotic arm to perform a sequence of actions including approaching, gripping, lifting, and placing.

2. The method according to claim 1, characterized in that, The multimodal segmentation model is sized and normalized before the color image is input; When the multimodal segmentation model outputs a detection box, it expands the detection box outward by a preset number of pixels in the four directions of up, down, left, and right, then crops it to the image boundary and rasterizes it into a binary mask. The preset number of pixels ranges from 10 to 40 pixels.

3. The method according to claim 1, characterized in that, The depth value is valid if it is within the camera's range and is not zero, or is not NaN or an invalid marker value. The back projection uses the inverse of the camera intrinsic parameter matrix to convert pixel coordinates and depth values ​​into three-dimensional spatial coordinates, and the joint filtering of mask and depth is performed pixel by pixel.

4. The method according to claim 1, characterized in that, The fixed layer thickness ranges from 5 mm to 100 mm, and the fixed layer thickness is related to the opening width of the two-finger gripper and the curvature of the top surface of the object. The extraction of the thin point cloud on the top surface is based on the premise that the angle between the camera's optical axis and the direction of gravity is less than a preset angle threshold, and the installation is either top-down or near-top-down.

5. The method according to claim 1, characterized in that, The opening and closing direction angle of the gripper is determined by the following steps: The horizontal coordinates of each point are obtained by projecting the thin point cloud on the top surface onto a horizontal plane. Construct a two-dimensional covariance matrix and perform eigenvalue decomposition; Rotate the eigenvector corresponding to the largest eigenvalue by 90°. The rotated orientation angle is normalized to the interval [0°, 180°) by taking the modulus of 180°.

6. The method according to claim 5, characterized in that, The gripping center point and the gripper opening / closing direction angle share the same top surface thin-layer point cloud. A single point cloud traversal completes the solution of both parameters, and the gripper opening / closing direction angle is obtained through closed-form interpretable principal component analysis.

7. The method according to claim 1, characterized in that, The sequence of actions performed by the driven robotic arm, including approaching, grasping, lifting, and placing, includes: The robotic arm moves to a proximity pose that is offset vertically by a preset proximity height directly above the grasping center point, and the gripper has opened to a preset width before moving to the proximity pose; It then descends vertically to the gripping center point and, in accordance with the gripping posture of the opening and closing direction angle of the gripper, performs gripper closure to complete the gripping; After clamping, the object is lifted vertically upwards to a preset lifting height, thus detaching it from the worktable. Move to the preset target position and perform the gripper opening and placement action.

8. The method according to claim 1, characterized in that, The method further includes: When the binary mask is empty or its area is less than a preset threshold, skip the current frame and re-acquire the image; When the number of points in the top surface thin-layer point cloud is less than a preset lower limit, the current target is abandoned and an alarm is triggered; When the ratio of the largest eigenvalue to the second largest eigenvalue of the two-dimensional covariance matrix is ​​less than a preset threshold, the opening and closing direction angle of the gripper adopts a preset default direction or is manually specified by the operator.

9. The method according to claim 1, characterized in that, When the text prompt is set to "object", the scattered workpieces on the workbench are picked up sequentially. When the text prompt is set to a specific category, specific parts are sorted from mixed workpieces; Changing the text prompt switches the sorting object.

10. A robotic arm grasping system based on semantic point cloud top surface principal axis analysis, characterized in that, Using the method of any one of claims 1 to 9, the system comprises: The camera is used to acquire time-synchronized color images, depth images, and camera intrinsic parameters; An input unit is used to input text prompts, and a binary mask of the target is generated from the color image and the text prompts using a multimodal segmentation model; The target point cloud unit is used to back-project the depth map using the binary mask and depth validity as joint filtering conditions, and retain only the three-dimensional points corresponding to pixels that simultaneously satisfy the mask being true and the depth value being valid, to obtain the target point cloud. The top surface thin-layer point cloud extraction unit is used to traverse all points in the target point cloud along the camera optical axis direction corresponding to the depth map, and extract the top surface thin-layer point cloud by using the depth value of the nearest point plus a fixed layer thickness as the slicing threshold. The top surface thin-layer point cloud is a set of all three-dimensional points whose depth values ​​fall within the range of the depth value of the nearest point to the depth value of the nearest point plus a fixed layer thickness. The grasping calculation unit is used to calculate the three-dimensional arithmetic mean of all points in the top surface thin point cloud to obtain the grasping center point in the camera coordinate system. After projecting the top surface thin point cloud onto the horizontal plane, a two-dimensional covariance matrix is ​​constructed and eigenvalue decomposition is performed. The gripper opening and closing direction angle is obtained by rotating and normalizing the eigenvector direction corresponding to the largest eigenvalue. The robotic arm and drive unit are used to transform the gripping center point and the opening and closing direction angle of the gripper to the coordinate system of the robotic arm base according to the pre-calibrated homogeneous transformation matrix from the camera to the robotic arm base, and drive the robotic arm to perform a sequence of approaching, gripping, lifting and placing actions.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-9.

12. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images