Robot grabbing posture generation method and related device

Abstract

The invention discloses a robot grabbing posture generation method and a related device, and the method comprises the steps: obtaining an RGB image, a depth image and a position coordinate of a target object, and generating a spatial calibration matrix through data preprocessing; target object segmentation is carried out on the calibrated RGB matrix according to the target point coordinate sequence, a segmentation mask is generated, and object mass center coordinates are calculated; generating a three-dimensional point cloud by using the calibrated depth matrix, the segmentation mask and the camera parameters, extracting a plane point set and a non-plane point set through an RANSAC algorithm, and analyzing to obtain an axial feature vector and a plane normal vector; finally, grabbing parameters are calculated according to the feature vectors and the centroid coordinates, and a grabbing posture transformation matrix is generated through vector operation. According to the technical scheme, under the condition of not depending on a preset model library, the grabbing posture of an unknown object is generated by analyzing the geometrical characteristics of the object, the accuracy problem in the process of converting two-dimensional image information into three-dimensional grabbing parameters is solved, and the adaptability and reliability of a robot grabbing task are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot vision perception and motion control, and particularly to a method for generating a robot grasping posture and related devices. Background Art

[0002] Robot grasping posture generation refers to the process of automatically calculating and planning the optimal posture parameters when a robot executes a grasping task through algorithms. In scenarios such as industrial production, warehousing logistics, and crop collection, the robot needs to accurately grasp the position and posture information of the target object, and combine the geometric features and physical properties of the object to generate a suitable grasping strategy. This process involves multiple links such as object recognition, spatial positioning, and motion planning, and it is necessary to convert two-dimensional image information into three-dimensional grasping parameters, and ensure that the generated grasping posture can not only ensure the stability of grasping, but also meet the kinematic and dynamic constraints of the robot.

[0003] Currently, the mainstream methods for generating robot grasping postures mainly include template matching-based methods, heuristic rule-based methods, and deep learning-based methods. The template matching-based method needs to pre-establish a large three-dimensional model library of standard objects, and determines the grasping posture by matching the object to be grasped with the samples in the model library. This method has poor adaptability to unknown objects, and maintaining the model library requires a large amount of human resources. The heuristic rule-based method generates the grasping posture through preset geometric features and judgment rules. Although the calculation efficiency is relatively high, due to relying on simplified assumptions about the shape of the object, it is difficult to handle objects with complex shapes. The deep learning-based method has strong environmental adaptability, but it requires a large amount of labeled data for training, and the prediction results lack interpretability, and the system reliability is difficult to guarantee. More importantly, these methods often have the problem of the disconnection between the perception and execution links in practical applications. Especially when dealing with unknown objects, it is difficult to establish an accurate mapping relationship from two-dimensional image information to three-dimensional grasping parameters. Therefore, how to achieve accurate grasping posture generation for unknown objects without relying on a preset model library, and at the same time solve the accuracy problem in the process of converting two-dimensional image information into three-dimensional grasping parameters has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main object of the present invention is to solve the technical problem that the existing methods for generating robot grasping postures have a disconnection between the perception and execution links, and it is difficult to establish an accurate mapping relationship from two-dimensional image information to three-dimensional grasping parameters.

[0005] The first aspect of the present invention provides a method for generating a robot grasping posture, and the method for generating a robot grasping posture includes: Obtain the RGB image, depth image and position coordinates of the target object, convert the position coordinates into a sequence of target point coordinates, convert the RGB image into an original RGB matrix, convert the depth image into an original depth matrix, and perform spatial alignment processing on the original RGB matrix and the original depth matrix to generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix; Segment the region of the target object from the spatially calibrated RGB matrix according to the sequence of target point coordinates to generate a target object segmentation mask, and calculate the centroid coordinates of the target object in the segmentation mask; Generate a three-dimensional point cloud of the target object based on the spatially calibrated depth matrix, the target object segmentation mask and the camera intrinsic matrix, use the RANSAC algorithm to extract a planar point set and a non-planar point set from the three-dimensional point cloud, perform principal direction analysis on the non-planar point set, extract the axial feature vector of the target object, and calculate the plane normal vector of the planar point set; Calculate the grasping parameters of the target object based on the plane normal vector, the axial feature vector and the centroid coordinates of the object, determine the plane normal vector as the grasping approach vector, determine the axial feature vector as the gripper opening and closing vector, obtain the grasping attitude auxiliary vector through the cross product operation of the grasping approach vector and the gripper opening and closing vector, combine the grasping approach vector, the gripper opening and closing vector and the grasping attitude auxiliary vector to generate a grasping rotation matrix, and combine with the centroid coordinates of the object to generate a grasping attitude transformation matrix.

[0006] Preferably, the obtaining the RGB image, depth image and position coordinates of the target object, converting the position coordinates into a sequence of target point coordinates, converting the RGB image into an original RGB matrix, converting the depth image into an original depth matrix, and performing spatial alignment processing on the original RGB matrix and the original depth matrix to generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix includes: Receive the RGB image, depth image and position coordinates of the target object, verify the formats of the RGB image and the depth image to generate RGB image format parameters and depth image format parameters; Perform data structure conversion and normalization processing on the RGB image according to the RGB image format parameters to generate an original RGB matrix, and perform data structure conversion and numerical standardization processing on the depth image according to the depth image format parameters to generate an original depth matrix; Parse and convert the position coordinates according to a preset coordinate format rule to generate a sequence of target point coordinates; Calculate the spatial coordinates of the pixel points in the original RGB matrix and the original depth matrix in the camera coordinate system respectively according to the camera intrinsic matrix to generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix.

[0007] Preferably, the method for segmenting the target object area from the spatially calibrated RGB matrix according to the target point coordinate sequence to generate a target object segmentation mask and calculating the centroid coordinates of the target object segmentation mask includes: Mapping the coordinate points in the target point coordinate sequence to the spatially calibrated RGB matrix to generate an initial target area mask; Performing region growing on the initial target area mask to generate a region expansion mask; Performing boundary detection on the spatially calibrated RGB matrix according to the region expansion mask to generate a target object segmentation mask; Calculating the centroid of the target object segmentation mask based on the distribution of pixel points in the target object segmentation mask to generate centroid coordinates of the object.

[0008] Preferably, the method for generating a three-dimensional point cloud of the target object according to the spatially calibrated depth matrix, the target object segmentation mask, and the camera intrinsic matrix, using the RANSAC algorithm to extract a planar point set and a non-planar point set from the three-dimensional point cloud, performing principal direction analysis on the non-planar point set to extract an axial feature vector of the target object, and calculating the plane normal vector of the planar point set includes: According to the camera intrinsic matrix, performing a projection transformation on the depth values in the spatially calibrated depth matrix and the pixel point coordinates corresponding to the target object segmentation mask to generate a set of projection coordinate points, and performing a conversion from the camera coordinate system to the world coordinate system on the set of projection coordinate points to generate a three-dimensional point cloud of the target object; Using the RANSAC algorithm for plane fitting and dividing the three-dimensional point cloud into a planar point set and a non-planar point set according to the fitting residuals; Performing principal component analysis on the non-planar point set to extract a principal direction component, and normalizing the principal direction component to generate an axial feature vector; Performing singular value decomposition on the planar point set, taking the singular vector corresponding to the variance value in the decomposition result, and normalizing the singular vector to generate a plane normal vector.

[0009] Preferably, the method for using the RANSAC algorithm for plane fitting and dividing the three-dimensional point cloud into a planar point set and a non-planar point set according to the fitting residuals includes: Dividing the three-dimensional point cloud into a feasible grasping area, removing the point cloud data with a distance from the camera exceeding the reach of the robot's grasping arm to generate valid grasping point cloud; Calculating the local curvature of the valid grasping point cloud according to a curvature threshold, and dividing the point cloud with a curvature value lower than the curvature threshold into a candidate planar point set; Perform RANSAC iterative calculation on the candidate plane point set, extract the plane equation parameters, and calculate the distance deviation value of each point from the plane; Perform plane support analysis on the candidate plane point set according to the distance deviation value, judge whether the plane area meets the gripper base support requirements, and determine the point set that meets the support requirements as the plane point set; Perform clustering processing on the point cloud data that does not belong to the plane point set, and generate a non-plane point set after removing outliers.

[0010] Preferably, calculating the grasping parameters of the target object according to the plane normal vector, the axial feature vector and the object centroid coordinates, determining the plane normal vector as the grasping approach vector, determining the axial feature vector as the gripper opening and closing vector, obtaining the grasping attitude auxiliary vector through the cross product operation of the grasping approach vector and the gripper opening and closing vector, and combining the grasping approach vector, the gripper opening and closing vector and the grasping attitude auxiliary vector to generate a grasping rotation matrix, and generating a grasping attitude transformation matrix in combination with the object centroid coordinates, including: Judge the direction consistency of the plane normal vector, perform reverse processing on the plane normal vector according to the judgment result, and perform unitization operation on the processed plane normal vector to obtain the grasping approach vector; Perform numerical normalization processing on the axial feature vector, perform orthogonal correction on the normalized axial feature vector, and perform unitization operation on the corrected vector to obtain the gripper opening and closing vector; Perform cross product operation on the grasping approach vector and the gripper opening and closing vector, verify the orthogonality of the cross product result, and perform unitization processing on the verified vector to obtain the grasping attitude auxiliary vector; Combine the grasping approach vector as the first column vector, the gripper opening and closing vector as the second column vector, and the grasping attitude auxiliary vector as the third column vector to generate a grasping rotation matrix; Combine the grasping rotation matrix and the object centroid coordinates in the form of a homogeneous transformation matrix, calculate the rigid body transformation parameters of translation and rotation, and generate a grasping attitude transformation matrix.

[0011] Preferably, combining the grasping rotation matrix and the object centroid coordinates in the form of a homogeneous transformation matrix, calculating the rigid body transformation parameters of translation and rotation, and generating a grasping attitude transformation matrix, including: Convert the grasping rotation matrix into Euler angle representation, perform reachability analysis according to the robot joint motion constraints, and screen out the Euler angle parameters that meet the joint limit requirements; Perform gravity compensation calculation on the object centroid coordinates, predict the gravity influence during the grasping process, and generate the compensated object centroid coordinates; Generate a set of candidate grasping postures based on the Euler angle parameters and the compensated centroid coordinates of the object, and calculate the robot joint angle solutions for each candidate posture; Perform collision detection and analysis on the set of candidate grasping postures, and eliminate the grasping postures that may cause interference between the robot and the environment; Comprehensively score the remaining candidate grasping postures according to the joint movement distance and the posture change amplitude, and select the Euler angle parameters and position parameters corresponding to the posture with the optimal score; Convert the Euler angle parameters back to the form of a rotation matrix, and combine them with the position parameters to generate a grasping posture transformation matrix.

[0012] The second aspect of the present invention provides a robot grasping posture generation device, and the robot grasping posture generation device includes: A data preprocessing module, configured to obtain the RGB image, depth image and position coordinates of the target object, convert the position coordinates into a target point coordinate sequence, convert the RGB image into an original RGB matrix, convert the depth image into an original depth matrix, perform spatial alignment processing on the original RGB matrix and the original depth matrix, and generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix; An object segmentation module, configured to segment the target object region of the spatially calibrated RGB matrix according to the target point coordinate sequence, generate a target object segmentation mask, and calculate the centroid coordinates of the target object in the target object segmentation mask; A feature extraction module, configured to generate a three-dimensional point cloud of the target object according to the spatially calibrated depth matrix, the target object segmentation mask and the camera intrinsic matrix, extract a plane point set and a non-plane point set from the three-dimensional point cloud by using the RANSAC algorithm, perform principal direction analysis on the non-plane point set, extract the axial feature vector of the target object, and calculate the plane normal vector of the plane point set; A posture generation module, configured to calculate the grasping parameters of the target object according to the plane normal vector, the axial feature vector and the centroid coordinates of the object, determine the plane normal vector as the grasping approach vector, determine the axial feature vector as the gripper opening and closing vector, obtain a grasping posture auxiliary vector through the cross product operation of the grasping approach vector and the gripper opening and closing vector, combine the grasping approach vector, the gripper opening and closing vector and the grasping posture auxiliary vector to generate a grasping rotation matrix, and combine the centroid coordinates of the object to generate a grasping posture transformation matrix.

[0013] The third aspect of the present invention provides a robot grasping posture generation device, including: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected through a line; the at least one processor invokes the instructions in the memory to enable the robot grasping posture generation device to execute the steps of the above-mentioned robot grasping posture generation method.

[0014] The fourth aspect of the present invention provides a computer-readable storage medium, in which instructions are stored. When it runs on a computer, it enables the computer to execute the steps of the above-mentioned robot grasping posture generation method.

[0015] The technical solution provided by the embodiments of the present application realizes the precise grasping of unknown objects by constructing a complete processing link from perception to grasping. This method systematically processes the acquired RGB images, depth images and position coordinates, and ensures the spatial consistency of different sensing data through spatial alignment, laying a foundation for subsequent feature extraction.

[0016] In the target object region segmentation stage, this method segments the spatially calibrated RGB matrix based on the target point coordinate sequence, generates a target object segmentation mask and calculates the centroid coordinates of the object. This image segmentation-based processing method does not require a pre-established object model library, overcoming the defect of poor adaptability of traditional template matching methods to unknown objects. Through the generation of the segmentation mask, the system obtains the precise contour information of the target object, providing a reliable region definition for subsequent three-dimensional feature extraction.

[0017] In the three-dimensional feature extraction stage, this method generates a three-dimensional point cloud of the target object based on the spatially calibrated depth matrix, the target object segmentation mask and the camera intrinsic matrix. The plane point set and the non-plane point set are extracted from the point cloud through the RANSAC algorithm, and the main direction analysis is performed on the non-plane point set to obtain the axial feature vector and the plane normal vector. This geometric feature analysis method abandons the simplified assumptions of the traditional heuristic rules about the object shape and can accurately describe the spatial features of objects with complex shapes.

[0018] In the grasping posture generation stage, this method determines the plane normal vector as the grasping approach vector, determines the axial feature vector as the gripper opening and closing vector, and obtains the grasping posture auxiliary vector through cross product operation. These three orthogonal vectors are combined to form a grasping rotation matrix, and the grasping posture transformation matrix is generated in combination with the centroid coordinates of the object. This geometric feature-based posture generation method establishes a direct mapping relationship from two-dimensional images to three-dimensional grasping parameters, avoiding the problem that the prediction results in deep learning methods are difficult to interpret.

[0019] Through the above processing steps, without relying on a preset model, the method establishes an accurate conversion mechanism from two-dimensional image information to three-dimensional grasping parameters through a systematic analysis of the geometric features of the object, solving the problem of the disconnection between the perception and execution links in the robot grasping process. The method has good generality and can adapt to unknown objects of different shapes, ensuring the accuracy and reliability of the grasping posture generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on the structures shown in these drawings.

[0021] Figure 1 It is a schematic diagram of an embodiment of the method for generating a robot grasping posture in an embodiment of the present invention; Figure 2 It is a schematic diagram of an embodiment of the device for generating a robot grasping posture in an embodiment of the present invention; Figure 3 It is a schematic diagram of an embodiment of the device for generating a robot grasping posture in an embodiment of the present invention.

[0022] The realization, functional features, and advantages of the objectives of the present invention will be further described in conjunction with the embodiments with reference to the drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0024] It should be noted that if there are directional indications (such as up, down, left, right, front, back...) involved in the embodiments of the present invention, the directional indications are only used to explain the relative positional relationship and movement conditions between the components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will also change accordingly.

[0025] In addition, the descriptions involving "first", "second", etc. in the present invention are for descriptive purposes only, and should not be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, "and / or" throughout the text includes three scenarios. Taking A and / or B as an example, it includes technical solution A, technical solution B, and the technical solution where both A and B are satisfied simultaneously. In addition, the technical solutions between various embodiments can be combined with each other, which must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or inability to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0026] An embodiment of the present application provides a method for generating a robot grasping posture. Figure 1 It is a flowchart of the method for generating a robot grasping posture provided by an embodiment of the present application. In this embodiment, the method includes: Please refer to Figure 1 , obtain the RGB image, depth image and position coordinates of the target object, convert the position coordinates into a target point coordinate sequence, convert the RGB image into an original RGB matrix, convert the depth image into an original depth matrix, perform spatial alignment processing on the original RGB matrix and the original depth matrix, and generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix; In an embodiment of the present invention, the obtaining the RGB image, depth image and position coordinates of the target object, converting the position coordinates into a target point coordinate sequence, converting the RGB image into an original RGB matrix, converting the depth image into an original depth matrix, and performing spatial alignment processing on the original RGB matrix and the original depth matrix to generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix includes: Receive the RGB image, depth image and position coordinates of the target object, perform format verification on the RGB image and the depth image, and generate RGB image format parameters and depth image format parameters; According to the RGB image format parameters, perform data structure conversion and normalization processing on the RGB image to generate an original RGB matrix, and according to the depth image format parameters, perform data structure conversion and numerical standardization processing on the depth image to generate an original depth matrix; Parse and convert the position coordinates according to a preset coordinate format rule to generate a target point coordinate sequence; Calculate the spatial coordinates of the pixel points in the original RGB matrix and the original depth matrix in the camera coordinate system respectively according to the camera internal parameter matrix, and generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix.

[0027] The following is a detailed description of the steps involved in the above embodiment: The specific implementation method of receiving the RGB image, depth image and position coordinates of the target object, format-verifying the RGB image and depth image, and generating the RGB image format parameters and depth image format parameters is as follows: first, use a standard image receiving device (such as an industrial camera or a robot's built-in visual sensor) to obtain the RGB color image and depth image of the target object. RGB images are usually three-channel images, containing information on the three color channels of red, green and blue; while depth images are single-channel images, and each pixel value represents the distance from the camera to the surface of the object. The position coordinates are the characteristic positions of the target object marked by the user through the human-computer interaction interface (such as a touch screen or a mouse click). After obtaining these three types of data, the system will perform a format verification process, which includes checking whether the RGB image is a valid color image (checking whether the number of channels is 3, whether the bit depth is correct, and whether the resolution meets the requirements), and verifying the validity of the depth image (checking whether it is a single-channel image and whether the depth value range is reasonable). After the verification is completed, the system generates RGB image format parameters (including image size, number of channels, encoding method, etc.) and depth image format parameters (including image size, depth unit, effective depth range, etc.). For example, for an RGB image with a resolution of 640×480, its format parameters include: size 640×480 pixels, number of channels 3, encoding method 8-bit unsigned integer; and the corresponding depth image format parameters may include: size 640×480 pixels, depth unit in millimeters, effective depth range 0.5 meters to 5 meters. This step ensures the validity and consistency of the input data and avoids subsequent processing errors caused by incorrect image format. The system's strict verification of format parameters is based on the stability requirements of the robot vision system, because inconsistent formats will directly affect the accuracy of spatial coordinate calculations, and thus affect the grasping accuracy.

[0028] The specific implementation of performing data structure conversion and normalization processing on the RGB image according to the RGB image format parameters to generate the original RGB matrix, and performing data structure conversion and numerical standardization processing on the depth image according to the depth image format parameters to generate the original depth matrix is as follows: For the RGB image, first, according to the previously determined format parameters, the image is converted from its original format (which may be a compressed format such as JPEG, PNG, etc.) to a standard three-dimensional array structure, that is, the original RGB matrix, with dimensions [height, width, 3], where 3 represents the three RGB channels. During this process, the system performs normalization processing, converting the pixel values of each channel from the original range (usually integers from 0 - 255) to floating-point numbers in the range of 0 - 1 for subsequent algorithm processing. For the depth image, similarly according to its format parameters, the depth image is converted to a two-dimensional array structure, that is, the original depth matrix, with dimensions [height, width]. The depth values usually need to be numerically standardized, converting the original depth values (which may be in millimeters or centimeters) to a standard unit (usually meters), and adjusting the range to ensure that the depth values are within the valid range. In addition, for invalid values in the depth image (such as areas not captured by the sensor, usually represented as 0 or extremely large values), the system performs special marking or filling processing to avoid subsequent calculation errors. For example, for a depth image with different depth units, the system uniformly converts it to depth values in meters; for an 8-bit RGB image, the system divides the pixel values from 0 - 255 by 255 to convert them to normalized values in the range of 0 - 1. Data structure conversion and normalization processing are standard preprocessing steps in computer vision. The purpose is to convert image data from different sources and formats into a unified format so that the algorithm can process this data consistently. This standardization processing is crucial for ensuring the accuracy of subsequent point cloud generation and feature extraction because even a tiny numerical deviation may be amplified into a significant position error in three-dimensional space.

[0029] The specific implementation of parsing and converting the position coordinates according to the preset coordinate format rules to generate a target point coordinate sequence is as follows: The position coordinates marked by the user are usually two-dimensional coordinates on the image plane (such as pixel coordinates on a display), and need to be parsed and converted according to the preset format rules. Specifically, the system first verifies whether the format of the position coordinates conforms to the preset rules, for example, checks whether the coordinate values are within the image range (0 ≤ x < width, 0 ≤ y < height), and whether the number of coordinate points meets the requirements (at least one point is required to indicate the target position). Then, the system converts the verified coordinate points into a target point coordinate sequence in a standard format, which is a two-dimensional array, where each row represents a coordinate point and contains two values, x and y. During the conversion process, the system may also need to adjust the coordinate values, for example, considering the interface scaling factor, cropping or scaling operations in image preprocessing, etc. For the case of multi-point marking, the system will maintain the order of the points, which may be important for subsequent region growing or contour extraction. For example, if the user marks three points on the display to indicate an object (such as clicking on the upper left corner, upper right corner, and center position of the object), the system will convert the screen coordinates of these points into the corresponding image coordinates and organize them into a 3×2 matrix, where each row contains the x and y coordinates of a point. The accuracy of coordinate conversion is crucial for subsequent object segmentation, because even a small coordinate deviation may lead to segmentation errors, especially for objects with fine edges or complex textures. The preset coordinate format rules fully consider the convenience of human-computer interaction and the requirements of subsequent algorithm processing, and are the key bridge connecting the user's intention and machine processing.

[0030] According to the camera intrinsic matrix, calculate the spatial coordinates of the pixel points in the original RGB matrix and the original depth matrix in the camera coordinate system. The specific implementation of generating the spatially calibrated RGB matrix and the spatially calibrated depth matrix is as follows: The camera intrinsic matrix is a 3×3 matrix that describes the optical characteristics of the camera and contains parameters such as focal length, principal point coordinates, and pixel size. Using these parameters, the system can convert the two-dimensional coordinates on the image plane into three-dimensional coordinates in the camera coordinate system. Specifically, first obtain the accurate intrinsic matrix from the camera calibration process, which is usually obtained through professional camera calibration tools (such as the checkerboard calibration method). Then, for the original RGB matrix, the system calculates the direction vector of each pixel point in the camera coordinate system, that is, the direction of the ray starting from the camera optical center and passing through this pixel point. This step does not involve depth information and only establishes the mapping relationship between pixels and spatial directions. For the original depth matrix, the system combines the depth value of each pixel with the corresponding direction vector to calculate the complete three-dimensional coordinates of this point in the camera coordinate system. During the calculation process, the system uses pixel coordinates, depth values, and camera intrinsic parameters (such as principal point coordinates and focal length) for three-dimensional coordinate transformation. Each pixel point in the depth image corresponds to a three-dimensional point in the real world, and its spatial position is jointly determined by the position of the pixel in the image and the corresponding depth value. For example, for a pixel point at position (100, 200) with RGB values (0.5, 0.6, 0.7) in the original RGB matrix, if its depth value in the original depth matrix is 1.5 meters, then after spatial calibration, the accurate three-dimensional coordinates of this point in the camera coordinate system can be obtained while keeping its RGB values unchanged. This spatial calibration based on the camera intrinsic matrix is a fundamental step in 3D reconstruction in computer vision, ensuring the accurate mapping of two-dimensional image information to three-dimensional space. The accuracy of spatial calibration directly affects the accuracy of subsequent point cloud generation and feature extraction. Therefore, in practical applications, the camera calibration process usually needs to be performed periodically to compensate for possible camera parameter drift. The importance of this step lies in that it solves the key problem of the conversion from two-dimensional images to three-dimensional space and lays an accurate data foundation for subsequent object geometric feature extraction and grasping pose calculation.

[0031] Please continue to refer to Figure 1 , segment the target object area from the spatially calibrated RGB matrix according to the target point coordinate sequence to generate a target object segmentation mask, and calculate the centroid coordinates of the target object segmentation mask; In an embodiment of the present invention, the segmenting the target object area from the spatially calibrated RGB matrix according to the target point coordinate sequence to generate a target object segmentation mask and calculating the centroid coordinates of the target object segmentation mask includes: Map the coordinate points in the target point coordinate sequence to the spatially calibrated RGB matrix to generate an initial target area mask; Perform region growing on the initial mask of the target region to generate a region expansion mask; Perform boundary detection on the spatially calibrated RGB matrix according to the region expansion mask to generate a target object segmentation mask; Calculate the centroid of the target object segmentation mask based on the distribution of pixel points in the target object segmentation mask to generate the object centroid coordinates.

[0032] The following is a specific description of the steps involved in the above embodiments: The specific implementation of mapping the coordinate points in the target point coordinate sequence to the spatially calibrated RGB matrix to generate the initial mask of the target region is as follows: The system first creates a binary mask matrix with the same size as the spatially calibrated RGB matrix, and all elements are initialized to 0 (representing the background). Then, each coordinate point in the target point coordinate sequence marked by the user is mapped to this mask matrix, and the value at the corresponding position is set to 1 (representing the foreground). Specifically, for each coordinate point (x, y) in the target point coordinate sequence, the system sets the value at position (x, y) in the mask matrix to 1, indicating that this point is marked as part of the target region. To enhance the marking effect, the system also sets a small neighborhood (usually a 3×3 or 5×5 pixel window) around each marked point, and marks all pixels within this neighborhood as the foreground, thus forming an initial seed region, rather than just a single pixel point. For example, if the user marks three points on the object with coordinates (100, 150), (120, 170), and (140, 160) respectively, the system will set a region with a value of 1 at these three positions in the mask matrix and around each of them, forming an initial mark of the target region. These initial marked points serve as the seed points for subsequent region growing, and their accuracy directly affects the quality of the segmentation result. Through a small amount of user interaction marking, combined with the human's intuitive recognition ability of the target object, it provides the initial clues for the algorithm segmentation, avoiding the problem of incorrect target selection that may exist in fully automatic segmentation algorithms. At the same time, the method of expanding a single-point mark into a small region takes into account the accuracy limitation of user marking and the possible subtle texture differences in the image, making the initial seed region more stable and reliable. This user-interaction-based initialization method is particularly suitable for the precise positioning of specific target objects in grasping tasks, and compared with fully automatic segmentation methods, it greatly reduces the risk of mis-segmentation.

[0033] The specific implementation method of performing region growing on the initial mask of the target region to generate a region expansion mask is as follows: Region growing is a segmentation method that starts from an initial seed point and gradually incorporates surrounding pixels that meet specific conditions into the target region. The system adopts an iterative expansion strategy, starting from the foreground pixels (pixels with a value of 1) in the initial mask of the target region as seed points, and examines the neighboring pixels of these seed points. For the neighboring pixels of each seed point, the system calculates their color similarity or texture similarity with the seed point in the spatially calibrated RGB matrix. Similarity calculation is usually based on the Euclidean distance or cosine similarity in the RGB color space. If the color difference between two points is less than a preset threshold (the empirical value is usually set between 15 - 30, depending on the color variation degree of the image), then the neighboring pixel is considered to belong to the same object and is added to the target region. In addition, the system also combines depth information to check whether the depth value of the neighboring pixel is close to that of the seed point to avoid incorrect expansion across object boundaries. The region growing process continues, and the newly added pixels will become the new seed points for the next iteration until no new pixels can be added. For example, in a segmentation scenario of toy building blocks, if the initial mask marks several points in the center of the building block, the region growing process will gradually expand to the entire building block region according to color and depth information, but will stop at the edge of the building block because the color or depth changes significantly at the edge. This region growing strategy particularly considers the continuity and local similarity of the actual object surface characteristics. The setting of the similarity threshold should not only ensure that it can adapt to the object surface texture and illumination changes, but also avoid incorrect expansion to the background or other objects. In complex scenarios, the similarity metric will also adopt an adaptive threshold, dynamically adjusted according to the color variance of the local region to adapt to the surface characteristics of different objects and improve the robustness and accuracy of segmentation. After the region growing process is completed, the system generates a region expansion mask, which contains the entire target object region expanded from the initial seed region.

[0034] The specific implementation method of generating the target object segmentation mask by performing boundary detection on the spatially calibrated RGB matrix according to the region expansion mask is as follows: The region expansion mask provides the approximate region of the target object, but there may be problems such as uneven edges or local leakage. To obtain a more accurate object boundary, the system performs boundary detection and optimization processing on the region expansion mask and the spatially calibrated RGB matrix. First, the system applies edge detection operators (such as Sobel, Canny, or Laplacian operators) to process the spatially calibrated RGB matrix, extracts the edge information in the image, and forms an edge map. Then, the system combines and analyzes the edge map with the region expansion mask to determine whether the edges of the region expansion mask are consistent with the edge features of the actual image. For inconsistent regions, the system makes local adjustments based on the image gradient information to make the segmentation boundary more conform to the actual object edge. In addition, the system also applies morphological processing (such as opening operation, closing operation) to optimize the segmentation mask, removes small holes or noise, and smooths the boundary contour. During the processing, the system considers the continuity and local structure characteristics of the object boundary, and usually uses a morphological processing kernel size of 5×5 or 7×7. This size range is sufficient to smooth the edge noise without over-blurring important structural details. For example, when segmenting a cup, due to the possible depth and color changes between the cup handle and the cup body, region growing may be difficult to completely cover the entire cup. The boundary detection step will identify the complete cup contour by analyzing the edge features in the RGB image and correct the segmentation result to ensure that the handle and the cup body are accurately included in the segmentation mask. After the boundary detection is completed, the system generates the final target object segmentation mask, which accurately describes the position and shape of the target object in the image. The selection of various detection operators and morphological processing parameters in the boundary detection step is based on the saliency of the object edge and the local structure complexity. While ensuring the segmentation accuracy, it also guarantees the smoothness and continuity of the boundary, which is very important for subsequent point cloud generation and geometric feature extraction. This boundary optimization strategy effectively solves the over-segmentation or under-segmentation problems that may occur in the region growing method in complex texture regions, and improves the accuracy and robustness of object segmentation.

[0035] The specific implementation method of calculating the centroid of the target object segmentation mask according to the distribution of pixels in the target object segmentation mask and generating the centroid coordinates of the object is as follows: the centroid coordinates of the object are important geometric features representing the center position of the object, and are of key significance for the subsequent grasping posture planning. The system calculates the centroid coordinates of the object based on the spatial distribution of pixels in the target object segmentation mask. Specifically, the system first traverses all pixels with a value of 1 (indicating foreground) in the segmentation mask and records the coordinates of these pixels. Then, the system calculates the arithmetic mean of the coordinates of these foreground pixels to obtain the centroid coordinates of the object on the image plane. The calculation process takes into account the distribution of pixels in the mask, and each foreground pixel contributes equally to the centroid position. In order to improve the calculation accuracy, the system also combines depth information to convert the centroid coordinates on the two-dimensional plane into the centroid position in three-dimensional space. The specific method is to median filter the depth value corresponding to the centroid coordinate (x_c, y_c) and the depth value in the surrounding small area (usually a 9×9 pixel window) to obtain a stable depth estimate, and then convert it into a three-dimensional coordinate in combination with the camera intrinsic parameter. For example, for an irregularly shaped object, such as a wrench, its segmentation mask may present an elongated shape. The system calculates the average position of all foreground pixels to obtain the center of mass position of the wrench, which is usually located in the middle area of the wrench and is the ideal target point for grasping. The choice of the center of mass calculation method takes into account the balance between computational efficiency and physical meaning. The simple arithmetic average is used instead of the weighted average because in the robot grasping scene, the object is usually regarded as a homogeneous body, and each part has an equal impact on the grasping. In some objects with special shapes (such as L-shaped or ring-shaped objects), the center of mass may be located outside the object entity. The system will further analyze the geometric structure of the object and adjust the center of mass to the entity part of the object through morphological skeleton or convex hull analysis to ensure that the grasping point is located in the actual graspable area of the object. This center of mass calculation method is simple and efficient, while taking into account the overall shape characteristics of the object, providing accurate target position information for subsequent grasping posture planning.

[0036] Please continue reading Figure 1 , generating a three-dimensional point cloud of the target object according to the spatial calibration depth matrix, the target object segmentation mask and the camera intrinsic parameter matrix, extracting a planar point set and a non-planar point set from the three-dimensional point cloud using a RANSAC algorithm, performing a main direction analysis on the non-planar point set, extracting an axial feature vector of the target object, and calculating a plane normal vector of the planar point set; In one embodiment of the present invention, to generate the three-dimensional point cloud of the target object based on the spatially calibrated depth matrix, the target object segmentation mask, and the camera intrinsic matrix, and use the RANSAC algorithm to extract the planar point set and the non-planar point set from the three-dimensional point cloud, and perform a principal direction analysis on the non-planar point set to extract the axial feature vector of the target object and calculate the plane normal vector of the planar point set, including: According to the camera intrinsic matrix, perform a projection transformation on the depth values in the spatially calibrated depth matrix and the pixel point coordinates corresponding to the target object segmentation mask to generate a set of projected coordinate points, and perform a conversion from the camera coordinate system to the world coordinate system on the set of projected coordinate points to generate the three-dimensional point cloud of the target object; Use the RANSAC algorithm for plane fitting, and divide the three-dimensional point cloud into a planar point set and a non-planar point set according to the fitting residuals; Perform principal component analysis on the non-planar point set, extract the principal direction component, and generate the axial feature vector after normalizing the principal direction component; Perform singular value decomposition on the planar point set, take the singular vector corresponding to the variance value in the decomposition result, and generate the plane normal vector after normalizing the singular vector.

[0037] The following specifically describes the steps involved in the above embodiment: According to the camera intrinsic matrix, a projection transformation is performed on the depth values in the spatially calibrated depth matrix and the pixel coordinates corresponding to the target object segmentation mask to generate a set of projected coordinate points. The specific implementation of converting the set of projected coordinate points from the camera coordinate system to the world coordinate system to generate the three-dimensional point cloud of the target object is as follows: The system first extracts the pixel coordinates of all values equal to 1 (indicating the target object area) from the target object segmentation mask. Then, for each such pixel point, the system reads its corresponding depth value from the spatially calibrated depth matrix. Using the camera intrinsic matrix (including parameters such as focal length and principal point coordinates), pixel coordinates, and depth values, the system performs back-projection calculations from the image plane to three-dimensional space. Specifically, the system establishes a mapping relationship from the image plane coordinate system (u, v coordinates) to the camera coordinate system (X, Y, Z coordinates), where the Z-axis direction points forward of the camera, and the X-axis and Y-axis are parallel to the horizontal and vertical directions of the image plane respectively. For each pixel point (u, v) in the image and its depth value z, the system calculates its three-dimensional coordinates in the camera coordinate system. This conversion takes into account the physical size of the pixels and the optical characteristics of the camera to ensure accurate restoration of spatial positions. After generating the point cloud in the camera coordinate system, the system further converts these points from the camera coordinate system to the world coordinate system. This conversion is achieved through the camera extrinsic matrix, which describes the position and orientation of the camera in the world coordinate system. The conversion process includes a rotation transformation and a translation transformation to transform the points in the camera coordinate system to the world coordinate system relevant to the robot operation. For example, in an industrial grasping scenario, if the camera is installed above the robot arm, the system will convert the object point cloud seen from the camera's perspective to the world coordinate system defined by the robot base for subsequent grasping path planning. This two-stage conversion based on intrinsic and extrinsic parameters ensures the accuracy of three-dimensional reconstruction. Especially in scenarios where the depth values vary over a large range, this method provides higher accuracy than simple linear projection. The world coordinate system selected in this step is usually consistent with the robot base coordinate system, so that it can be directly used for subsequent motion planning, avoiding additional coordinate conversions and reducing cumulative errors.

[0038] The specific implementation of using the RANSAC algorithm for plane fitting and dividing the three-dimensional point cloud into a plane point set and a non-plane point set according to the fitting residuals is as follows: The RANSAC (Random Sample Consensus) algorithm is an iterative method for robustly estimating model parameters from data containing a large number of outliers. In this step, the RANSAC algorithm is used to identify plane structures from the three-dimensional point cloud. The system first randomly selects three points from the point cloud (the minimum point set for plane definition) and fits a plane equation based on these three points. Then the system calculates the distance from each point in the point cloud to this plane and marks the points with a distance less than a preset threshold (usually set to 5 - 10 millimeters, depending on the grasping accuracy requirements) as inliers (i.e., points that conform to the current plane model). The system repeats the above random selection and fitting process multiple times (the number of iterations is usually 500 - 1000 times to balance computational efficiency and result stability), and records the plane model with the largest number of inliers each time. Finally, the system selects the plane model with the largest number of inliers as the best-fitting plane, defines the corresponding inlier set as the plane point set, and the remaining points as the non-plane point set. For example, when grasping a box placed on a table, the RANSAC algorithm can accurately identify the plane point set representing the top surface of the box, even if there are some irregular protrusions on the box surface or noise in the point cloud data. These plane point sets correspond to stable grasping surfaces, while the non-plane point sets may represent the edges or other structural features of the object. The key parameters of the RANSAC algorithm include the distance threshold and the number of iterations, and the selection of these parameters is based on an assessment of the point cloud noise level and the geometric complexity of the object. If the distance threshold is too small, plane recognition will be incomplete; if it is too large, non-plane regions may be wrongly included. If the number of iterations is too small, the optimal plane may not be found; if it is too large, it will increase the unnecessary computational burden. In practical applications, these parameters will be reasonably set according to sensor characteristics and application scenarios. For example, in a high-precision camera system, the distance threshold can be set smaller (2 - 3 millimeters) to obtain more accurate plane segmentation. The advantage of the RANSAC algorithm lies in its robustness to outliers. Even in the presence of a large amount of noise or outliers in the point cloud, it can accurately identify the main plane structures, which is crucial for object grasping in complex environments.

[0039] The specific implementation method of performing principal component analysis on the non-planar point set, extracting the principal direction component, and generating the axial feature vector after normalizing the principal direction component is as follows: Principal Component Analysis (PCA) is a dimensionality reduction technique used to find the main directions of variation in data. When dealing with a non-planar point set, the system first collects the coordinates of all points into a data matrix, where each row represents the X, Y, and Z coordinates of a three-dimensional point. Then the system calculates the covariance matrix of this data matrix, and the covariance matrix reflects the distribution of the point set in various directions. Next, the system performs eigenvalue decomposition on the covariance matrix to obtain three eigenvalues and their corresponding eigenvectors. These three eigenvectors respectively represent the three main directions of the point set distribution, and the corresponding eigenvalues represent the variance magnitudes of the point set in these directions. The larger the eigenvalue, the more dispersed the distribution of the point set in that direction. The system sorts these three eigenvectors in descending order according to the corresponding eigenvalues, and takes the eigenvector with the largest eigenvalue as the principal direction component, and this direction usually corresponds to the main axis direction of the object in the point set. Finally, the system normalizes the principal direction component (i.e., adjusts the vector length to 1) to obtain the axial feature vector with unit length. For example, for an elongated object such as a screwdriver, the non-planar point set is mainly distributed in the long axis direction of the object, and the PCA analysis will identify this long axis direction as the principal direction, and the generated axial feature vector will point to the main axis direction of the screwdriver. This method is particularly suitable for objects with obvious extension directions, such as rod-shaped, tubular, or cuboid objects. A key consideration in principal component analysis is the preprocessing of the data. Before performing PCA, the system will normalize the point cloud data to ensure that the scales of all dimensions are consistent and avoid analysis biases caused by scale differences. In addition, the system will also perform noise reduction and outlier removal on the point cloud to improve the accuracy of the principal direction analysis. In practical applications, the extraction of the principal direction is very important for determining the grasping posture. Especially for elongated objects that need to be grasped along a specific direction, the accurate axial feature vector can guide the robot to approach and grip the object in the appropriate direction, improving the success rate and stability of grasping.

[0040] Perform singular value decomposition on the plane point set, take the singular vector corresponding to the variance value in the decomposition result, and normalize the singular vector to generate the plane normal vector. The specific implementation method is: Singular Value Decomposition (SVD) is a powerful matrix decomposition technology that can reveal the intrinsic structural characteristics of the data. For a plane point set, the system first organizes the coordinates of all points into an N×3 matrix X, where N is the number of points and each row contains the X, Y, and Z coordinates of a point. Before performing SVD, the system will center the point set, that is, subtract the centroid coordinates of the point set from the coordinates of each point so that the point set is centered on the origin. Then, the system performs SVD decomposition on the centralized data matrix X: X=US , where U and V are orthogonal matrices, S is a diagonal matrix whose diagonal elements are called singular values and are arranged in descending order. In three-dimensional space, SVD will generate three singular values and their corresponding singular vectors. For a planar point set, since the points are mainly distributed on a two-dimensional plane, the third singular value is usually significantly smaller than the first two, and the corresponding third singular vector points to the direction with the least distribution of the point set, that is, the normal direction of the plane. The system takes the third singular vector and normalizes it to unit length to obtain the normal vector of the plane. This normal vector is perpendicular to the plane and indicates the spatial orientation of the plane. For example, for a tablet placed on a desktop, SVD analysis will identify the normal vector of the plane where the tablet screen is located, which is perpendicular to the screen plane and points above the screen. This SVD-based plane normal vector calculation method has high numerical stability and can accurately estimate the direction of the plane even when there is a certain amount of noise in the point cloud. In the process of determining the plane normal vector, the system will additionally check the direction consistency of the normal vector to ensure that the normal vector points to the outside of the object rather than the inside, which is crucial for the subsequent determination of the correct grasping approach direction. For example, when grabbing a book placed on a table, the normal vector should point to the direction of the book cover, not the direction where the book touches the table, so that the robot can approach and grab the object from the correct direction. This method provides an accurate representation of the plane direction by analyzing the geometric distribution characteristics of the plane point set, which provides an important basis for the subsequent determination of the grasping posture.

[0041] In one embodiment of the present invention, the plane fitting is performed using the RANSAC algorithm, and the three-dimensional point cloud is divided into a plane point set and a non-plane point set according to the fitting residual, including: The three-dimensional point cloud is divided into a feasible grasping region, and point cloud data whose distance from the camera exceeds the grasping arm span of the robot is eliminated to generate a valid grasping point cloud; Performing local curvature calculation on the effective captured point cloud according to the curvature threshold, and dividing the point cloud with a curvature value lower than the curvature threshold into a candidate plane point set; Perform RANSAC iterative calculation on the candidate planar point set, extract the plane equation parameters, and calculate the distance deviation value of each point from the plane; Perform planar support analysis on the candidate planar point set according to the distance deviation value, determine whether the planar region meets the gripper base support requirements, and determine the point set that meets the support requirements as the planar point set; Perform clustering processing on the point cloud data that does not belong to the planar point set, and generate a non-planar point set after removing the outliers.

[0042] The following specifically describes the steps involved in the above embodiments: The specific implementation method for dividing the graspable region of the three-dimensional point cloud and removing the point cloud data whose distance from the camera exceeds the reach of the robot's grasping arm to generate the effective grasp point cloud is as follows: The system first obtains the physical parameters of the robot, especially the maximum working radius (i.e., reach) of the robot arm, which is usually read from the configuration file of the robot control system. Then, the system calculates the Euclidean distance from each point in the three-dimensional point cloud to the origin of the camera, that is, the straight-line distance between two points in space. For each point, the system takes the square root of the sum of the squares of its spatial coordinates to obtain the straight-line distance from the point to the camera. The system compares this distance with the effective grasping distance of the robot converted after considering the installation position of the camera in the robot's working space. If the distance of the point exceeds the effective grasping range of the robot, the point is marked as a non-graspable point and removed from the point cloud. This removal process takes into account the fixed transformation relationship between the camera position and the robot base to ensure the accuracy of the removal. For example, in an industrial sorting scenario, if the reach of the robot arm is 0.8 meters, the system will remove the object point cloud that is more than 0.8 meters away from the camera (considering the offset of the camera installation position) because even if these points are successfully recognized, the robot cannot actually grasp them. In addition, the system will also remove the points that are too close to the camera (usually less than 0.2 meters) because these points may be within the minimum working radius of the robot and cannot be effectively grasped either. This division of the graspable region takes into account the kinematic constraints of the robot, and the setting of the reach range is directly related to the feasibility of the grasping task. An overly large range will result in a grasping task that cannot be executed, while an overly small range may miss effective grasping opportunities. An appropriate range setting (usually 80%-90% of the robot's rated reach) not only ensures the feasibility of grasping but also leaves enough safety margin to prevent the robot from reaching the joint limit position during the grasping process, improving the smoothness and reliability of the grasping action.

[0043] The specific implementation method of calculating the local curvature of the effective grasping point cloud according to the curvature threshold and dividing the point cloud with a curvature value lower than the curvature threshold into the candidate plane point set is as follows: Curvature is a geometric quantity that describes the degree of curvature of a surface near a certain point. The curvature of a plane is zero, while the curvature of a surface is greater than zero. The system performs local curvature calculations for each point in the effective grasping point cloud. The specific method is as follows: First, a local neighborhood is determined for each point. Usually, all points within a spherical region with a radius of 5-10 millimeters around the point are selected as the neighborhood point set. Then, for each point and its neighborhood point set, the system uses the principal component analysis (PCA) method to calculate the local curvature. The system organizes the coordinates of the neighborhood point set into a matrix, calculates the covariance matrix of this matrix, and performs eigenvalue decomposition on the covariance matrix to obtain three eigenvalues λ1, λ2, λ3 (sorted from largest to smallest). The local curvature can be estimated through the proportional relationship of the eigenvalues. The specific calculation formula is: Curvature = λ3 / (λ1 + λ2 + λ3). The closer this ratio is to zero, the closer the region where the point is located is to a plane; the larger the ratio, the higher the curvature of the region. The system sets a curvature threshold (the empirical value is usually 0.02-0.05), and divides the points with a curvature value lower than this threshold into the candidate plane point set. For example, when identifying the flat bottom of a cup, the system will calculate the curvature of the points in the bottom region of the cup. Since the bottom of the cup is approximately a plane, the curvature values of these points are very low and will be successfully divided into the candidate plane point set, while regions with higher curvatures such as the cup wall and cup mouth will be excluded. The setting of the curvature threshold needs to consider the flatness of the actual object surface and the noise level of the point cloud. If the threshold is set too low, many actual plane points will be wrongly excluded, and if it is set too high, some slightly curved surfaces will be misjudged as planes. In the scenario of grasping precision parts, due to the high requirement for plane recognition accuracy, the curvature threshold is usually set relatively low (0.01-0.02); while in the grasping of daily items, considering the changes in the manufacturing accuracy and surface state of the objects, the curvature threshold will be appropriately relaxed (0.03-0.05). This pre-screening strategy based on curvature significantly improves the efficiency and accuracy of subsequent RANSAC plane fitting, especially when dealing with large-scale point cloud data, reducing the computational burden and the possibility of misclassification.

[0044] The specific implementation of performing RANSAC iterative calculation on the candidate plane point set, extracting the plane equation parameters, and calculating the distance deviation value of each point from the plane is as follows: The system performs RANSAC plane fitting on the candidate plane point set, and the specific process is as follows: First, the system randomly selects 3 points (the minimum point set for plane definition) from the candidate plane point set, and calculates the plane equation parameters (ax + by + cz + d = 0, where (a, b, c) is the plane normal vector and d is the signed distance from the plane to the origin) based on these 3 points. Then, the system calculates the perpendicular distance from each point in the candidate plane point set to this plane, that is, the distance deviation value of the point from the plane. The distance calculation formula is: distance = |ax + by + cz + d| / √(a² + b² + c²), where (x, y, z) are the coordinates of the point. The system counts the number of points whose distance deviation value is less than the set threshold (usually 1 - 5 mm), and these points are called "inliers", that is, the points that conform to the current plane model. The system repeats the above random selection and calculation process multiple times (the number of iterations is related to the size of the point set and the desired success probability, usually 500 - 2000 times), and records the number of inliers and the corresponding plane parameters each time. Finally, the system selects the plane model with the largest number of inliers as the best fitting plane, and records its plane equation parameters and the distance deviation values of each point. For example, when identifying the cover of a folder, even if there are some slight undulations or wrinkles on the folder surface, the RANSAC algorithm can accurately fit the equation representing the overall plane and calculate the specific deviation value of each point from this plane. The setting of the RANSAC iteration number needs to balance the calculation efficiency and result reliability. For scenarios with a large number of points (such as more than 10,000 points), the iteration number is usually set above 1000 to ensure a high enough probability of finding the optimal plane; for scenarios with a small number of points, 500 iterations are usually sufficient. The setting of the distance threshold reflects the requirement for the flatness of the plane. In high-precision industrial applications, the threshold is usually set to 1 - 2 mm; in ordinary object grasping, considering the manufacturing tolerance and surface state of the object, the threshold can be relaxed to 3 - 5 mm. The advantage of the RANSAC algorithm lies in its strong resistance to outliers. Even if some non-plane points are mixed in the candidate plane point set, the algorithm can still accurately estimate the plane parameters, which is particularly important for processing noisy point cloud data in the actual environment.

[0045] Based on the distance deviation value, perform planar support analysis on the candidate planar point set to determine whether the planar region meets the support requirements of the gripper base. The specific implementation of determining the point set that meets the support requirements as the planar point set is as follows: Planar support analysis is a process of evaluating whether the identified plane is suitable for the stable placement of the robotic gripper. First, the system determines the points with a distance deviation less than the threshold as in-plane points according to the plane equation and distance deviation value calculated by RANSAC. These points form a preliminary planar region. Then, the system conducts a spatial distribution analysis on these in-plane points, including density analysis and connectivity analysis. Density analysis calculates the spatial density of the points in the planar region to ensure that the density is not lower than a preset threshold (usually not less than 10 - 15 points per square centimeter) to guarantee the reliability of plane recognition. Connectivity analysis uses the region growing algorithm to group points with a mutual distance less than a certain threshold (usually 5 - 10 mm) into the same connected region and find the largest connected region as the main planar region. Next, the system calculates the geometric characteristics of the main planar region, including area, perimeter, contour shape, etc. The system compares the calculated planar area with the contact area requirement of the gripper base. Usually, the gripper base requires a certain support area to achieve stable grasping. This area depends on the type of gripper. For ordinary parallel grippers, the support area is usually not less than 4 square centimeters; for suction cup grippers, it is required to be not less than 1.2 times the suction cup area. In addition, the system also checks whether the shape of the planar region meets the shape requirements of the gripper base. For example, for a gripper with a rectangular base, the aspect ratio of the support plane should not be too large. Usually, the ratio of the long side to the short side is required not to exceed 3:1. For example, when grasping a precision part, even if there are some small bumps and depressions on the part surface, the system can identify a large enough and flat enough region based on distance deviation analysis to ensure that the gripper can stably contact and grasp the part. The setting of each parameter in planar support analysis is directly related to the stability and reliability of grasping. The setting of the support area requirement takes into account the physical characteristics of the gripper and the weight distribution of the grasped object. An overly small area will lead to unstable grasping, while an overly large area may limit the grasping ability for small objects. The setting of the density threshold takes into account the acquisition accuracy and noise level of the point cloud. In a high-precision point cloud acquisition system, the density threshold can be set higher to obtain a more accurate plane recognition result.

[0046] The specific implementation of clustering the point cloud data that does not belong to the planar point set and removing outliers to generate a non-planar point set is as follows: The system processes the point cloud data that does not belong to the planar point set to extract the non-planar features of the object. First, the system uses the Euclidean clustering algorithm to perform clustering analysis on these points. Euclidean clustering is a point cloud segmentation method based on spatial distance. The system sets a distance threshold (usually 5 - 15 mm, adjusted according to the object size), and groups the points in space with a distance less than this threshold into the same cluster. Specifically in implementation, the system starts from an unclassified point, finds all the points in its neighborhood (points with a distance less than the threshold), marks these points as the same cluster, and then recursively processes the newly added points until no more neighborhood points can be found. The system repeats this process until all points are classified. After clustering, the system analyzes each cluster, calculates features such as the size (number of points), volume, and spatial distribution of the cluster. The system sets a minimum number of points threshold (usually 1% - 5% of the total number of points, depending on the object complexity), and regards the clusters with a number of points less than this threshold as outliers or noise and removes them from the data. These outliers are usually misdetected points caused by factors such as sensor noise, occlusion, or reflection. Removing these points helps improve the accuracy of subsequent processing. The remaining point clusters constitute the non-planar point set, and these points usually represent non-planar features such as the edges, corners, and curved surfaces of the object. For example, when processing the point cloud data of a teacup, the planar point set may include the bottom of the cup, while the non-planar point set contains the curved surface parts such as the cup wall and the cup handle. By removing outliers, the system can obtain a more pure representation of non-planar features and improve the accuracy of subsequent feature extraction. The setting of the clustering distance threshold needs to consider the geometric characteristics of the object and the density of the point cloud. If the threshold is too small, it will lead to over-segmentation, dividing points that should belong to the same part of the object into different clusters; if the threshold is too large, it may wrongly merge different parts. The setting of the minimum number of points threshold needs to balance the denoising effect and feature retention. If the threshold is too high, it may wrongly remove small but important features of the object; if the threshold is too low, it may retain too many noise points. The reasonable setting of these parameters directly affects the quality of non-planar feature extraction and thus affects the planning accuracy of the grasping posture.

[0047] Please continue to refer to Figure 1 According to the plane normal vector, the axial feature vector, and the centroid coordinates of the object, calculate the grasping parameters of the target object. Determine the plane normal vector as the grasping approach vector, determine the axial feature vector as the gripper opening and closing vector, obtain the grasping posture auxiliary vector through the cross product operation of the grasping approach vector and the gripper opening and closing vector, combine the grasping approach vector, the gripper opening and closing vector, and the grasping posture auxiliary vector to generate a grasping rotation matrix, and combine with the centroid coordinates of the object to generate a grasping posture transformation matrix.

[0048] In an embodiment of the present invention, calculating the grasping parameters of the target object according to the plane normal vector, the axial feature vector, and the centroid coordinates of the object, determining the plane normal vector as the grasping approach vector, determining the axial feature vector as the gripper opening and closing vector, obtaining the grasping attitude auxiliary vector through the cross product operation of the grasping approach vector and the gripper opening and closing vector, combining the grasping approach vector, the gripper opening and closing vector, and the grasping attitude auxiliary vector to generate a grasping rotation matrix, and generating a grasping attitude transformation matrix in combination with the centroid coordinates of the object, includes: Perform a direction consistency judgment on the plane normal vector, perform a reverse process on the plane normal vector according to the judgment result, and perform a normalization operation on the processed plane normal vector to obtain the grasping approach vector; Perform a numerical normalization process on the axial feature vector, perform an orthogonal correction on the normalized axial feature vector, and perform a normalization operation on the corrected vector to obtain the gripper opening and closing vector; Perform a cross product operation on the grasping approach vector and the gripper opening and closing vector, perform an orthogonality verification on the cross product result, and perform a normalization process on the verified vector to obtain the grasping attitude auxiliary vector; Combine the grasping approach vector as the first column vector, the gripper opening and closing vector as the second column vector, and the grasping attitude auxiliary vector as the third column vector to generate a grasping rotation matrix; Combine the grasping rotation matrix and the centroid coordinates of the object in the form of a homogeneous transformation matrix, calculate the rigid body transformation parameters of translation and rotation, and generate a grasping attitude transformation matrix.

[0049] The following specifically describes the steps involved in the above embodiment: The specific implementation of performing direction consistency judgment on the plane normal vector, performing reverse processing on the plane normal vector according to the judgment result, and performing unitization operation on the processed plane normal vector to obtain the grasping approach vector is as follows: The system first needs to ensure that the direction of the plane normal vector points to the outside of the object rather than the inside, because the grasping approach vector should indicate that the robot approaches the object from the outside. The direction consistency judgment is achieved by calculating the dot product of the plane normal vector and the line-of-sight vector (the vector from the camera to the centroid of the object). If the dot product is positive, it indicates that the plane normal vector generally points in the camera direction (the outside of the object), and the original direction is maintained; if it is negative, it indicates that the normal vector points to the inside of the object, and reverse processing is required, that is, each component of the vector is negated. For example, for a book placed on a table, the plane normal vector should point to the cover direction of the book, rather than the direction where the back of the book touches the table. After reverse processing, the system performs unitization processing on the plane normal vector, that is, normalizes the vector length to 1 to obtain the grasping approach vector. The unitization processing is achieved by calculating the magnitude of the vector (the square root of the sum of the squares of each component), and then dividing each component by the magnitude. This direction consistency judgment ensures that the robot always approaches the object from the correct direction, avoiding situations such as penetrating the object or attempting to grasp from unreachable positions such as the bottom of the object, greatly improving the grasping success rate.

[0050] The specific implementation of performing numerical normalization on the axial feature vector, performing orthogonal correction on the normalized axial feature vector, and performing unitization operation on the corrected vector to obtain the gripper opening and closing vector is as follows: The axial feature vector represents the principal axis direction of the object and will be used to determine the opening and closing direction of the gripper. The system first performs numerical normalization on the axial feature vector, mapping each component of the vector to the range [-1, 1] to eliminate numerical differences that may be brought by objects of different scales. Then the system performs orthogonal correction on the normalized axial feature vector and the already determined grasping approach vector to ensure that the two vectors are perpendicular to each other. The orthogonal correction is achieved through the Gram - Schmidt orthogonalization process: The system calculates the projection of the axial feature vector in the direction of the grasping approach vector, and then subtracts this projection from the original vector to obtain a new vector that is orthogonal to the grasping approach vector. For example, for a cylinder, the grasping approach vector is usually perpendicular to its top surface, and the gripper opening and closing vector after orthogonal correction is along the radial direction of the cylinder, ensuring that the gripper can correctly grip the object. After orthogonal correction, the system performs unitization processing on this vector to obtain the gripper opening and closing vector with unit length. The establishment of this orthogonal relationship ensures the spatial correctness of the grasping posture, making the opening and closing direction of the gripper always perpendicular to the approaching direction, which conforms to the physical structure constraints of most mechanical grippers.

[0051] The specific implementation method of performing a cross product operation on the grasping approach vector and the gripper opening / closing vector, verifying the orthogonality of the cross product result, and normalizing the verified vector to obtain the grasping pose auxiliary vector is as follows: The system performs a cross product operation on the determined grasping approach vector and the gripper opening / closing vector to obtain a third vector that is perpendicular to the first two vectors. The cross product operation is a vector operation, and its result is a new vector perpendicular to the plane where the original two vectors lie. The system verifies the orthogonality of the cross product result by checking whether this vector is indeed perpendicular to the grasping approach vector and the gripper opening / closing vector (calculating the dot product value to ensure it is close to zero). If the orthogonality condition is not met, the system will re-perform the cross product calculation or adjust the first two vectors. For example, when grasping a cuboid, if the grasping approach vector points to the top surface of the cuboid and the gripper opening / closing vector is along the length direction of the cuboid, then the obtained grasping pose auxiliary vector by cross product will be along the width direction of the cuboid. After verification, the system normalizes this vector to obtain the grasping pose auxiliary vector. This three-vector orthogonal system constructed by cross product ensures the complete definition of the grasping pose in three-dimensional space and guarantees the internal consistency of the pose, avoiding pose distortion that may be caused by numerical errors.

[0052] The specific implementation method of combining the grasping approach vector as the first column vector, the gripper opening / closing vector as the second column vector, and the grasping pose auxiliary vector as the third column vector to generate the grasping rotation matrix is as follows: The system combines the three unit orthogonal vectors by columns to form a 3×3 rotation matrix. Specifically, the three components of the grasping approach vector form the first column of the matrix, the three components of the gripper opening / closing vector form the second column, and the three components of the grasping pose auxiliary vector form the third column. These three column vectors respectively correspond to the three coordinate axis directions of the end effector coordinate system of the robot. For example, for a standard mechanical gripper, the grasping approach vector usually corresponds to the Z axis of the gripper (towards the front of the gripper), the gripper opening / closing vector corresponds to the X axis (along the gripper opening / closing direction), and the grasping pose auxiliary vector corresponds to the Y axis (perpendicular to the first two axes). The rotation matrix constructed in this way completely describes the pose of the robot end effector relative to the world coordinate system. Since the three vectors have been normalized to unit vectors and are mutually orthogonal, the generated rotation matrix is a standard orthogonal matrix with a determinant value of 1, ensuring the rigid body property of the rotation transformation, that is, no scaling or shear deformation occurs. This construction method directly derives the grasping pose from the geometric features of the object, ensuring that the grasping pose matches the object shape and improving the success rate and stability of grasping.

[0053] The specific implementation of combining the grasping rotation matrix with the centroid coordinates of the object in the form of a homogeneous transformation matrix to calculate the rigid body transformation parameters of translation and rotation and generate the grasping pose transformation matrix is as follows: The system expands the 3×3 grasping rotation matrix into a 4×4 homogeneous transformation matrix, where the upper left 3×3 part is the rotation matrix, the upper right 3×1 part is the centroid coordinates of the object (translation vector), the lower left is a 1×3 zero vector, and the lower right is the scalar 1. The homogeneous transformation matrix is the standard form for representing position and orientation in robotics and can describe both rotation and translation transformations simultaneously. The system may also need to fine-tune the centroid position, for example, offset a certain distance along the grasping approach vector direction (usually half of the gripper thickness plus a safety margin of 10 - 20 mm) to ensure that the gripper can correctly contact the object surface instead of directly moving to the object centroid. For example, when grasping a cube, the grasping pose transformation matrix generated by the system will guide the robot to first move the gripper to the appropriate position of the cube and then approach and grip the object in the correct pose. This complete transformation description combining rotation and translation enables the robot to perform precise grasping actions, taking into account the position, shape, and orientation characteristics of the object, as well as the physical structure constraints of the robot gripper, and realizing a seamless conversion from perception to execution.

[0054] In an embodiment of the present invention, the combining the grasping rotation matrix with the centroid coordinates of the object in the form of a homogeneous transformation matrix to calculate the rigid body transformation parameters of translation and rotation and generate the grasping pose transformation matrix includes: Convert the grasping rotation matrix into Euler angle representation, perform reachability analysis according to the robot joint motion constraints, and filter out the Euler angle parameters that meet the joint limit requirements; Perform gravity compensation calculation on the centroid coordinates of the object, predict the gravity influence during the grasping process, and generate the compensated centroid coordinates of the object; Generate a set of candidate grasping poses based on the Euler angle parameters and the compensated centroid coordinates of the object, and calculate the robot joint angle solutions for each candidate pose; Perform collision detection analysis on the set of candidate grasping poses, and eliminate the grasping poses that may cause interference between the robot and the environment; Comprehensively score the remaining candidate grasping poses according to the joint movement distance and the pose change amplitude, and select the Euler angle parameters and position parameters corresponding to the pose with the optimal score; Convert the Euler angle parameters back into the rotation matrix form, and combine them with the position parameters to generate the grasping pose transformation matrix.

[0055] The following specifically describes the steps involved in the above embodiments: The specific implementation of converting the grasping rotation matrix into Euler angle representation and performing reachability analysis according to the robot joint motion constraints to screen out the Euler angle parameters that meet the joint limit requirements is as follows: The system first converts the 3×3 grasping rotation matrix into Euler angle representation. Euler angles are a way to describe three-dimensional rotations, usually represented as the rotation angles (α,β,γ) around the X, Y, and Z axes. The conversion process is based on the mathematical relationships between the elements in the rotation matrix. For example, for the ZYX Euler angle sequence, β can be obtained by the arcsine of the element in the first row and third column of the matrix. Due to the multi-solution nature of Euler angle representation (the same pose may have multiple sets of Euler angle representations), the system usually generates all possible Euler angle combinations. Then the system obtains the robot joint motion constraint parameters, including the angle limit ranges of each joint (usually read from the robot technical manual or configuration file). For each set of Euler angles, the system uses the inverse kinematics algorithm to calculate the corresponding robot joint angles and checks whether each joint angle is within its limit range. For example, for a 6-degree-of-freedom robotic arm, if the limit of its fourth joint is ±120 degrees, the system will eliminate the Euler angle parameters that cause the joint angle to exceed this range. This step takes into account the physical structure limitations of the robot, avoids planning poses that the robot cannot actually reach, and improves the practicality and reliability of the grasping plan.

[0056] The specific implementation of performing gravity compensation calculation on the object centroid coordinates, predicting the gravity influence during the grasping process, and generating the compensated object centroid coordinates is as follows: The system considers the influence of gravity on the grasping process, especially for heavier or unbalanced objects. First, the system estimates the mass and mass distribution of the target object, usually based on the object volume and preset density parameters, or queries from the object database. Then the system calculates the possible displacement or deformation of the object under the action of gravity, and this calculation takes into account factors such as the mass of the object, the rigidity, the grasping force of the gripper, and the contact area. For relatively rigid objects (such as metal parts), gravity compensation mainly considers the selection of the clamping points; for flexible objects (such as fabrics), the deformation of the object after grasping needs to be considered. The system adjusts the object centroid coordinates according to the calculation results, usually by adding a compensation amount in the vertical direction, and this compensation amount is proportional to the object mass and the gripper stiffness. For example, for a 500-gram object with a gripper stiffness of 2000 N / m, the compensation amount in the vertical direction is approximately 2.5 millimeters. This gravity compensation ensures that the robot can accurately grasp the object under the actual gravity conditions of the object, reducing the position deviation and slippage risk caused by gravity during the grasping process.

[0057] Generate a set of candidate grasping postures based on the Euler angle parameters and the compensated centroid coordinates of the object. The specific implementation of calculating the robot joint angle solutions for each candidate posture is as follows: The system combines the filtered Euler angle parameters with the compensated centroid coordinates of the object to generate multiple sets of candidate grasping postures. For each set of postures, the system uses the inverse kinematics algorithm to calculate the corresponding robot joint angles. Inverse kinematics is the process of calculating the joint angles based on the position and posture of the end effector. For robots with high redundancy (such as 7-degree-of-freedom robotic arms), one end effector posture may correspond to multiple sets of joint angle solutions. The system uses numerical iterative methods (such as the Jacobian matrix method) or analytical methods (such as the geometric method) to solve the inverse kinematics equations and obtain the set of joint angle solutions corresponding to each candidate posture. The system also calculates the joint motion amounts from the current robot posture to each candidate posture for subsequent scoring and ranking. For example, when grasping a part on a workbench from the initial state, the system may generate 5 - 10 different grasping postures and calculate the 6 joint angle values corresponding to each posture. This step ensures that the system fully considers the kinematic characteristics of the robot, generates diverse candidate grasping strategies, and provides a basis for subsequent optimization and selection.

[0058] The specific implementation of performing collision detection and analysis on the set of candidate grasping postures and eliminating the grasping postures that may cause interference between the robot and the environment is as follows: The system uses a collision detection algorithm to evaluate the safety of each candidate grasping posture. First, the system constructs a three-dimensional model of the environment, including fixed obstacles such as workbenches, fences, and other objects, and this information usually comes from a pre-established environmental map or real-time sensor data. Then the system loads the geometric model of the robot, including the shapes and sizes of each joint and link. For each candidate posture, the system determines the exact positions of each part of the robot in space according to the calculated joint angle solutions and checks whether there is a collision with the environmental model. The collision detection uses a fast collision detection algorithm (such as the hierarchical bounding box method), which not only checks the collision situation at the final state but also simulates the motion trajectory of the robot from the current posture to the target posture to ensure that no collision occurs during the entire motion process. For example, on a workbench with multiple objects, the system will detect whether the robotic arm will touch other surrounding objects during the process of approaching the target object. This step filters out unsafe grasping postures, ensures that the robot can perform the grasping task safely, and avoids possible equipment damage or task failure.

[0059] Based on the joint movement distance and the amplitude of pose change, the specific implementation method for comprehensively scoring the remaining candidate grasping poses and selecting the Euler angle parameters and position parameters corresponding to the pose with the optimal score is as follows: The system conducts a comprehensive evaluation of the candidate poses that pass the collision detection and establishes a multi-index scoring mechanism. The scoring indicators include: the total joint movement distance (the sum of the absolute values of the angle changes of each joint), the maximum joint movement (the angle change of the joint with the largest change), the length of the end-effector movement path, the amplitude of pose change (the rotation angle between the current pose and the target pose), and the grasping stability indicator (based on the relative position of the grasping point and the object's center of gravity). The system assigns weights to each indicator. The weight of the total joint movement distance is usually between 0.3 and 0.4, the weight of the grasping stability indicator is between 0.2 and 0.3, and the weights of the remaining indicators are assigned between 0.1 and 0.2. The weight setting takes into account the task characteristics. For example, for high-speed grasping tasks, the weight of the movement speed indicator will increase; for precision grasping tasks, the weight of the grasping stability indicator will increase. The system calculates the comprehensive score of each candidate pose and selects the pose with the highest score as the final grasping pose. For example, when grasping a tool, the system tends to select a pose with a small joint movement and a stable grasping point, avoiding large-amplitude joint movements. This multi-dimensional evaluation mechanism ensures the overall optimality of the selected grasping pose in terms of efficiency, stability, and safety, improving the success rate and efficiency of the grasping task.

[0060] The specific implementation method for converting the Euler angle parameters back to the rotation matrix form and combining them with the position parameters to generate the grasping pose transformation matrix is as follows: The system converts the selected Euler angle parameters (α, β, γ) back to the 3×3 rotation matrix form. The conversion process is the standard calculation from Euler angles to the rotation matrix, including calculating the basic rotation matrices for each axis rotation and then multiplying them in the specified rotation order (such as the ZYX order). Then, the system combines the rotation matrix with the position parameters to construct a 4×4 homogeneous transformation matrix, which completely describes the position and orientation of the grasping pose. Finally, the system converts this grasping pose transformation matrix into an instruction format that the robot control system can understand, such as joint angle instructions or Cartesian space pose instructions, and sends them to the robot controller to execute the grasping action. For example, for a robot used in industrial assembly, the final generated grasping instruction may contain 6 joint angle values, or a pose represented by position coordinates and quaternions. This step completes the final conversion from grasping planning to actual execution, ensuring the consistency between theoretical calculation and actual execution and realizing the closed-loop control from perception to action.

[0061] The method for generating the robot grasping pose in the embodiments of the present invention has been described above. Next, the robot grasping pose generation device in the embodiments of the present invention will be described. Please refer to Figure 2 , an embodiment of the robot grasping pose generation device in the embodiments of the present invention includes: The data preprocessing module 101 is used to obtain the RGB image, depth image and position coordinates of the target object, convert the position coordinates into a target point coordinate sequence, convert the RGB image into an original RGB matrix, convert the depth image into an original depth matrix, and perform spatial alignment processing on the original RGB matrix and the original depth matrix to generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix; The object segmentation module 102 is used to segment the target object area of the spatially calibrated RGB matrix according to the target point coordinate sequence, generate a target object segmentation mask, and calculate the object centroid coordinates of the target object segmentation mask; The feature extraction module 103 is used to generate a three-dimensional point cloud of the target object according to the spatially calibrated depth matrix, the target object segmentation mask and the camera intrinsic matrix, extract a plane point set and a non-plane point set from the three-dimensional point cloud by using the RANSAC algorithm, perform principal direction analysis on the non-plane point set, extract the axial feature vector of the target object, and calculate the plane normal vector of the plane point set; The pose generation module 104 is used to calculate the grasping parameters of the target object according to the plane normal vector, the axial feature vector and the object centroid coordinates, determine the plane normal vector as the grasping approach vector, determine the axial feature vector as the gripper opening and closing vector, obtain a grasping pose auxiliary vector through the cross product operation of the grasping approach vector and the gripper opening and closing vector, combine the grasping approach vector, the gripper opening and closing vector and the grasping pose auxiliary vector to generate a grasping rotation matrix, and generate a grasping pose transformation matrix in combination with the object centroid coordinates.

[0062] Above Figure 2 The robot grasping pose generation device in the embodiments of the present invention is described in detail from the perspective of modular functional entities. Next, the robot grasping pose generation device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0063] Figure 3FIG. 0 is a schematic structural diagram of a robot grasping posture generation device provided by an embodiment of the present invention. The robot grasping posture generation device 200 may vary greatly due to different configurations or performances, and may include one or more processors (central processing units, CPUs) 210 (for example, one or more processors) and a memory 220, and one or more storage media 230 for storing application programs 233 or data 232 (for example, one or more mass storage device terminals). Among them, the memory 220 and the storage media 230 may be transient storage or persistent storage. The program stored in the storage media 230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the robot grasping posture generation device 200. Further, the processor 210 may be configured to communicate with the storage media 230 and execute a series of instruction operations in the storage media 230 on the robot grasping posture generation device 200 to implement the steps of the above-mentioned robot grasping posture generation method.

[0064] The robot grasping posture generation device 200 may further include one or more power supplies 240, one or more wired or wireless network interfaces 250, one or more input / output interfaces 260, and / or one or more operating systems 231, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, and so on. Those skilled in the art can understand that Figure 3 the shown structural diagram of the robot grasping posture generation device does not limit the robot grasping posture generation device provided by the present invention, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0065] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions run on a computer, the computer is caused to execute the steps of the robot grasping posture generation method.

[0066] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, or units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0067] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0068] The foregoing are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structural transformation made by using the content of the specification and drawings of the present invention under the inventive concept of the present invention, or direct / indirect application in other related technical fields, is included in the patent protection scope of the present invention.

Claims

1. A method for generating a robot grasping posture, characterized in that, Including: Obtain the RGB image, depth image and position coordinates of the target object, convert the position coordinates into a target point coordinate sequence, convert the RGB image into an original RGB matrix, convert the depth image into an original depth matrix, and perform spatial alignment processing on the original RGB matrix and the original depth matrix to generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix; Perform target object region segmentation on the spatially calibrated RGB matrix according to the target point coordinate sequence to generate a target object segmentation mask, and calculate the object centroid coordinates of the target object segmentation mask; Generate a three-dimensional point cloud of the target object according to the spatially calibrated depth matrix, the target object segmentation mask and the camera intrinsic matrix, use the RANSAC algorithm to extract the plane point set and the non-plane point set from the three-dimensional point cloud, perform principal direction analysis on the non-plane point set, extract the axial feature vector of the target object, and calculate the plane normal vector of the plane point set; Calculate the grasping parameters of the target object according to the plane normal vector, the axial feature vector and the object centroid coordinates, determine the plane normal vector as the grasping approach vector, determine the axial feature vector as the gripper opening and closing vector, obtain the grasping attitude auxiliary vector through the cross product operation of the grasping approach vector and the gripper opening and closing vector, combine the grasping approach vector, the gripper opening and closing vector and the grasping attitude auxiliary vector to generate a grasping rotation matrix, and combine the object centroid coordinates to generate a grasping attitude transformation matrix.

2. The method for generating a robot grasping posture according to claim 1, wherein The obtaining the RGB image, depth image and position coordinates of the target object, converting the position coordinates into a target point coordinate sequence, converting the RGB image into an original RGB matrix, converting the depth image into an original depth matrix, and performing spatial alignment processing on the original RGB matrix and the original depth matrix to generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix includes: Receive the RGB image, depth image and position coordinates of the target object, perform format verification on the RGB image and the depth image to generate RGB image format parameters and depth image format parameters; Perform data structure conversion and normalization processing on the RGB image according to the RGB image format parameters to generate an original RGB matrix, and perform data structure conversion and numerical standardization processing on the depth image according to the depth image format parameters to generate an original depth matrix; Parse and convert the position coordinates according to a preset coordinate format rule to generate a target point coordinate sequence; Calculate the spatial coordinates of the pixel points in the original RGB matrix and the original depth matrix in the camera coordinate system according to the camera intrinsic matrix to generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix.

3. The method for generating a robot grasping posture according to claim 1, wherein, The performing target object region segmentation on the spatially calibrated RGB matrix according to the target point coordinate sequence to generate a target object segmentation mask, and calculating the object centroid coordinates of the target object segmentation mask includes: Map the coordinate points in the target point coordinate sequence to the spatially calibrated RGB matrix to generate an initial target region mask; Performing region growing processing on the initial mask of the target region to generate a region expansion mask; Performing boundary detection on the spatial calibration RGB matrix according to the region expansion mask to generate a target object segmentation mask; The centroid of the target object segmentation mask is calculated according to the distribution of pixels in the target object segmentation mask to generate the object centroid coordinates.

4. The method for generating a robot grasping posture according to claim 1, wherein The method generates a three-dimensional point cloud of the target object according to the spatial calibration depth matrix, the target object segmentation mask and the camera intrinsic parameter matrix, extracts a plane point set and a non-plane point set from the three-dimensional point cloud using a RANSAC algorithm, performs a main direction analysis on the non-plane point set, extracts an axial feature vector of the target object, and calculates a plane normal vector of the plane point set, including: According to the camera intrinsic parameter matrix, the depth values in the spatial calibration depth matrix are projected and transformed with the pixel coordinates corresponding to the target object segmentation mask to generate a projection coordinate point set, and the projection coordinate point set is converted from the camera coordinate system to the world coordinate system to generate a three-dimensional point cloud of the target object; Using the RANSAC algorithm to perform plane fitting, the three-dimensional point cloud is divided into a plane point set and a non-plane point set according to the fitting residual; Performing principal component analysis on the non-planar point set, extracting principal direction components, and generating axial feature vectors after standardizing the principal direction components; Perform singular value decomposition on the plane point set, obtain the singular vectors corresponding to the variance values in the decomposition results, and normalize the singular vectors to generate the plane normal vectors.

5. The robot grasping attitude generation method according to claim 4, wherein The RANSAC algorithm is used to perform plane fitting, and the three-dimensional point cloud is divided into a plane point set and a non-plane point set according to the fitting residual, including: The three-dimensional point cloud is divided into a feasible grasping region, and point cloud data whose distance from the camera exceeds the grasping arm span of the robot is eliminated to generate a valid grasping point cloud; Performing local curvature calculation on the effective captured point cloud according to the curvature threshold, and dividing the point cloud with a curvature value lower than the curvature threshold into a candidate plane point set; Perform RANSAC iterative calculation on the candidate plane point set, extract plane equation parameters, and calculate the distance deviation value from each point to the plane; Performing a plane support analysis on the candidate plane point set according to the distance deviation value to determine whether the plane area meets the gripper base support requirement, and determining the point set that meets the support requirement as the plane point set; The point cloud data that does not belong to the plane point set is clustered, and the outliers are removed to generate a non-plane point set.

6. The method for generating a robot grasping posture according to claim 1, characterized in that, The method calculates the grasping parameters of the target object according to the plane normal vector, the axial characteristic vector and the object center of mass coordinates, determines the plane normal vector as the grasping approach vector, determines the axial characteristic vector as the gripping jaw opening and closing vector, obtains the grasping posture auxiliary vector by cross multiplication of the grasping approach vector and the gripping jaw opening and closing vector, combines the grasping approach vector, the gripping jaw opening and closing vector and the grasping posture auxiliary vector to generate a grasping rotation matrix, and generates a grasping posture transformation matrix in combination with the object center of mass coordinates, including: Perform direction consistency judgment on the plane normal vector, reverse the plane normal vector according to the judgment result, and perform unitization operation on the processed plane normal vector to obtain the grasping approach vector; Perform numerical normalization on the axial feature vector, perform orthogonal correction on the normalized axial feature vector, and perform unitization operation on the corrected vector to obtain the gripper opening and closing vector; Perform cross product operation on the grasping approach vector and the gripper opening and closing vector, perform orthogonality verification on the cross product result, and perform unitization processing on the verified vector to obtain the grasping attitude auxiliary vector; Combine the grasping approach vector as the first column vector, the gripper opening and closing vector as the second column vector, and the grasping attitude auxiliary vector as the third column vector to generate a grasping rotation matrix; Combine the grasping rotation matrix and the object centroid coordinates in the form of a homogeneous transformation matrix, calculate the rigid body transformation parameters of translation and rotation, and generate a grasping attitude transformation matrix.

7. The robot grasping attitude generation method according to claim 6, wherein, The combining the grasping rotation matrix and the object centroid coordinates in the form of a homogeneous transformation matrix, calculating the rigid body transformation parameters of translation and rotation, and generating a grasping attitude transformation matrix includes: Convert the grasping rotation matrix into Euler angle representation, perform reachability analysis according to the robot joint motion constraints, and screen out the Euler angle parameters that meet the joint limit requirements; Perform gravity compensation calculation on the object centroid coordinates, predict the gravity influence during the grasping process, and generate the compensated object centroid coordinates; Generate a candidate grasping attitude set according to the Euler angle parameters and the compensated object centroid coordinates, and calculate the robot joint angle solutions for each candidate attitude; Perform collision detection analysis on the candidate grasping attitude set, and eliminate the grasping attitudes that may cause interference between the robot and the environment; Perform comprehensive scoring on the remaining candidate grasping attitudes according to the joint movement distance and the attitude change amplitude, and select the Euler angle parameters and position parameters corresponding to the attitude with the best score; Convert the Euler angle parameters back to the rotation matrix form, and combine them with the position parameters to generate a grasping attitude transformation matrix.

8. A robot grasping posture generation device, characterized in that, The robot grasping attitude generation device adopts the robot grasping attitude generation method according to any one of claims 1 to 7, and the robot grasping attitude generation device includes: A data preprocessing module, configured to obtain the RGB image, depth image and position coordinates of the target object, convert the position coordinates into a target point coordinate sequence, convert the RGB image into an original RGB matrix, convert the depth image into an original depth matrix, and perform spatial alignment processing on the original RGB matrix and the original depth matrix to generate a spatially calibrated RGB matrix and a spatially calibrated depth matrix; An object segmentation module, configured to segment the target object region of the spatially calibrated RGB matrix according to the target point coordinate sequence, generate a target object segmentation mask, and calculate the object centroid coordinates of the target object segmentation mask; A feature extraction module, configured to generate a three-dimensional point cloud of the target object according to the spatial calibration depth matrix, the target object segmentation mask, and the camera intrinsic matrix, extract a planar point set and a non-planar point set from the three-dimensional point cloud by using the RANSAC algorithm, perform a principal direction analysis on the non-planar point set, extract an axial feature vector of the target object, and calculate a plane normal vector of the planar point set; An attitude generation module, configured to calculate the grasping parameters of the target object according to the plane normal vector, the axial feature vector, and the object centroid coordinates, determine the plane normal vector as the grasping approach vector, determine the axial feature vector as the gripper opening and closing vector, obtain a grasping attitude auxiliary vector through the cross product operation of the grasping approach vector and the gripper opening and closing vector, combine the grasping approach vector, the gripper opening and closing vector, and the grasping attitude auxiliary vector to generate a grasping rotation matrix, and combine the object centroid coordinates to generate a grasping attitude transformation matrix.

9. A robot grasping posture generation device, characterized in that, The robot grasping attitude generation device includes: a memory and at least one processor, and instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the robot grasping attitude generation device executes the steps of the robot grasping attitude generation method according to any one of claims 1-7.

10. A computer-readable storage medium, on which instructions are stored, characterized in that, When the instructions are executed by the processor, the steps of the robot grasping attitude generation method according to any one of claims 1-7 are implemented.

Images